IDSes should be able to accurately detect, and in some cases defend against, a range of common attacks, but their added value stems from their informational and procedural capabilities. They need to function and identify attacks properly, but more important, they need to be able to aid the administrator in operating a secure environment. Keeping this as the focus, we crafted test procedures that demonstrated both novice and experienced hacking skills.

Because some of our brute-force pummeling involved really nasty packet construction, we also conducted some tests in a closed lab environment. Using sets of home-brewed tools in conjunction with commercial products, such as Ganymede's Chariot, we built baseline traffic loads on the Ethernet segments and began hurling swarms of reconnaissance and exploit data across the wire.

Our attacks never originated from the target segment--all attacks traversed at least one router. Sensors were placed on the common (target) network segment allowing them equal access to the same traffic. Our initial runs were reconnaissance-based: We chose to use Fyodor's NMAP (see www.insecure.org), an incredibly useful tool for constructing anomalous packets and scans.

By playing with timing thresholds and scanning methods (syn, fin, xmas and null scanning) we could frequently avoid detection while still gaining valuable reconnaissance data. Although we used the data we obtained from watching alarm trends, an attacker could avoid detection simply by scanning conservatively. Without the ability to fine-tune thresholds, an attacker need only learn the default time-out values to do stealth reconnaissance.

Getting a little bolder, we moved to actual exploit scripts (such as FTP, IMAP and CGI holes), which were quickly identified by the IDSes that had matching signatures. Again, we were able to pull some newer exploits (a newer WU-FTPD hole, for example) that went undetected, a testament to the danger of relying on known problems.

Convinced of the basic functionality of identifying known signatures, we launched into denial of service (DoS). The most vicious DoS attack we used was winfreeze, which saturates the wire with millions of ICMP (Internet Control Message Protocol) redirect packets. The only system that caught this attack was NetRanger; the other IDSes crashed or ignored it. As suspected, each IDS identified most of the DoS attacks out there.

Leaving the realm of destruction and going into the real world, we focused on our live targets. We used what we knew about the scanning thresholds to map out the objective in stealth mode. Satisfied with our port scans, we began logging the "banners" of specific services, which helped us identify the versions and platforms of the targeted hosts. Because these connections appear as normal sessions, they went unflagged. Finally, using the knowledge gained from our reconnaissance efforts, success was a simple matter of finding and exploiting the weakest link. We pinpointed a hole in IIS and ripped out a local NT SAM. After brute-forcing the hashed passwords, we gleaned enough account information to authenticate to other machines without detection, and we were in.