DePaul's environment comprises about 10,000 nodes spread across greater Chicago. It's a typical university environment, with a variety of platforms, remote sites, research networks, residence-hall networks and an active student population. The school's network averages an Internet throughput of 30- to 38-Mbps, with traffic ranging from 5,000 to 7,000 pps (packets per second). This might not sound like the ultimate NIDS (network intrusion-detection system) thrasher, but within the first week of deployment DePaul's network whittled the NIDS field down to BlackICE, Dragon, Secure IDS and Snort. NFR's IDS dropped 2 million frames after the first 48 hours, and CA's eTrust told us we needed a processor about 4,000 times faster than our 700-MHz Pentium III box. The Bruisernet was kicking ass and taking names right from the get go.

What Do Readers Think? Check out our e-poll results on security technologies.

Our first task after the initial deployment was tuning the sensors. In their default configurations, the sensors bombarded us with tens of thousands of alerts per day. Tuning took a few weeks, as we had to determine which alerts we didn't believe to be accurate. For example, hundreds of times per day Cisco Secure IDS would alert on Qmail buffer overflows, when the traffic appeared to be just common mail. We also opted to disable signatures we didn't feel were valuable. For example, many IDSes will report on weird TCP packet anomalies. This might be useful on small networks, but after your thousandth alert on rogue fin packets, they become just plain annoying. Those deploying NIDS solutions will have to decide what they want to deal with -- tuning is definitely an issue.

After you get over the initial shock of just how hostile the Internet is -- expect reconnaissance probes and relentless exploit attempts on a daily basis -- and how blind you were without these systems, you start asking complex questions like: What now? Small organizations might see only a dozen or so attacks per week. But if your environ- ment is of significant size, you'll suffer from information overload. This is precisely why data management and presentation issues are sure to become the next hurdle for IDS vendors. In this area, Cisco's IDS event viewer in Cisco Secure Policy Manager (CSPM) and Enterasys' Dragon, even with its raw Web-based interface, are ahead of the competition. With Cisco's solution, for example, you can visually scan, sort and drill into thousands of alerts in seconds. Console interfaces from products such as RealSecure and SecureNet Pro are neat -- when you're dealing with dozens of alerts. Crank that up to a few thousand and you'll get dizzy trying to click on things as they fly by.

IDS Wish List

There are a number of components critical to a useful IDS. Two years ago we were entirely focused on the sensor portion of the technology, particularly its susceptibility to basic evasion techniques, such as fragmentation and TCP reassembly attacks. While products such as CyberSafe's Centrax and Symantec's NetProwler have yet to address many of these problems, after living with these beasts for six months we realized that we were looking at only a small piece of the big picture. Network inspection and signature technology will most likely continue to have problems for some time, but if an IDS is incapable of performing basic tasks under load or consistently overwhelms its operators with extraneous data, its ability to handle advanced evasion techniques becomes moot. Minimally, we see evolved NIDS solutions containing these foundational components:

>> Robust engines. The primary purpose of a signature-based NIDS device is to identify known attack patterns. If intruder X uses attack Y to gain access to machine Z, you want to be notified of that attack and the subsequent details. This functionality is dependent on two elements: the ability to inspect the packets that comprise the attack and the ability to recognize the attack for what it is. The first is dependent on the inspection engine; the second is dependent on a database of known attack signatures. If the inspection engine is not capable of investigating all the traffic at high rates, the effectiveness of the NIDS device is compromised.

A number of products -- Cisco's Secure IDS, Enterasys' Dragon, ISS' BlackICE, Intrusion.com's SecureNet Pro and Snort -- now have engines that appear quite robust. These were the only engines that did not crash hard on us at some point during our tests. Moreover, we found it extremely disturbing that most solutions were incapable of notifying us when they started dropping frames. If you're not alerted when frames are being dropped, you have a temperamental alarm system, at best. Imagine a home security system that turned itself off from time to time without letting you know. Anzen, NetProwler and NFR offered the only systems that made packet-drop statistics available, but they still didn't send alerts to the consoles when these situations occurred.

>> Strong and timely signature bases. All the NIDS products we reviewed are signature-based (as opposed to anomaly based). The current signature-based systems rely on updated pattern-matching routines to flag attacks. For example, when the Microsoft Internet Information Server (IIS) version 5 ISAPI overflow vulnerability was released, most NIDS devices couldn't "see it." While some generic signatures, like the Intel NOP code alert (long strings of NOP codes, 0x90 in hex, are common in Intel-based remote buffer overflow-based attacks), can sometimes catch new hits, NIDS vendors have to update their signature bases to catch most new attack types. In short, current NIDS devices are able to look only for known attack types.

Some products that perform protocol analysis, such as BlackICE, are able to flag new attacks proactively. For example, BlackICE alerted on the IIS ISAPI vulnerability (before the vulnerability was publicized) as a "field overflow" because it violates the HTTP specification. Whether a security administrator will pay attention is another matter, but it's definitely a nice feature to have.

In many ways, the NIDS scene continues to suffer the same fate as that of the antivirus market: It's reactive, at best. ISS' Express-Update feature gives RealSecure users the ability to easily update signature sets, but ISS is one of the few vendors that has streamlined this process. In our testing, we picked some of the top vulnerabilities being exploited on the Internet today and used them as a base for comparison. A quick glance at "Network IDS Signature Results" shows which vendors are on top of their signature-development efforts. The open-signature model of Dragon and Snort (see Whitehats' free arachNIDS database) will most likely continue to propel these two to the top from a coverage standpoint. We were disappointed at how far behind products such as Centrax, NetProwler and SecureNet Pro were on the signature front.

>> Remote manageability. While marketing pitches may say all sensors are easily controlled via centralized management consoles, experience taught us otherwise. Throughout our trials we were constantly rebooting sensors, checking logs, looking at processor utilization and performing myriad other general housekeeping tasks. While we don't consider ourselves biased toward Unix, the simple fact is that the Unix-based systems (Dragon, Secure IDS, SecureNet Pro and Snort) were much easier to maintain, remotely, than the Microsoft Windows NT-based solutions. Opening up remote telnet and SSH (secure shell) shells in a bandwidth-constrained environment is easy. Struggling with VNC (virtual network computing), Symantec's pcAnywhere or Famatech's Remote Administrator for the Win32 solutions was painful. When our VPN tunnel was slow, remotely performing maintenance on the NT-based sensors was nearly impossible.

>> Scalable frameworks. Populating a console with a few alerts is one thing. Trying to navigate thousands of alerts from dozens of sensors is quite another. For example, the alerts in Intrusion.com's SecureNet Pro scrolled by so quickly that we barely had enough time to click on them. In contrast, even Dragon's less-than-friendly Web interface gave us enough control to casually sift. Apparently Cisco saw this problem coming early on, as its event viewer has a "freeze updates" button that is a godsend.

Also of concern are the back-end databases driving these solutions. Some systems use flat-file architectures, while others can integrate with real relational database management systems. Also, how the management interfaces, database and console handle massive amounts of data becomes a huge issue in large environments. This will be less of a problem for a small organization, which will see just a couple dozen alerts from a few sensors, but in the enterprise space, it's critical.

>> Usable interfaces. If you can't quickly and easily extract or monitor incoming data, you're at a tremendous disadvantage from the outset. Cisco's event viewer in CSPM does a good job on the real-time viewing and sorting of alerts but falls flat in long-term data mining. We had problems with Dr. Watson and Mr. CSPM when dealing with command-line tools and large data sets. Dragon can present the data, but getting data out of the interface is painful, and the situation goes downhill from there. It's obvious to us that a number of the IDS vendors haven't used their products in large environments, as their interfaces and data-extraction abilities are horrendous.

>> Data mining and correlation functionality. It's one thing to be able to see an alert message in real-time, quite another to correlate it with other data points. Put simply, intrusion-detection solutions just aren't there yet. Cisco and Enterasys make it easy to correlate data from multiple sensors, but their products are a long way from taking in data from routers, firewalls or other devices that might have data on attackers. For now, organizations wishing to correlate data will have to turn to products like Intellitactics, netForensics and OpenSystems' Private I, or develop home-grown solutions.

Bottom Line

While we would like to declare a clear winner and move on, one-size-fits-all is not in the current IDS vocabulary. Adopters of intrusion-detection technology have to make lots of decisions, the first of which is the classic HIDS (host-based IDS) versus NIDS debate. NIDS solutions are still far easier to deploy, but they can't deliver some of the in-depth data that HIDS devices offer. However, while HIDS solutions monitor more under-the-hood items, such as system calls, binary integrity and OS event logs, they're still difficult to deploy and also require installations on every machine you want to monitor. Right now, NIDS products give more bang for the buck in terms of coverage. (For more on HIDS, see "What's New With HIDS".)

The best choice, of course, is an integrated HIDS/NIDS solution (see "Integrated IDS Solutions" report card). If you choose this route however, you're in a tough spot. The strongest NIDS players do not have strong HIDS offerings, and the strongest HIDS products are weak on the NIDS side. For example, if your network is a larger (5,000-plus nodes), medium- to high-bandwidth (6,000-plus packets per second or 40-plus Mbps) environment looking to deploy a NIDS solution, based on our testing, Cisco's Secure IDS and Enterasys' Dragon are your only real options.

It's not the sensor/engine technology that limits many of the other products; it's their management frameworks. BlackICE, SecureNet Pro and Snort all have robust NIDS engines that can keep up with high traffic volumes. However, good luck trying to manage the thousands of alerts you'll receive -- these management consoles can serve as human denial-of-service attacks when under load. Because Snort is open source, you do have the flexibility of building your own management interfaces or using other open-source packages, such as ACID (Analysis Console for Intrusion Databases) or the SnortSnarf PERL script. Unfortunately, most organizations don't have the development resources needed to get a Snort interface as usable as, say, CSPM's event viewer. It will be interesting to see if organizations such as Sourcefire will be able to slap a robust commercial back end and console to Snort.

On the HIDS side it's a completely different story. Cisco doesn't have a HIDS offering, and Enterasys' product is in its infancy. ISS and Symantec are strong on the HIDS front, boasting support for many platforms, but their NIDS offerings have some serious limitations. CyberSafe probably has one of the best HIDS visions, and if ISS doesn't ax the interoperability between Centrax and BlackICE, that will be one product bundling to watch.