We set up NetScreen's IDP 500 and Network Associate's McAfee IntruShield 4000 in our Syracuse University Real-World Labs® and scrutinized their detection, management, performance and reporting capabilities (see "How We Tested NIP Devices"). We were impressed--though each device had some quirks, we were able to work around them without too much pain.
The Setup
We installed each product sensor in both inline and one-arm mode. The sensors were configured to communicate with one management server, and we used a management client on a workstation. Our criteria were simple: Products must detect attacks and anomalies accurately while minimizing false positives (alerts generated by legitimate traffic) and false negatives (letting attacks go undetected).
Effective intrusion prevention depends on effective intrusion detection. Any vendor claiming to have zero false positives is feeding you a line. There are too many attacks and variations of attacks, and too much legitimate traffic that looks like attack traffic, for a claim of "no false positives" to hold water. It's laughable, really.
Sure, on our controlled test bed, we didn't find any false positives, but we didn't expect any because the traffic was manufactured to our exacting specifications. When we placed the devices on our live network, however, we initially had numerous false positives, including supposed application-protocol anomalies, such as a TTL of 255 in a RIP packet, various SNMP messages, HTTP header-length messages and apparent Unicode encoding generated from the Mac OS X AOL Instant Messenger client. Tuning reduced the false-positive rate considerably.
We are convinced both of these products will block known malicious traffic, though we did need to carefully choose which alerts to block because of the potential for blocking legitimate traffic.
However, we are not convinced these products are sufficiently accurate at spotting unknown attacks through protocol-anomaly detection. Application vendors don't always write network-protocol stacks that conform to standards, thus causing legitimate traffic to be pegged as anomalous and malicious. We can't stress this enough: You must know what constitutes malicious traffic, then use existing signatures or develop your own signatures to block it.
So should you buy a NIP device? If you're planning to implement a new IDS or replace an existing IDS, we recommend choosing a NIP appliance instead to take advantage of easy deployment, good conformance to stated performance claims and flexible network integration. Plus, you get the option of blocking attacks. But we don't recommend NIP in addition to an existing IDS deployment. Better to spend the cash patching server vulnerabilities--that's something you should be doing anyway. Throwing money in that direction is a more direct way to ensure that your servers are protected because even if attack traffic gets through, patched hosts should be immune. Blocking attacks is not as critical as proper patch management (see "PatchLink Helps Keep Windows Closed,", and "Patching Patch Problems").
