• Coverage: Because a vulnerability scanner is only as good as its ability to discover vulnerabilities, we rated each product's skill in accurately identifying system and application vulnerabilities on various OSs and platforms. We reviewed results from each product for accurate OS identification, improper identification of nonexistent vulnerabilities (false positives) and failure to identify known vulnerabilities (false negatives).

• Performance and scalability: The performance of a vulnerability scanner often tips the scales on whether it will be a help or a hindrance. A scanner that reports a vulnerability after it has been exploited is pointless, as is a scanner that hits the servers it's testing with a DoS (denial of service) attack because it isn't tuned to scale down its assessment.

We reviewed each VA for its ability to fine-tune its assessment settings: Can the product's thread count and packet intervals be adjusted? We found a tremendous amount of discrepancy here, as several scanners by default scanned at an average rate of about 50 Kbps while others thrashed about at 3.5 Mbps. Although this won't account for an inordinate amount of an enterprise's network bandwidth, it helped us understand why several scanners took hours to complete our tests and others finished in minutes. We think it also helps explain why, during simple tests, such as Web crawling, some scanners crashed targets more frequently than others. When comparing apples-to-apples vulnerability scans, the products used about the same amount of total bandwidth; some were just tuned, by default, to do it quicker.

Of course, mere packet count wasn't the primary factor determining whether a target suffered an outage. Invasive tests, such as brute-forcing accounts and executing DoS attacks, can also crash a target system.

• OS fingerprinting: Scanners send targets malformed IP requests in an attempt to extract a response. The manner in which an OS responds to these requests helps the scanner identify the type of OS that has replied. Depending on the request, as well as the maturity of the OS's IP stack, a system might encounter a failure. For example, Nessus' methods will crash older systems, while FoundScan's more RFC-friendly approach to fingerprinting rarely does.

Furthermore, we tested each product for its ability to remain stable while scanning large address ranges. Although our test bed contained fewer than 30 machines, a VA scanner must examine any range of systems designated as its target base. Our network was segmented into four class "C" address ranges, so that's what we submitted to our scanners. Most of the products handled the load with ease. We input all our addresses into each of the products; however, Beyond Security's scanner wasn't able to finish the workload, and Vigilante.com's SecureScan NX failed several times before presenting us with a completed scan.

Product Features click to enlarge

In enterprise environments, a more distributed deployment method--as opposed to deploying a single scanning device--can prove beneficial. Enterprises do not want to burn WAN bandwidth with vulnerability-scanner traffic, and scanners often encounter problems with system identification across multiple routers, proxy servers and firewalls. In fact, we found one segment of our mock environment especially tricky for several products under test--on TCP- and UDP-based identification scans, Rapid7's NeXpose and Vigilante.com's SecureScan reported responses from systems that didn't exist! Best we could tell, our Cisco PIX firewall (acting as a simple router in this case) was sending replies to the scanner, indicating that there was no host on the other end; the scanner interpreted the PIX's response as a positive host finding. This caused a tremendous amount of overhead, as these scanners spent hours attempting to identify what services were running on nonexistent servers. This is where products such as eEye Retina and Tenable Lightning can prove useful, by allowing multiple scanners to be deployed throughout the environment, all reporting back to a single aggregator.

• Price: We waited until we were nearly finished testing to look at prices because we didn't want our opinions skewed by our perception of what a particular product "should" provide for its price. We found that product pricing accurately matched the features being offered, with a few exceptions: Tenable's Nessus appliance, which retails at $20,000 with an additional $12,000 to license Lightning for five users; Beyond Security's Automated Scanning Server, which retails at $12,000; and Rapid7's NeXpose software, starting at $8,750 for only 64 specified IP addresses. These products don't seem worth the price.

Our analysis of the top seven finishers follows. You'll find details about the other four products here. In addition, our extensive table of vulnerabilities sought and detected can be found here.