We found that, though the VA market holds promise, these products still need time to mature. For example, every system we tested suffered from one problem or another: Foundstone's FoundScan, Qualys' QualysGuard and eEye's Retina had the best management and reporting features but came up short on vulnerability detection. Vigilante.com's SecureScan, SAINT and Tenable's Nessus all reported a much higher percentage of vulnerabilities but were weak on management and reporting. No product identified an acceptable percentage of vulnerabilities, though eEye's Retina and QualysGuard came close. And network administrators beware: We found these scanners far from nonintrusive. All caused adverse reactions on our network servers. The products from Qualys and Vigilante.com were by far the biggest offenders--each crashed at least five servers during our tests. The three systems that took the most abuse: Novell NetWare running Web services; a version of SuSE Linux Groupware running an exploitable version of Lotus Notes; and Windows NT 4.0 running Exchange and IIS.
Still, though no VA scanner tested was what we'd consider fully mature, we can't envision living without one. Foundstone's FoundScan is our Editor's Choice because of its detailed reporting, thorough coverage and scalable design, but only by a small margin. Retina from eEye was a close second, and we were intrigued by some of the features found in products from nCircle and Harris.
As always, real-life practicality was the focus of our tests, so we drew our criteria from concerns that have been expressed to us by security professionals across the globe.
Management: An application that is difficult to install, configure, troubleshoot and maintain will be underutilized. For example, we've seen organizations that own licensed copies of Tivoli, Computer Associates' Unicenter TNG and other network-management products but never use them because of the time and effort required to get them deployed and keep them working.
We also recognize that the data produced by a VA scanner could place an enterprise at tremendous risk if compromised, so the scanner must require authentication. Better yet, it should permit multitiered authentication. By tiering authentication, an enterprise can limit an administrator's exposure to the area he or she is responsible for, and nothing more.