VA Scanners Pinpoint Your Weak Spots

We found that, though the VA market holds promise, these products still need time to mature. For example, every system we tested suffered from one problem or another: Foundstone's FoundScan, Qualys' QualysGuard and eEye's Retina had the best management and reporting features but came up short on vulnerability detection. Vigilante.com's SecureScan, SAINT and Tenable's Nessus all reported a much higher percentage of vulnerabilities but were weak on management and reporting. No product identified an acceptable percentage of vulnerabilities, though eEye's Retina and QualysGuard came close. And network administrators beware: We found these scanners far from nonintrusive. All caused adverse reactions on our network servers. The products from Qualys and Vigilante.com were by far the biggest offenders--each crashed at least five servers during our tests. The three systems that took the most abuse: Novell NetWare running Web services; a version of SuSE Linux Groupware running an exploitable version of Lotus Notes; and Windows NT 4.0 running Exchange and IIS.

Still, though no VA scanner tested was what we'd consider fully mature, we can't envision living without one. Foundstone's FoundScan is our Editor's Choice because of its detailed reporting, thorough coverage and scalable design, but only by a small margin. Retina from eEye was a close second, and we were intrigued by some of the features found in products from nCircle and Harris.

As always, real-life practicality was the focus of our tests, so we drew our criteria from concerns that have been expressed to us by security professionals across the globe.

• Management: An application that is difficult to install, configure, troubleshoot and maintain will be underutilized. For example, we've seen organizations that own licensed copies of Tivoli, Computer Associates' Unicenter TNG and other network-management products but never use them because of the time and effort required to get them deployed and keep them working.

We also recognize that the data produced by a VA scanner could place an enterprise at tremendous risk if compromised, so the scanner must require authentication. Better yet, it should permit multitiered authentication. By tiering authentication, an enterprise can limit an administrator's exposure to the area he or she is responsible for, and nothing more.• Data management & reporting: Over the years, we've tested enterprise-class firewalls, intrusion-detection systems, SIM suites and other high-level security systems. From those tests and our experience in the field, we've learned that reporting is both important to security professionals and often overlooked by vendors. IDSs, VA scanners and log aggregators maintain a great deal of data, but they're all worthless unless they can be used by the individuals they're supposed to help. Because a typical scan can return thousands of findings--all of which require analysis by security professionals--we placed a heavy emphasis on reporting capabilities.

We rated each product on its report content, ability to sort and cross-reference, and ability to export results to a transportable or shared medium. We also tested each application for its ability to report changes from previous scans.

• Coverage: Because a vulnerability scanner is only as good as its ability to discover vulnerabilities, we rated each product's skill in accurately identifying system and application vulnerabilities on various OSs and platforms. We reviewed results from each product for accurate OS identification, improper identification of nonexistent vulnerabilities (false positives) and failure to identify known vulnerabilities (false negatives).

• Performance and scalability: The performance of a vulnerability scanner often tips the scales on whether it will be a help or a hindrance. A scanner that reports a vulnerability after it has been exploited is pointless, as is a scanner that hits the servers it's testing with a DoS (denial of service) attack because it isn't tuned to scale down its assessment.

We reviewed each VA for its ability to fine-tune its assessment settings: Can the product's thread count and packet intervals be adjusted? We found a tremendous amount of discrepancy here, as several scanners by default scanned at an average rate of about 50 Kbps while others thrashed about at 3.5 Mbps. Although this won't account for an inordinate amount of an enterprise's network bandwidth, it helped us understand why several scanners took hours to complete our tests and others finished in minutes. We think it also helps explain why, during simple tests, such as Web crawling, some scanners crashed targets more frequently than others. When comparing apples-to-apples vulnerability scans, the products used about the same amount of total bandwidth; some were just tuned, by default, to do it quicker.Of course, mere packet count wasn't the primary factor determining whether a target suffered an outage. Invasive tests, such as brute-forcing accounts and executing DoS attacks, can also crash a target system.

• OS fingerprinting: Scanners send targets malformed IP requests in an attempt to extract a response. The manner in which an OS responds to these requests helps the scanner identify the type of OS that has replied. Depending on the request, as well as the maturity of the OS's IP stack, a system might encounter a failure. For example, Nessus' methods will crash older systems, while FoundScan's more RFC-friendly approach to fingerprinting rarely does.

Furthermore, we tested each product for its ability to remain stable while scanning large address ranges. Although our test bed contained fewer than 30 machines, a VA scanner must examine any range of systems designated as its target base. Our network was segmented into four class "C" address ranges, so that's what we submitted to our scanners. Most of the products handled the load with ease. We input all our addresses into each of the products; however, Beyond Security's scanner wasn't able to finish the workload, and Vigilante.com's SecureScan NX failed several times before presenting us with a completed scan.

In enterprise environments, a more distributed deployment method--as opposed to deploying a single scanning device--can prove beneficial. Enterprises do not want to burn WAN bandwidth with vulnerability-scanner traffic, and scanners often encounter problems with system identification across multiple routers, proxy servers and firewalls. In fact, we found one segment of our mock environment especially tricky for several products under test--on TCP- and UDP-based identification scans, Rapid7's NeXpose and Vigilante.com's SecureScan reported responses from systems that didn't exist! Best we could tell, our Cisco PIX firewall (acting as a simple router in this case) was sending replies to the scanner, indicating that there was no host on the other end; the scanner interpreted the PIX's response as a positive host finding. This caused a tremendous amount of overhead, as these scanners spent hours attempting to identify what services were running on nonexistent servers. This is where products such as eEye Retina and Tenable Lightning can prove useful, by allowing multiple scanners to be deployed throughout the environment, all reporting back to a single aggregator.

• Price: We waited until we were nearly finished testing to look at prices because we didn't want our opinions skewed by our perception of what a particular product "should" provide for its price. We found that product pricing accurately matched the features being offered, with a few exceptions: Tenable's Nessus appliance, which retails at $20,000 with an additional $12,000 to license Lightning for five users; Beyond Security's Automated Scanning Server, which retails at $12,000; and Rapid7's NeXpose software, starting at $8,750 for only 64 specified IP addresses. These products don't seem worth the price.Our analysis of the top seven finishers follows. You'll find details about the other four products here. In addition, our extensive table of vulnerabilities sought and detected can be found here.

FoundScan was one of the most polished products we tested. Its management interface is clean, understandable and relatively stable, though we did lock up several times during invasive scans. We especially liked being able to restrict user access, allowing multiple levels of administrative control. This feature fits well with a product that also offers a ticketing system for remediation of identified vulnerabilities.

Although Foundstone didn't offer much in the way of data export, its HTML reports were clean, relatively easy to understand and could be sorted in a variety of ways, letting administrators efficiently view reports. Furthermore, because the scanner writes directly to a Microsoft SQL Server, organizations can build their own reports by directly accessing the system's databases.

What this product lacks in reporting it makes up for with its remediation ticketing system, which lets enterprises share the "vulnerability joy" among network and security administrators. The only other product we saw with this type of system was eEye's Retina. Unfortunately, these ticket systems don't integrate with other helpdesk/trouble-ticket software.

Foundstone allows a great deal of flexibility for tuning performance. VA administrators can change the total number of concurrent threads, the overall scan acceleration, the packet interval and the total number of scan objects allowed. Although we found the defaults solid for the test group we were scanning, performance can be adjusted to scan a larger test group more efficiently.

Foundstone's vulnerability database exceeds 2,000 entries, but it detected only about 50 percent of our vulnerabilities. Unfortunately, the 50 percent mark wasn't all that shabby compared with its rivals: No product came close to detecting all the vulnerabilities.As a side note, similar to other products on the market, Foundstone has preconfigured several scanning templates for one purpose or another. The "safe scan" template is intended to prevent target system outages during scanning. Unfortunately, we did encounter outages with NetWare using the "safe scan." In fact, Foundstone's Web crawler feature caused that outage. Fear not, though: Novell has a patch for that DoS. The key to remember here is that no automated scanner is completely safe; caution should always be used.

Overall, we felt that Foundstone offers a substantial bang for the buck. With any luck, the next release will take care of some of the reporting shortfalls, stabilize the system during invasive tests, maybe even integrate the two separate management interfaces into one complete front end. We'd also like to see more integration with an organization's asset-classification effort. When asset classification is calculated with vulnerability severity, an enterprise can better direct its resources to the areas that need the most protection, and these are features Foundstone identified on its road map.

Foundstone Enterprise with FoundScan Engine 2.6, starts at $15,000. Foundstone, (877) 91-FOUND, (949) 297-5600. www.foundstone.com

Qualys' internal scanning appliance is a gateway to its Internet-based scanning service. This setup is very similar to that of nCircle's IP360, where several scanning appliances reference a single management server for all configuration and vulnerability information. However, the management and aggregation server resides on Qualys' system, not at the customer location.

Organizations install the QualysGuard appliances inside their enterprises and administer them from the Internet-based interface. The gateway device simply makes outgoing SSL-encrypted requests to Qualys' servers, asking if there are any jobs to perform. If the appliance finds a job, it downloads it, and away it goes. In a nutshell, nothing is stored on the QualysGuard internal appliance. Scan requests, reports and even scan signatures reside on Qualys' network. Although this might seem like a strange model, it means you never have to worry about attack-signature updates. Because of its design, this system is an excellent choice for large enterprises wanting to deploy scanners throughout their networks.

Qualys' reports can be customized, and its vulnerability detection was acceptable, though we hope to see better coverage in the future. Tiered user privileges can be tailored to an organization's demands, and user preferences can be adjusted for dead-host scans, load-balancer detection and even password brute-forcing. To add a layer of device segregation, network hosts can be separated into groups, enabling security administrators to create itemized reports based on business criticality, for example.Our wish list items: Being able to export raw data to additional formats (currently raw data can be exported in XML, HTML and MHT only), integration with a ticketing system and greater integration with enterprise asset-classification efforts.

QualysGuard Intranet Scanner $2,995; price of annual subscription service depends on number of hosts scanned. Qualys, (800) 745-4355, (650) 801-6100. www.qualys.comHarris is on to something with STAT Scanner--it not only scans a very wide array of vulnerabilities but also incorporates policy/registry checking and remediation. This product lets an administrator set registry, log and user policies that can be manually or automatically updated upon detection.

One area that sets STAT Scanner apart from peers is its noninvasive nature. This product doesn't offer a "safe scan," because it doesn't need it. However, this design is both an asset and a liability. Because there are no unsafe scans available, the risk of target meltdown is almost completely mitigated (we still recommend caution because we did encounter a few application issues); however, this product does require authentication for each and every target, and failure to provide such authentication will result in a tremendous number of false positives and false negatives.

We attempted scanning without any authentication parameters on several hosts; the system simply indicated that the open port might be a Trojan. This could be a serious problem for large organizations, particularly those with varied administrative realms. This limitation hinders the ability to scan a large number of nonsimilar networks without a great deal of intervention and departmental cooperation. Although administrators can create authentication groups and assign those groups usernames and passwords, we still see this as crippling.

Finally, STAT was incapable of assessing our NetWare servers. Although STAT will attempt to assess other system types, it is best-suited for Microsoft and Unix environments.When it comes to reporting, STAT Scanner offers the widest array of export options we've seen. Out of the box, STAT Scanner results can be exported to .MDB format, with all database tables and even a couple of query tables preformatted for Microsoft Access. There are also several reports to choose from, each of which can be exported into various formats, such as CSV, Excel, Word, Lotus and HTML.

Harris offers STAT Analyzer to complement STAT Scanner. STAT Analyzer uses Ipswich's What'sUpGold for system monitoring and inventory; can execute and control Nessus Vulnerability Scanner and Harris' STAT Scanner; and can import test results from ISS' Internet Security Scanner. The result is a complete report of aggregated data from multiple scanners, likely producing a larger percentage of detected vulnerabilities than any one system alone.

STAT Scanner Professional Edition 5, as tested with a 50-node license and a one-year maintenance license, $1,995. Harris Corp., (888) 725-7828, (321) 727-9100. www.stat.harris.com

The only area we found lacking with Retina was its ability to detect the vulnerabilities we laid out for it. Unfortunately, that hurt its score considerably. On the bright side, eEye's enterprise version of Retina is a fully distributed model that uses Microsoft SQL for data storage and a management and aggregation server to control remote scanners. What's more, eEye has incorporated multiuser authentication, much better reporting than its nonenterprise version (though exportability is still a bit lacking), and a comprehensive ticketing system, similar to that used by Foundstone.

It's evident that Retina enterprise was built to scale, and it should suit larger organizations quite well. We look forward to seeing what eEye has to offer in the future. Although Retina's Unix scanning capability is a bit lacking, the product did find a large percentage of the Windows vulnerabilities--pushing its overall percentage of detected vulnerabilities to about 55 percent.

We would like to see more integration with corporate asset-classification efforts, more detection capabilities and some additional options for exporting reports.Retina Network Security Scanner, $995; management console, $15,000. eEye Digital Security, (866) 339-3732, (949) 349-9062. www.eeye.comSecureScan NX has a well-designed user interface and several slick features, such as in-scan vulnerability review and the ability to select scan type by risk, service, platform, impact CVE and several others.

We ran into problems, though, during large scans of multiple networks. We locked up the application several times before getting a good clean scan, especially while attempting to identify hosts based on TCP and UDP scans, instead of simply ICMP. Once we worked out the bugs, however, we were able to obtain a good result, and SecureScan NX stood up well against its peers at detecting vulnerabilities; in fact, SecureScan NX detected the largest number--65 percent--of vulnerabilities in our test group.

Although SecureScan NX's reporting is good, we would have liked a remediation explanation in the main body of the report rather than being forced to follow a link to Vigilante's Web site to obtain it.

SecureScan NX 2.6.50, as configured for this test $635 for 10 IPs/year. Vigilante.com, (503) 579-3464. www.vigilante.com

SAINT proved a formidable opponent but unfortunately, like every other scanner, it sails in some areas, sinks in others. SAINT's vulnerability coverage was above average, and its price is right, but we felt the product could be improved on the management and reporting fronts.

Although SAINT takes a bit more know-how than do the products from Foundstone, Qualys and nCircle, it runs over a standard Linux distribution and has the easiest install script we've seen over a Linux command shell. We highly recommend the Express plug-in (www.saintcorporation.com/products/saint_express.html); without it, performing updates is a tedious process. We hope SAINT will build Express into the standard product in the future.The most annoying problem was with adding IP addresses. You'd think this should be a simple task, but not so: To enter address ranges from multiple subnets, you must pull from a text file. If any address fails, the entire scan fails, but not necessarily right away. On several occasions we had to wait for half an hour before SAINT bombed on one address that we'd entered incorrectly. SAINT would benefit from a more intuitive interface for programming multiple addresses and address blocks.

Although SAINT doesn't offer much in the way of exportable reports, it does provide some well-designed prebuilt reports and lets you create your own. SAINT's reports make extensive use of hyperlinks--letting us jump from an address to an explanation of that entire system and so on; unfortunately, we soon found ourselves lost in the jumps. We believe that a dynamic reporting interface would prove much more efficient than simple hyperlinks. SAINT is a good solution for small-to-midsize organizations, but it doesn't have the aggregation capabilities needed for larger enterprises.

SAINT 4.3, 10 hosts: $639; Class C: $2,495; 500 hosts: $5,195; auditing licenses: $395 to $9,495, SAINT Corp., (800) 596-2006, (301) 656-0521. www.saintcorporation.comnCircle's IP360 is an extremely low maintenance and highly distributed VA system. Scanning appliances pull jobs, updates and vulnerability signatures from a master control and aggregation server over a secure transport. Once the appliances have been assigned jobs, they run at routine intervals, sending results back to the aggregator for reporting. Although this push/pull method makes the IP360 unique and easy to maintain, it also is a hindrance: Once a job is assigned to a scanning appliance, it scans indefinitely until it's stopped manually. In fact, we couldn't find any way to issue a simple one-time scan. Even when we tried to stop the appliance through the management server, the scans continued to function, and we were forced to restart each of the appliances manually. However, we were able to reboot appliances from the management interface.

On the management side, the Web-based user interface is a bit cumbersome: It consists of a series of tabbed pages that continually populate downward. One simple tab selection can result in an additional two or three rows of items. At one point we were confronted with four rows of tabs. IP360 would benefit from a new GUI design.

Although the IP360 doesn't offer an abundance of reports, those it does offer are well-designed, can be sorted by IP address or vulnerability, and provide an excellent array of cross-referencing hyperlinks to take you from one area in the report to another.We liked being able to add commentary about individual assets. Although the main purpose of this is to better identify assets, it can be taken a step further. By including additional information, such as a corporate risk factor, about each asset and integrating such information with nCircle's numerical rating system, the company could offer a very useful tool for prioritizing internal security efforts.

IP360 Vulnerability Management System 5.3, $30,000 (one scanning appliance with 256 active hosts and management/aggregation server). nCircle Network Security, (888) 464-2900, (415) 625-5900. www.ncircle.com

Read about the other four products we tested here.

Kevin Novak is the director of consulting services for Chicago-based security consultancy Neohapsis. Write to him at [email protected].

Post a comment or question on this story.

Tenable Nessus Appliance 1.0 with Tenable Lightning 1.1 | BindView Corp. bv-Control for Internet Security 7.2 | Rapid7 NeXpose 3.0 | Beyond Security Automated Scanning Server 1.4

Tenable Nessus Appliance 1.0 with Tenable Lightning 1.1

Tenable Lightning is a commercial front-end and correlation solution for the popular Nessus open-source scanner. Lightning adds some additional scanning and reporting capabilities; the ability to ticket and comment on found vulnerabilities; the ability to deploy scan sensors across the enterprise; and the integration of output generated by Nessus Scan and various network intrusion-detection systems, such as Bro, Dragon, RealSecure and Snort. By combining VAs with IDSs enterprises can see a detailed picture of how an open vulnerability might be an active compromise within their environments.

Although we did find several enhancements to Tenable's new Nessus front end, we found drawbacks as well. For instance, administrators no longer have a real-time display of the scanner's progress, and attempting to stop an active job sent us deep into the CLI, where we had to remove the active scan file manually. There's no mistaking this is still a new product, and it requires knowledge of the back-end OS--Linux--to make things happen.

Lightning is off to a good start. More comprehensive reporting, additional work with the ticketing interface, more granularity for user permissions and an overall more user-friendly interface, and Tenable might take Nessus to bigger and better places.

Tenable Nessus Appliance 1.0, $20,000. Tenable Network Security, (410) 872-0555. www.tenablesecurity.com

BindView Corp. bv-Control for Internet Security 7.2

Bv-Control for Internet Security is only a small piece of BindView's complete bv-Control Suite, but it has a great deal of potential. The management interface, a snap-in to the Microsoft Management Console, is uncluttered and easy to understand. The application incorporates policy-compliance scanning and lets administrators fix some registry and policy vulnerabilities that appear in its reports.Unfortunately, determining whether this product could locate all our vulnerabilities was an overwhelming task. Bv-Control reported more than 800 pages of results, but we found little evidence of CVE numbers. In fact, of the small percentage of vulnerabilities in our list (12 percent), only about half actually noted the CVE number; the other half were found by sheer grunt work.

Overall, like many of the other products we tested, bv-Control for Internet Security is strong on one front and weak on another. However, with more thorough tests and a richer reporting interface, this product would do quite well.

bv-Control for Internet Security, per IP address: $19.95; per class C subnet: $3,995; per class B subnet: $32,000; bv-Control for Internet Security requires the use of BindView RMS, which is priced at $1,995 for one nonconcurrent user. BindView Corp., (800) 813-5869, (713) 561-4000. www.bindview.com

Rapid7 NeXpose 3.0

If you can get past its retro, flashback-to-GEOS-in-the-mid-1980s look, this application has quite a bit to offer. The management interface is simple and offers many of the elements we look for in a scanner, plus a few extras, such as network monitoring/sniffing. However, it could not detect all our vulnerabilities, and it had an abnormally long hang time between starting a scan and producing results.

NeXpose's reports are clear and easy to read and can be exported to various database formats, including Oracle, Microsoft SQL and ODBC, as well as HTML, XML and text. One really helpful report created by Rapid7 is the "Remediation Report, which clearly defines the steps needed to fix the vulnerabilities it detects, including the amount of time the repair should take. This product may not patch your servers automatically, but it does a fine job instructing how to do it manually. NeXpose's reports aren't very flexible about re-sorting and manipulating data, but we could have just as easily created our own reports once the data had been exported to a database.NeXpose 3.0, One 64 IP fixed license to allow scanning of 64 specific IP addresses lists at $8,750; two fixed Class C licenses list at $40,000; prices include one year of support, upgrades and vulnerability subscriptions. Rapid7, (866) 7RAPID7, (212) 558-8700. www.rapid7.com

Beyond Security Automated Scanning Server 1.4

Beyond Security's Scanning Server was the least mature of the products we tested. The Web-based interface is difficult to work with and lacking in features; it rarely performed as expected. Simple tasks, such as initiating a scan, failed almost as often as they worked, especially when attempting to scan our entire test base (four Class C networks). Report extraction is an interesting process because the only method by which to obtain reports is via an e-mail (albeit, there is a secure e-mail option).

Scanning Server did a decent job finding the more hazardous vulnerabilities plaguing our network (35 percent overall), however, so it might seem an OK pick for smaller organizations--until you consider the cost. This turnkey system has a price tag of around $12,000! Beyond Security needs to beef up its product, lower its price, or both.

Automated Scanning Server 1.4, as tested, server (hardware and software), including a license to scan 100 specific IPs an unlimited number of times: $12,000. Beyond Security, (800) 801-2821, (323) 882-8286. www.beyondsecurity.comWe modeled our vulnerability-assessment tests on real-world conditions. Our approach was straightforward: We deployed 27 devices of different types--Windows, Linux, BSD, NetWare, Solaris, firewalls, routers and switches--with varying levels of patches and ran each scanning solution against this environment to identify known vulnerabilities. We then compared the results, measured the time each scanner took to complete the scans, and noted the state of the target systems after the scanner completed its job.

Although the task of testing 11 VA scanners against a static environment and comparing the results may seem simple, we found the exercise far from easy. Each product offers a different set of features, has different configuration methods and covers various applications and OSs to varying degrees. But what really plagued us was the comparison method: How do you evaluate hundreds of vulnerabilities--sometimes close to a thousand pages of text--across 11 products?Because many of the products we tested reported thousands of vulnerabilities, we needed a common taxonomy to compare results. We chose CVE numbers (see cve. mitre.org) because they were the lowest common denominator between products, and for the most part, the effort is comprehensive. Unfortunately, using CVE addressed only part of the problem; the real challenge lay in parsing the reports (see our list of CVE numbers tested and how the products fared at www.nwc.com/1412/1412rd5.html).

Although Foundstone Enterprise and QualysGuard have well-designed reporting utilities, others, including Tenable Lightning (Nessus), Beyond Security's Automated Scanning Server and bv-Controls for Internet Security, have reports that are difficult to read and even more difficult to manipulate and re-sort. Complicating matters, we found that report content was often dissimilar. For example, nCircle's reports were so detailed we could review the entire attack decode to see how the vulnerability worked, while Vigilante's reports didn't even include remediation information; we had to follow a link to its Web site for further details.

Although all the products display a common vulnerability ID number (such as CVE or CERT) somewhere, they don't always list the place upfront. In fact, we noted several occasions where the vendor rolled several vulnerabilities into one heading and failed to list all the CVE numbers it represented. We wound up with a best attempt at digesting and comparing thousands of pages of reports. We ran all the scanners an exhausting number of times and spent weeks rebooting and resetting our systems and test bed. However, it is possible that a scanner may have flown under our radar as it knocked a service offline, inaccurately detected something it claimed to detect, or functioned irregularly in our environment. None of these factors would have radically change our results--and they are situations most organizations will face--but there is a margin of error. After all, even we need four hours of sleep once in a while.Our vulnerability-assessment chart

Security white papers & research reports

Security booksOur weekly vulnerability and patch newsletter

Current Internet threat report

"Secure to the Core"

"Application-Level Firewalls: Smaller Net, Tighter Filter"

"One for All"CERT/CC Overview Incident and Vulnerability Trends

SANs paper on distributed scan model for enterprise-wide network vulnerability assessmentUpdated July 15th, 2003