Internet Scanner found close to 14 of our 17 vulnerabilities, beating out everything except Nessus Security Scanner on the vulnerability identification front. While the wording in the reports could use a little work, they were clear, and the fix information was incredibly thorough. We had our fair share of problems getting Internet Scanner 6.1 up and running, though, as it is quite fussy about service packs and Windows platforms. (It will not run on some Windows NT 4 servers, for example.) Once we did get it properly installed and running, however, we were impressed with its abilities.

Internet Scanner provides a wide range of vulnerability checks. We were impressed right off the bat when it dug into the NetWare server, as most of the other scanners pretty much ignored that system. Internet Scanner was the only scanner that had something on the NetWare front other than guessable SNMP community strings: It alerted us to general problems in the LDAP service, which at this point we can't confirm as a vulnerability, but seeing a scanner tell us something we didn't already know was nice.

Internet Scanner is not without its quirks, however. For example, when running it against our Solaris host, it alerted us to (older) security problems in sadmind; however, this alert was not in reference to the severe sadmind buffer overflow that is being exploited on the Internet today. Internet Scanner also alerted us to security problems in statd, and yet, the presented report told us that the host (running Solaris 2.6) is not vulnerable. The product also appears to have a bizarre fixation with the shareware BisonWare FTP service for Windows--it reported a BisonWare alert on every host that ran FTP. We're not sure where it got BisonWare or why it thought we were running a piece of Windows shareware on our Solaris host, but this did force us to question the product's accuracy. Making matters worse, Internet Scanner reported that it was able to use the test user account to log into FTP on our Linux machines. Even after double-checking, we still don't know where this claim came from, as our Linux machines did not have a test account.

When the product scanned Windows NT, the results were a little more accurate. In fact, Internet Scanner even alerted us to the recent IIS Unicode bug, which was not counted as one of our 17 predefined vulnerabilities. However, we were amused by the scanner's recommendation that we remove the guest account and be suspicious of the internal IIS accounts (IWAM_machine and IUSR_ machine). The guest account is a system account and cannot be removed, and obviously the IWAM and IUSR accounts are created by the IIS product suite. Not so amusing was the report that webhits.dll allowed remote directory traversal, while the included output of the vulnerability check clearly indicated it did not. Also disconcerting was the report that our old version of wu-ftpd core dumped (Linux), even though the machine in question was a Windows host running IIS ftpd. Limiting false positives is something ISS obviously needs to address.

So while the scanner caught a good chunk of our holes and did a decent job of bringing them to our attention, a number of issues left us a little nervous. If nothing else, Internet Scanner reminded us that scanners are not always accurate. Companies that offer security assessments using scanners should be concerned: You face an interesting dilemma if you intend to endorse your results, as those results may be blatantly wrong.

Found 13.5 out of 17 vulnerabilities -- Internet Scanner, Internet Security Systems (ISS), (678) 443-6000; fax (678) 443-6477; www.iss.net.

Network Associates CyberCop Scanner

We found CyberCop to be a well-rounded scanner. Had it discovered more vulnerabilities accurately and not bombarded us with false positives, it would have scored higher. However, Network Associates appears to be putting more time into the product's 3-D effects than into ensuring reporting accuracy, and we have the paper stack to prove it.

CyberCop actually proved worse than most of the other scanners on our NetWare system, because the scanner incorrectly reported the existence of myriad vulnerable CGI scripts. But it did catch the easy-to-guess SNMP community string. CyberCop's main flaw seems to be in its overwhelming verbosity. For example, we didn't get just one alert indicating that CyberCop guessed our SNMP community string, we got eight: one saying CyberCop guessed it, and seven enumerating all the information the product found.

Even worse were the anonymous NetBIOS logons on Windows NT. While products like eEye's Retina actually use this data for discovering further information, CyberCop just gathers it and spews it at you. This could be useful in certain scenarios, but we found it superfluous in its current form. Another example is in regard to BIND (DNS) information (Linux). CyberCop makes a point of enumerating the version of BIND (in our case, 8.2) but completely missed the fact that it is vulnerable to the NXT root-level compromise.

We did, however, like how CyberCop consolidated port scan information into a single alert. One entry stating "you have these 20 ports open" is more manageable than 20 entries stating the same data. CyberCop also has some useful tools not found in other products, such as CASL, its packet-level scripting language, and the SMB Grinder, which can be used much as L0phtcrack can to crack NT passwords.

Found 12 out of 17 vulnerabilities -- CyberCop Scanner, PGP Security/Network Associates, (408) 988-3822; fax (408) 970-9727; www.pgp.com.

BindView Corp. HackerShield

Our initial impression of HackerShield was quite positive; then we saw the reports it generated. The information looks great in the GUI, but trying to get it into a more portable format results in a horrific mess. For an organization that scans a few hosts, this might not be much of a problem. Scan a few thousand, however, and you'll be as scared as we were. To BindView's credit, the reporting problem is a confirmed bug in the version we tested and is in the process of being corrected. Just be forewarned if you decide to take a peek at the online copies of the reports.

HackerShield proved fair in detecting the target vulnerabilities, but it wasn't without false positives. Many nonexistent CGI scripts were reported to exist on the Linux, NetWare and NT host, and it also indicated the IIS SMTP service was vulnerable to the "Sendmail pipe attack," which is obviously not possible on an IIS SMTP/ Windows-based system.

One nice feature of HackerShield is the attempt to consolidate the loads of resulting information. For example, we didn't get a large laundry list of all the sample scripts that were to be found on our NT system; instead, we received a general alert that vulnerable sample scripts were found. While this is quite refreshing, the detailed information concerning the exact offending CGI programs appears to have gotten lost in the chaos of the generated reports. Without that information, solving the problem at hand becomes a bit difficult.

We couldn't help but feel the "feast or famine" effect: We were either buried by boatloads of extraneous data coming out of CyberCop Scanner or hung out to dry by HackerShield's brevity.

Found 12 out of 17 vulnerabilities -- HackerShield, BindView Corp., (713) 561-4000, (800) 813-5869; fax (713) 561-1000; www.bindview.com.