Rolling Review: N-Stalker Seeks, Doesn't Find

The range of products calling themselves "security scanners" is so broad that the designation is flirting with irrelevance. You have your vulnerability assessment software, which uses large databases of known vulnerabilities. Then there are penetration testing applications that focus on fewer vulnerabilities, but include the ability to exploit flaws instead of just identify them. More relevant to this Rolling Review are Web application scanners, which attempt to uncover problems in newly developed software—before they get exploited.

As an added twist in this review, we've focused our testing on Ajax applications. We've already evaluated Hewlett-Packard's WebInspect (formerly from SPI Dynamics) and Cenzic's Hailstorm. Both are Web application vulnerability scanners aimed primarily at crawling new Web apps looking for exploitable flaws. Sure, they're able to detect some common misconfigurations within Web servers and languages, even pick up a few stock bugs in known programs. But that's not their primary focus.

Unfortunately, the newest entry in this Rolling Review, N-Stalker's Web Application Security Scanner 2006 Enterprise Edition (say that five times fast), doesn't measure up to the previously tested scanners, despite its hefty built-in database of vulnerabilities in known Web servers and Web applications.With three different iterations of the product—the QA Edition; the Infra Edition, for infrastructure scanning; and the Enterprise Edition, which includes the QA and Infra versions as well as audit and penetration test capabilities—N-Stalker has a great conceptual approach that, on paper, made it look like an ideal fit for this review. We're looking for products that take into account the different potential use cases for application scanners, and on the face of it, N-Stalker's three-pronged approach is perfect.

Unfortunately, while the QA and Infra offerings may be somewhat useful thanks to their large built-in vulnerability databases, the audit and penetration-test modes are plagued not only by poor detection capabilities for new vulnerabilities, but also a severe lack of tools to aid in advanced manual penetration.

In our evaluation, N-Stalker's scanner failed to find a number of vulnerabilities that all of the other products were able to identify. Additionally, the engine was too easily caught in unintentional scanning loops on one site that generated recursive links. Without recognizing the subsequent URLs as having repeated identical variables, the product was tripped up.

From a usability standpoint, N-Stalker's scanner not only fails to hit the bar set by WebInspect, it doesn't even compare well to the weaker interface found in Cenzic Hailstorm. Adding credentials for an application was a trivial matter with both WebInspect and Hailstorm, for example, but not only did N-Stalker fail to include any kind of automated log-in detection, even using the manual process was tedious, requiring at least twice the number of mouse clicks and keystrokes as rival products. Numerous other usability flaws and outright bugs abound: Multiple application windows that randomly failed to display in the Windows taskbar. Buttons silently failing to work. Having to guess a right-click is the next necessary step, non-resizable windows hiding necessary data, and more. N-Stalker says it is addressing at least some of these usability issues in its 2007 Edition release, due in October.

There were a few brief, shining moments where this product stood out, or at least broke even. As previously mentioned, it sports a large internal database of attack signatures for stock applications and would excel when primarily used to scan Web servers for known vulnerabilities. The reporting interface is flexible enough, and reports are attractive, though we would like to see XML output for further processing.

The product's lack of flexibility in scanning is made up for in part by access to a custom signature-writing language, which will appeal to power users. Additionally, an included log-analyzer utility takes advantage of that large internal database, analyzing Web server logs to detect a variety of malicious attacks. It can even be used to go back in time and find attacks that occurred before the application was scanned.

Of course, it can't distinguish between successful and unsuccessful assaults, so that function may or may not be useful, depending on how commonly the application is attacked.

Sadly, however, like the rest of the application, this function suffers from implementation flaws. When we scanned a site then immediately fed the resulting log into the log analyzer, it brought the analyzer to its knees. Extremely long URLs, part of the standard full pen-test scan on one of the applications tested, caused a huge increase in total processing time. Not only that, but even after all that work on a log chock full of attacks, only 121 requests out of over 30,000 log entries were flagged as suspicious. This is especially odd considering the only traffic in the log came from the scanner.

Of course, the fact that only six detections of XSS (cross-site-scripting) attempts were picked up might explain why the scanner failed to identify a live XSS in the application. To be fair, almost five hundred of the requests were HTTP post requests, so it's impossible to know what attacks might have been sent in those. Still, given the size of this particular application and the number of places to inject data, even if all 500 were XSS checks, the total number of checks was still not nearly enough to properly test the application for even standard XSS variants, let alone more complicated encodings and breakout techniques.Don't get us wrong—N-Stalker's idea has potential. If the log analyzer were integrated into the rest of the application and able to learn from the server profiling that the scanner is already doing, the data it produces could be potentially much more accurate and useful. And, it's inexpensive—$2,899 plus 20% maintenance per year. At that price, if you need log-scanner or infrastructure scanning using a large database of static vulnerabilities, and the bugs and quirks can be worked out of the system, this product might be a nice complement to another scanner better suited to finding unknown vulnerabilities.

As it is though, despite being called the Enterprise Edition, N-Stalker has its work cut out for it before we can recommend this scanner for enterprise use.

Continue Reading This Story...

Jordan Wiens is an NWC contributing technology editor and a network security engineer at the University of Florida, where he works on IDS/IPS, forensics, vulnerability assessment and system security. Write to him at [email protected].