Rolling Review: N-Stalker Seeks, Doesn't Find

In the third installment of this Rolling Review, we found that not all scanners are created equal.

August 31, 2007

9 Min Read
Network Computing logo

The range of products calling themselves "security scanners" is so broad that the designation is flirting with irrelevance. You have your vulnerability assessment software, which uses large databases of known vulnerabilities. Then there are penetration testing applications that focus on fewer vulnerabilities, but include the ability to exploit flaws instead of just identify them. More relevant to this Rolling Review are Web application scanners, which attempt to uncover problems in newly developed software—before they get exploited.

As an added twist in this review, we've focused our testing on Ajax applications. We've already evaluated Hewlett-Packard's WebInspect (formerly from SPI Dynamics) and Cenzic's Hailstorm. Both are Web application vulnerability scanners aimed primarily at crawling new Web apps looking for exploitable flaws. Sure, they're able to detect some common misconfigurations within Web servers and languages, even pick up a few stock bugs in known programs. But that's not their primary focus.

Unfortunately, the newest entry in this Rolling Review, N-Stalker's Web Application Security Scanner 2006 Enterprise Edition (say that five times fast), doesn't measure up to the previously tested scanners, despite its hefty built-in database of vulnerabilities in known Web servers and Web applications.With three different iterations of the product—the QA Edition; the Infra Edition, for infrastructure scanning; and the Enterprise Edition, which includes the QA and Infra versions as well as audit and penetration test capabilities—N-Stalker has a great conceptual approach that, on paper, made it look like an ideal fit for this review. We're looking for products that take into account the different potential use cases for application scanners, and on the face of it, N-Stalker's three-pronged approach is perfect.

N-Stalker ScreenshotClick to enlarge in another window

Unfortunately, while the QA and Infra offerings may be somewhat useful thanks to their large built-in vulnerability databases, the audit and penetration-test modes are plagued not only by poor detection capabilities for new vulnerabilities, but also a severe lack of tools to aid in advanced manual penetration.

In our evaluation, N-Stalker's scanner failed to find a number of vulnerabilities that all of the other products were able to identify. Additionally, the engine was too easily caught in unintentional scanning loops on one site that generated recursive links. Without recognizing the subsequent URLs as having repeated identical variables, the product was tripped up.

From a usability standpoint, N-Stalker's scanner not only fails to hit the bar set by WebInspect, it doesn't even compare well to the weaker interface found in Cenzic Hailstorm. Adding credentials for an application was a trivial matter with both WebInspect and Hailstorm, for example, but not only did N-Stalker fail to include any kind of automated log-in detection, even using the manual process was tedious, requiring at least twice the number of mouse clicks and keystrokes as rival products. Numerous other usability flaws and outright bugs abound: Multiple application windows that randomly failed to display in the Windows taskbar. Buttons silently failing to work. Having to guess a right-click is the next necessary step, non-resizable windows hiding necessary data, and more. N-Stalker says it is addressing at least some of these usability issues in its 2007 Edition release, due in October.

There were a few brief, shining moments where this product stood out, or at least broke even. As previously mentioned, it sports a large internal database of attack signatures for stock applications and would excel when primarily used to scan Web servers for known vulnerabilities. The reporting interface is flexible enough, and reports are attractive, though we would like to see XML output for further processing.

This article is the first of a series and is part of NWC's Rolling Review of Web Applications Scanners. Click on that link to go to the Rolling Reviews home page to read all the features and reviews now.

The product's lack of flexibility in scanning is made up for in part by access to a custom signature-writing language, which will appeal to power users. Additionally, an included log-analyzer utility takes advantage of that large internal database, analyzing Web server logs to detect a variety of malicious attacks. It can even be used to go back in time and find attacks that occurred before the application was scanned.

Of course, it can't distinguish between successful and unsuccessful assaults, so that function may or may not be useful, depending on how commonly the application is attacked.

Sadly, however, like the rest of the application, this function suffers from implementation flaws. When we scanned a site then immediately fed the resulting log into the log analyzer, it brought the analyzer to its knees. Extremely long URLs, part of the standard full pen-test scan on one of the applications tested, caused a huge increase in total processing time. Not only that, but even after all that work on a log chock full of attacks, only 121 requests out of over 30,000 log entries were flagged as suspicious. This is especially odd considering the only traffic in the log came from the scanner.

Of course, the fact that only six detections of XSS (cross-site-scripting) attempts were picked up might explain why the scanner failed to identify a live XSS in the application. To be fair, almost five hundred of the requests were HTTP post requests, so it's impossible to know what attacks might have been sent in those. Still, given the size of this particular application and the number of places to inject data, even if all 500 were XSS checks, the total number of checks was still not nearly enough to properly test the application for even standard XSS variants, let alone more complicated encodings and breakout techniques.Don't get us wrong—N-Stalker's idea has potential. If the log analyzer were integrated into the rest of the application and able to learn from the server profiling that the scanner is already doing, the data it produces could be potentially much more accurate and useful. And, it's inexpensive—$2,899 plus 20% maintenance per year. At that price, if you need log-scanner or infrastructure scanning using a large database of static vulnerabilities, and the bugs and quirks can be worked out of the system, this product might be a nice complement to another scanner better suited to finding unknown vulnerabilities.

As it is though, despite being called the Enterprise Edition, N-Stalker has its work cut out for it before we can recommend this scanner for enterprise use.

Continue Reading This Story...

RELATED LINKSRolling Review: Extrusion-Prevention SystemsAnalysis: SOA SecurityMore NWC Rolling Reviews


>THE INVITATION:Although there's no shortage of security consultants who'll audit an application by hand or using other companies' tools, we limited the scope of this Rolling Review--in part so we wouldn't still be rolling along years from now--to Web application scanning products and software-as-a-service models. As such, a few vendors may be included twice: Once for their standalone products, then again as their service offering does the heavy lifting.>THE PREMISE: NWC's Rolling Reviews present a comprehensive look at a hot technology category, beginning with market analysis and wrapping up with a synopsis of our findings. Our extended testing span lets us accommodate today's accelerated revisions cycle and focus our attention on individual products, while maintaining a consistent test bed. This installment focuses on Web application scanners. You'll find our impressions of SPI Dynamics' WebInspect in our May 28 issue, and Cenzic's AR in the June 11 issue.

>NEXT UP:IBM's (formerly Watchfire) AppScan

>PAST REVIEWS: Spi-Dynamics WebInspect, Cenzic Hailstorm

>OTHER VENDORS INVITED:Acunetix, Syhunt Technology, WhiteHat Security. Contact the author at [email protected] for consideration.

>THE TEST BED:We chose three applications from volunteer organizations to test our Web app scanners. All are relatively simple Web apps in use for real-world functions, and were built using a variety of development tools and platforms. Our first application was written in C# using Microsoft's with Ajax (also known as ATLAS) and deployed on IIS 6.0. The second was developed using the LAMP stack (the combination of Linux, Apache, MySQL and PHP), and the third was written in Java and deployed with JBoss and Tomcat on Linux. None of the applications has received a security audit, either at the source-code level or using external scanners. Throughout the process, all scanning applications will be leveled at the same applications--any changes to fix security vulnerabilities found in production systems will be left off test instances that are used for future scanning, to ensure that each product and service has the same potential vulnerabilities to find. Note that no vulnerabilities were intentionally added or seeded into an application. The applications will be scanned exactly as they existed in the wild at the start of the review. Each Web application scanning product will be evaluated for advanced features and flexibility for specialized security testing; ease of use for nonsecurity personnel; ability to map and scan Ajax functionality; prevalence of false positives, as well as ease in manual adjustments or product updates to address them; prevalence of false negatives; and price. Each SAAS offering will be evaluated on the same criteria, except for the first two items. At the end of our tests we'll show you how well each product and methodology did in identifying vulnerabilities in our sample apps.

Jordan Wiens is an NWC contributing technology editor and a network security engineer at the University of Florida, where he works on IDS/IPS, forensics, vulnerability assessment and system security. Write to him at [email protected].

Stay informed! Sign up to get expert advice and insight delivered direct to your inbox
More Insights