Network Computingwww.networkcomputing.com

A fantasy? Maybe not! We tested a raft of application monitoring systems in our Real-World Labs® at Syracuse University. Thanks to the cooperation and curiosity of the good folks in the Syracuse admissions office, we monitored some of their PeopleSoft applications, pummeling the products with traffic generated from an HR and admissions application. On the whole, we liked what we saw. We've never been in love with SNMP management platforms, and application monitoring seems to us a more useful approach.

Application monitoring solutions first appeared about a year ago and are quickly maturing. We tested 10 products that have never been compared before, assessing them from a network-centric point of view. We intentionally eliminated large, comprehensive frameworks such as Computer Associates International's Unicenter TNG Hewlett- Packard Co.'s HP OpenView and Tivoli Systems' TME-10. And we didn't explore development and tuning products such as RSW eLoad Suite Test Suite and Mercury Interactive, both of which monitor applications but report on performance from a code level.

Three invitees declined to participate. NetScout felt our tests wouldn't result in an apples-to-apples comparison, and Progressive Systems was between releases; Empirical Software closed its doors while our testing was under way. (Was it something we said?)

Our Editor's Choice award goes to International Network Services' VitalSuite, which came closest to fitting our idea of what an application-monitoring system should be. The components are well-conceived and easy to use, and INS paid attention to extensibility. Like that famous bunny, it just kept going, and gave us the best view of our applications' performance.

Close behind was NextPoint Networks NextPoint S3, with flexible, well-designed baselines and service-level management tracking. It is an excellent choice, especially if you want SNMP performance management as well. The combination is compelling.

As with any nascent software system, vendors have found more than one way to skin an application. We tested systems with active agents (they send out test transactions and measure the response) and passive ones (they watch the wire and track performance based on observation). Some products employ both. Passive agents can run on servers, desktops or as standalone boxes; active monitors are typically, but not exclusively, standalone.

Active monitors give you control over when and how data about your performance data is collected. If you flood your servers with transactions just to see where the breaking point comes, active probes can do the job nicely. However, you must rely on vendors or roll your own transactions. Further, you are limited to the protocols supported by the vendors. Some just do IP, for example, and that may not be sufficient.

The major advantage of passive probes, particularly those deployed on clients, is that they see exactly what the client sees. If performance falls off, the passive probe feels--and hopefully reports--the same pain that your user feels. Protocol issues can be diminished, too, but the probe still has to be able to classify the transactions it monitors by application. Also be aware that taking this approach to its logical conclusion--putting agents on all your clients--can get expensive.

Active or passive, it's monitoring your applications that counts. We found that the best solutions not only handle a wide range of known TCP and UDP ports for IETF standards and applications such as Exchange, Oracle and PeopleSoft, but also allow for user-defined transactions beyond simple port definitions. For our money, the more granularity and flexibility the better. Many of these solutions are good, but have room for improvement.

No product we tested could monitor a transaction winding its way through a multitiered server environment. Granted, we didn't expect to see transactions tracked between Web server, Java servers, load-balancing servers, native client-gateway servers, application servers and the final back-end database servers, but it would have been a nice surprise. All the vendors acknowledge this shortcoming. The holy grail seems to be an industry standard that establishes unique transaction IDs that would follow through an array of servers. It's a nice idea, but we won't hold our breath waiting for it. Following are our evaluations of the 10 products we tested.