Network Computingwww.networkcomputing.com

Overall, network-performance management entails polling statistics from key infrastructure points. Its purpose is not to produce an interactive real-time display for immediate diagnostics. Rather, the information is held in a database and processed to reveal baselines, trends and threshold violations, which can be presented as high-level overviews for business and application management; overall network-performance reports for IT management; and detailed network performance statistics for diagnosis and planning.

In fact, network performance management products' raison d'etre is reporting, and they all do a decent job of composing useful reports. The ability to deliver and control access to the reports and the administrative workload required in maintaining report delivery distinguish one product from the next.

Performance-management products don't try to tackle all tasks. They typically don't perform exception reporting or configuration management, nor do they provide event correlation or root-cause determination. Some products fire off an alarm on an exception, but they are not trap-collection points. Configuration also is not part of the tool set. SNMP sets are possible, but none are going to manage Cisco Systems IOS (Internetwork Operating System) images, or report VLAN (virtual LAN) configuration. These areas remain the sovereign domain of infrastructure vendors.

Compared with traditional network frameworks, which try to track every device and interface on the network, these products provide salient information by polling key devices.

The data collected comes primarily from MIB2, RMON, RMON 2 and proprietary MIBs. But beyond just collecting data, the products add value in two ways. First, they provide reports that meet the specific needs of different users, departments, divisions and customers. Second, they sort the reports based on criteria, such as type of device and application.

All these products produce high-level, nongranular reports that use a derived compound metric to indicate, at a glance, how well the network is doing. Although such reports are slightly more sophisticated than happy- or sad-face gauges, their means of boiling down statistics carries the risk that the intended audience may misinterpret a negative indicator as proof of network failure. For this reason, you should require the vendor to explain the details that make up these quality indicators, and make sure you understand them.

The creation of high-level happy-face reports does not let network engineers off the hook, either. Although the products make it easier to begin collecting, archiving and rolling up network usage data, accurate interpretation still requires comprehension of SNMP and vendor data sources. MIB2 walks aren't required, but you still need to understand the collected data.

None of these products is integrated into Novell NDS or Microsoft Active Directory. We'd like to see this happen for storage of device baselines.

NextPoint: Next to Perfect In Network Computing's Real-World Labs® at Syracuse University, we tested four network performance products: Agilent's NetMetrix/UX 6.02, Concord's Network Health 4.5, Lucent's VitalSuite/VitalNet 7.0, and NextPoint's NextPoint S3 2.5. We invited DeskTalk Systems to send its Trend product, but the vendor declined, citing the timing of our tests as inconvenient. We also talked with 3Com Corp. about testing Traffix, but because Traffix manages traffic at Layer 3 and above, it would have been unable to handle our MIB2 test environment.

All four products performed well and provided insight into who and what was using the network. All are suites with add-on modules that sometimes extended beyond the scope of our tests--most commonly including application monitoring, which we tested last summer (see "Application Monitoring Grows Up," at www.networkcomputing.com/1016/1016f1.html). This time we focused only on network-infrastructure performance.

NextPoint S3 received our Editor's Choice award for its strong blend of network usage and diagnostics statistics, and logical, almost all-Java administration tools. NextPoint S3 offers high-level management views that show actual network usage compared with service-level agreements, and useful diagnostic correlation of collected statistics. Its comparatively high price is misleading, as it includes all application- and network-management modules.

VitalNet came in a close second, with excellent all-Web administration. It was only hurt by some inconsistent data-collection performance. Network Health, the old kid on the block, provided solid reporting, but trailed in ease of administration. NetMetrix is a suite of RMON applications that provides superior diagnostics, but requires Agilent probes to track network usage fully.

How We Tested
We used two data sources for our tests. In our live test environment, our lab connected to the Syracuse University network of more than 10,000 devices and to our lab frame relay network. We pointed all four products at key university backbone, Internet and Internet II segments, where traffic is heavy. We also included the frame relay mesh that connects our labs, to collect SNMP data statistics.

Our second test environment was based on a mirror image of the real network, but controlled by us. We used Gambit Communications' Mimic, which crawls the network doing controlled MIB walks on SNMP-capable devices, records the results and then runs a simulation that alters the MIB variables in a deterministic way via TCL scripts. The simulation looked like a routed interface to the network performance products under test, and let us vary the MIB values in predictable and radical ways.

With these two rich SNMP data sources, we rated each product based on the information it provided about how the network was doing, as well as the product's reliability during our testing, its administration, ease of use and purchase price.

In our report card, we placed the heaviest value on the amount of information provided because this is the software's primary value. All produced the same basic information about baselines, trends and service levels. Events and alarms, which typically have not been part of these products, were also considered part of the network-performance information, but at a lower weight. We subdivided reporting into the categories of network usage, baseline, trending, service monitoring and diagnostic information.

Reliability, the second report card category, was based on our experience with the data's accuracy and consistency, and the server. Inconsistencies or inaccurate data were deemed unacceptable. Likewise, server failure or reduced availability was also noted.

Our administrative category marked the ability to control and understand each product. We considered the discovery, inventory, polling control, date and time exclusions for services, remote administration and the grouping of devices, users and the application to the performance data being created.

Ease of use, often a catch-all category, has specific meaning in our tests. Users will make judgments about the network based on the method and presentation of the data delivered, so "ease of use" included the intuitive nature of the GUI and the format of the data in terms of content and presentation. We couldn't overlook the speed of the GUI, as it contributes to user frustration. We also considered the flexibility users have to customize their presentation. We didn't forget admin- istrators and threw in a score to reflect our experience administering each product. Last, we graded cost based on the vendors' approximate retail prices for 1,000-node and 5,000-node scenarios. Also included in the pricing grade was the warranty and maintenance.