Performance Management Software

System-performance monitoring is not a singular activity, even if you have only one computer system. It involves the operating system and multiple applications and how they affect valuable IT resources like processors, memory, storage and the network. To understand how systems affect these resources, get a grip on throughput, queue and response time. These terms describe system-resource usage from the inside out.

High thresholds and long queues translate to reduced workloads for your technical people. But this information may not impress the company execs. Being end users themselves, they tend to focus instead on slow response times--a problem that can reduce sales. It's no wonder, then, that vendors continue to offer products that measure performance-response time from the outside in ("Monitoring System Performance From the Outside In").

Measuring the Managers

We looked for system-performance managers that could monitor and manage our NWC Inc. 24/7 production environment in Green Bay, Wis., from our Real-World Labs® at Syracuse University. The original design for this review called for managers that could remotely monitor Windows 2000 and Linux Red Hat servers. On the Windows side, we required support for IBM WebSphere 4.0.1., Microsoft IIS 5.0 (.Net) and Microsoft SQL Server 2000. Under Linux, we required support for Apache 1.3.22 and MySQL 3.23.41.

We sent invitations to nine vendors. BMC Software, Heroix Corp., Hewlett-Packard, NetIQ and Quest Software responded enthusiastically and planned to visit Syracuse, as their business models dictated installation support for enterprise customers. But Candle, Computer Associates, IBM (Tivoli) and Micromuse opted out because they were planning to release new products during the course of our testing.And to our surprise, our Green Bay lab also opted out, because of technical difficulties with our ISP. This shifted the entire weight of the review to our Syracuse University lab in midstream. Our participants took it in stride, but we had to adjust our original requirements (see "How We Tested System-Performance Management Products").

There's something for everyone among the products we tested. Looking at the prices, you would think that HP drove up in a Cadillac, with BMC, NetIQ and Quest following in Volvos and Heroix trailing in a VW van. But each one of these vendors can supply you with products to help manage your enterprise systems and get optimal performance out of your network. Your budget and needs in light of the features and functions of these products will dictate your solution.

For us, Quest's Foglight 4 edged out HP OpenView for Windows (with performance management) to win our Editor's Choice award. Had price been no object, HP would have taken the top spot.

Both Foglight and HP OpenView provided consistent system-performance management across our test bed. Foglight delivered both Windows and Linux agents to servers remotely, and had better out-of-the-box reporting than HP OpenView. Although OpenView gathered and used more performance metrics than other participants, it lacked an automatic installer for Linux, and without HP Reporter, it couldn't configure reports and export the data to alternative formats like CSV or PDF files.

With the exception of Heroix's eQ, each product we tested supports distributed administration to set up multiple roles for monitoring. And all the products identified the resource problems we created in the test bed.And no one came up short on metrics or rules for managed objects during operations testing. But we also judged on application (agent installation, configuration) and delivery (reporting). Quest's Foglight won the day with a consistent all-around performance in day-to-day operations, configuration and reporting.

NetIQ's AppManager followed HP with solid systems performance, offering strong event and fault management at a good price. BMC's Perform and Perceive suffered from its lack of message management and automatic actions for the wealth of events and alerts it can generate. Although BMC added Patrol Central to its submission, that product focused on the vendor's WebSphere Knowledge Module and was not used as a console for performance data.

We hadn't planned on judging the Web consoles available for system-performance management, but we couldn't help ourselves. In general, the Web consoles are offered as an alternative to Microsoft Management Console (MMC) or Windows to view critical messages and get an overview of system performance. These basics were available from HP, Heroix and NetIQ, which all leveraged IIS. BMC raised our eyebrows with its installation of Apache Tomcat, which sported configurable report generation and drill-down reporting. But Foglight (Apache) took the honor with the best Web site to view messages and reports.Quest Software Foglight 4, Foglight WebSphere Cartridge, Spotlight on Windows/Unix

Quest's Foglight is a full-featured event-reporting and management application with system- and application-performance monitoring. It came close to matching HP OpenView's fault and message management, but did not win out in system-performance monitoring and setting alerts. Nevertheless, Foglight merited our Editor's Choice award. Its pricing was good, and we were pleased that it distributed both Windows and Linux agents from the central console. And like BMC's offering, Foglight included configurable report generation. It also provided the best Web interface for monitoring and management.

Unlike Heroix's and NetIQ's products, Foglight didn't require MS SQL 2000 during installation. Instead, it relied on a proprietary database. But Foglight's Operations Console called for a Java Runtime Environment (SE 1.4.2_01).The Java console was snappy, letting us draw a map of our environment by IP address. It could also draw custom maps to identify computers by cities, subnets and hosts. We stuck with what we knew best: IP maps.

We remotely installed agents for both Windows and Unix computers from the Foglight central console. An install wizard walked us through a discovery of computers on the network; it allowed input for authentication information, target directory for the client and the port number to use (5150 default). Following configuration, an installation summary opened up to give us detailed information about the connection and the installation.

Foglight's event management is on par with OpenView's. The operations console can act as the central repository for all events affecting systems performance, which the IP map lets you view in a top-down fashion. All events for the network can be viewed by highlighting the network; events by subnet can be displayed by highlighting the subnet object.

Highlighting a subnet will give you the events for all the computers on the subnet. The same holds true when you click on a subnet object. But you can't create filters or views to limit the number of messages you receive, or collapse duplicate messages into message threads--tasks you can perform with OpenView and NetIQ.

The famous right-click in the MMC console continues with Foglight. Using this method, we could edit the message to take an automated action. The text of the message is much less busy than with OpenView.With Foglight, we could send an e-mail or issue an agent command to start, restart or stop a service. We could also issue a remote exec command or send an SNMP trap.

Diagnosing an event in Foglight brings up Spotlight on Windows or Unix. This feature gets our "cool tool" award hands down. Spotlight uses metrics, thresholds and severity levels to alert the viewer in a Java-based GUI. It gets its information from Windows performance counters and the registry, so we needed to authenticate to a system with administrator privileges. For Linux, it uses ifconfig, df and iostat to gather data. Metric properties such as thresholds and alerts can be changed with the Metric Editor.

By default, the display quickly alerted us to a bottleneck. Where we saw red, we could drill down into the display to view problem resources that were flagged.

The Foglight Cartridge for WebSphere had the most intrusive installation setup of the products tested. It went beyond simply installing a management object on the WebSphere 5 server.

Quest monitoring code is run in each JVM (Java Virtual Machine), so it needs to be installed as an application in WebSphere. The monitoring code communicates with the operations console via HTTP. This information, along with WebSphere PMI (Performance Monitoring Infrastructure), is obtained and inserted directly into the tables and views available at the operations console. More than 40 metrics, from Java Bean usage to Garbage Collection data, are provided. In addition, the Cartridge is used by Quest's PerformaSure product. Although we didn't test PerformaSure, we installed it to find an in-depth application performance monitoring solution with response-time metrics.From the Foglight perspective, we installed agents to obtain information from managed objects. Like the other products we tested, Foglight has agents to support NT and Linux. There are 24 views or standard reports for NT and 45 for Linux systems. These default views are extensive but not as deep as OpenView's. But Foglight's out-of-the-box reporting is sweet.

Double-clicking on a view produces a graphical chart that displays metrics for such items as processors memory and disk. We could change the chart to various types (bar graph, line or pie chart) and set the display to various time increments (real time, last hour, last four hours and so forth). By double-clicking on the graph, we could view the underlying data. We could also create a report with a print preview of the graphics and data.

We could send the output to a printer or export it to HTML or PDF. There was even an option to e-mail the report using the default SMTP mailer we had configured during installation.

Like OpenView and NetIQ, Foglight sports an Apache agent for Unix out of the box. To get started, we had to edit the agent start-up parameters and change the default path to /usr/local/apache. But once these tasks were done, the Apache agent could report metrics ranging from server availability to hit rates, transactions and throughput.

Foglight 4, Foglight WebSphere Cartridge, Spotlight on Windows/Unix. Quest Software, (800) 306-9329. www.quest.comHewlett-Packard OpenView for Windows, bundled with Performance Manager 7.2, HP OpenView Performance Manager 4.4/Agent C.03.65.00, SPIs (Smart Plug-Ins) for WebSphere 4xAlthough HP came to the lab in a Cadillac toting some big guns, Quest is the one that rode off into the sunset with Editor's Choice. Still, if you like driving the best technology for system-performance management, you'll want to give HP OpenView for Windows a test drive. We installed it on our console server, and it was up and running in no time.

Like the other products we tested, OpenView can deploy Windows agents from the console automatically. But OpenView also performs automatic application and service discovery, and deploys the respective policies (rules) to remote nodes for monitoring and management. In addition, its SPI technology for mission-critical applications like Web and application servers proved faster and easier to install than any of the other packages we tested. But its price put OpenView in second place.

The basic OpenView Agent (coda) version A.07.20 includes more than 120 metrics, though not on every platform. Approximately 30 metrics are collected on all platforms. This covers the basics about a system's global configuration, CPU, file system, swap/paging and memory use.

In addition to coda, we used Performance Agent C.03.65.00 (formerly MeasureWare Agent), which added a number of metrics beyond the basics for system resources. For example, NetIQ's metrics for Windows CPUs included data on load, resources and processes, but nothing on interrupts. HP's Performance Agent includes a CPU idle time and interrupt time, along with user mode and system mode time. There are up to 13 global metrics and 25 specific metrics for CPU resources alone. And getting these big guns on our systems was no problem.

Installing the management server (read: data repository) and console was no different from our experience with the other products--namely, uneventful. OpenView for Windows uses an MS SQL Server 2000 (SP3) database, and supports only Windows 2000 Server or Advanced Server.By the time we figured out where to add a node OpenView had discovered most of the Windows computers in our lab, using DNS, the Registry, SNMP and WMI (Windows Management Instrumentation).

The MMC console is laid out in two windows. The left window depicts the managed environment in a hierarchical arrangement. The right window provides the views. Services, Nodes, Tools, Policy Management, and Reports and Graphs form the core areas of the console. The Services object groups applications and services in a logical area for configuration and review. The Nodes section groups computers together. Right-clicking on Nodes provides a configuration option to add computers. A dialogue box opens up to select Windows nodes for management. For Linux or Unix systems, we manually installed the agents, which then appeared in the console as computers eligible for management.

As we did with the products from Heroix, NetIQ and Quest, we deployed Windows agents from the operator's console. But we needed Domain Administrator status to get the job accomplished. Using DCE services (RPC Port 135), HPOV can push-install the Windows agent and automatically discover applications and services.

The Performance Manager agent is installed separately. It adds Microsoft runtime libraries and graphing components to get the real skinny on performance.

The Windows agents don't provide any GUI tools to issue commands on the managed node such as "collect data," "look" or "stop agent." But there are plenty of command-line options to start and stop the agent and conduct diagnostics. No doubt, this is from HP's background with the Unix command line. The Performance Agent does provide plenty of GUI utilities to get status, start and stop the agent, and configure data collection.Once we had the clients installed, we returned to the console. Then the fun began with configuring policies and creating custom commands for automatic actions like restarting servers or services, or sending alerts based on performance events. After data was collected, we could review standard reports and graphs easily.

The standard graphs gave us drill-down tables to select nodes, date ranges and time increments. We also could generate them in Java or HTML. With Java Plug-in 1.3.1, we got an interactive display to view the data in pie charts, tables, or area, bar or line graphs. But some views, including gauges and baselines, are available only in HTML.

With the Java graphs, we could reorder columns, drill down to get graphical views of smaller data increments, zoom in and out, and rotate 3-D images. We could also refresh the data while viewing a graph; when we selected an option to do this periodically, the data trickled to the top of the graphs section in the MMC for easy access.

In addition to the graph data, the standard reports bundled with OpenView left us wondering just what data views we would need to construct on our own. We could easily view top applications, CPU, disk, file-space utilization, memory and network reports across the enterprise in HTML. But neither graphs nor reports could be exported to other data formats like PDF, XLS or text files. And standard reports were not configurable. HP leaves that up to an add-on product: OpenView Reporter.

The easiest place to start with policy management and configuration of the environment being monitored is to run the standard policies for a period of time to see how default thresholds fail, meet or exceed your needs. For example, we found the Web server default thresholds for availability too low and easily changed them using a GUI display. We could also drill down and configure the policy's logic with VB Script or PERL skills, as in NetIQ. We could change threshold levels and configure automated actions, or an action for an operator to execute when he or she sees an event message triggered by the policy.As we found in NetIQ, we could edit the event policy. For example, we could set event messages to appear for each instance separately; or we could let the console process all instances once.

OpenView's SPIs for operating systems and applications were easy to deploy. For Unix nodes, the Web server SPI to discover Apache is run automatically after an agent is installed and added to the managed nodes in the console. Once Apache is found, SPIs are added to the computer to view the error log and make sure the server is running.

The SPIs provided IBM WebSphere data without requiring us to install applications on remote servers manually, as we had to do with the products from BMC, NetIQ and Quest. But this is not a fair comparison: OpenView supported WebSphere 4x during this review, whereas the other products supported 5.x.

HP OpenView for Windows, bundled with Performance Manager 7.2, HP OpenView Performance Manager 4.4/Agent C.03.65.00, SPIs for WebSphere 4x. Hewlett-Packard, (877) 686-9637. www.hp.com.NetIQ AppManager Suite 5.0.1 (SP1), WebSphere Knowledge ScriptNetIQ's AppManager Suite is a mature solution for monitoring and managing system performance. Our testing showed that it covers the basics and beyond in viewing and reporting how applications and services use system resources. But the product didn't measure up to the metrics available from BMC or HP.

AppManager does make it easy to configure and schedule recurring jobs with knowledge scripts to obtain and analyze data. It also provides scripts to ensure that systems are available. Standard and configurable reports come out of the box, as with BMC and Quest.Similar to what we found with Heroix, the only difficult thing about installing NetIQ AppManager on Windows was providing a copy of MS SQL Server 2000 to use as a data repository. The minimum management-server requirements are Pentium III 733+ MHz, 512 MB of RAM and 200+ MB of disk space, which our Windows 2000 servers easily met (see "How We Tested," page 70). Because we had the firepower on one server, we also ran the management server (NetIQms Service) that facilitates communication between agents and the repository and the Operator's Console. AppManager also uses Microsoft IIS 5.0 to support its Operator Web console.

NetIQ gave us the most choices in running an operator's console. Out of the box, we got a snap-in for MMC, a Web console and a Windows executable. We stayed with Windows. Once up, the AppManager's console was a visual relief from the usual run of MMC consoles in this review.

The console defaults showed all managed computers in out test environment in a hierarchical view flowing from a "Master" object. With a right-click on the Master object, we added computers to the managed list, as in HP OpenView. But that's where the comparison stops. OpenView discovered computers in the lab and let us select from a list, but AppManager Suite didn't. Although we could enter a Windows or Unix computer to add to our managed systems, we couldn't browse the network and select a computer. If the computer entered was unavailable, an error box opened up to inform us that either AppManager hadn't added the computer or the computer was down.

Knowledge scripts are at the heart of AppManager's performance-management strategy. They're associated with resource objects that gather performance data on managed systems (read: systems with agents). Scripts can be dragged and dropped onto managed computers or services in the operator's console. This creates a job on the destination node. Jobs can be configured to run on a schedule, with custom use depending on the task. For example, we set parameters for monitoring CPU load and indicated maximum values for high CPU utilization and queue length. (The defaults are 90 percent utilization and two processes in the run queue.) We then defined an action to take from a pull-down window. Actions range from sending mail or an SNMP trap to restarting a service or rebooting the system. The object or target for the script can be modified as well. Finally, there is an advance tab worth investigating.

The advanced tab in configuring a job deals with event options and data collection. We collapsed duplicate events into a single event. This makes for a very readable event log in a busy environment. Caveat: Changes to a knowledge script apply to everyone using the same repository, which is why AppManager provides version control. The script's metadata keeps track of the last editor and tacks on a version number. But don't mistake this for source control. There's no option to return to the previous version, or to show the differences between versions. You'll have to go back to your last backup or the installation CD for the source.Monitoring specific applications in AppManager can be as easy as applying a knowledge script to a resource or as complex as installing software on the console and the remote machine. On the complex side, we installed AppManager for WebSphere Application Server on the management console. This "checked in" new knowledge scripts associated with WebSphere. On the server running WebSphere, we used the same installer to put the WebSphere managed objects in place. Before leaving the WebSphere server, we deployed the perfservlet application included in the installable applications directory. (For the console server to discover WebSphere resources, the servlet had to be running and available via HTTP.) Then, from the Management console, we applied the WebSphereAppSrv Knowledge Script to discover WebSphere App Server resources. Voil--data.

AppManager includes a copy of Chart Console, a Windows-based application that plots and charts data from the repository. As with those in the BMC and HP products, charts can be rotated and data points moused over for details. We could drill down to specifics from the chart itself by clicking on the data point. For example, a bar chart showing CPU processes over time could be drilled down to specific days, hours and minutes. We could easily export data to HTML, XML or a text file.

AppManager Suite 5.0.1 (SP1), WebSphere Knowledge Script. NetIQ, (888) 323-6768. www.netiq.comBMC Software Patrol Perform and Perceive 7.1.20, Patrol Central Console 7.1.1.0, Patrol WebSphere KM (Knowledge Module) 2.1.01BMC submitted part of its Patrol Performance Assurance Platform: Perform and Perceive. These products went to the heart of the review to provide a consistent environment to monitor and manage system performance. With the addition of Patrol's Central Console and the WebSphere KM to meet our requirements for the review, BMC gets our Best Value award.

Perform and Perceive are squarely focused on system-performance management, with policy and scheduling controls for monitoring a rich set of metrics on Windows and Linux at a modest price. Like our Editor's Choice winner, it could configure reports out of the box in HTML and export them to other data formats like PDF for high visibility by technical and nontechnical personnel.

As with our other participants, BMC's business model called for on-site installation support. With the help of a software engineer, the Perform and Perceive solution was up and running in short order.The Patrol Performance Console is an MMC snap-in module. It didn't need MS SQL Server to start with this package. Using Microsoft Access Data Components got us by for this review. But for serious data collection, you'll need MS SQL Server 7 or 2000. BMC can also use Oracle 8.1.7 and above.

Agents run a collection process for data (bgscollect) on monitored computers and send the results to the Patrol Performance Manager on a central console. The Performance Manager investigates and analyzes the data from agents and saves it to the data repository. We could investigate performance data in the MMC or Web console. We could also get a real-time or historic report in the MMC or Web console using Visualizer, which BMC includes with Perform and Perceive at no extra cost. And we could export data to XLS, XML, PDF and text formats for other uses. But data visualization is also available from the Web console.

The Patrol Perceive Web console caught our attention when we saw the default installation put a copy of Tomcat on our central console. End users can select system performance from standard metrics for processors, such as utilization by user and process and the CPU's run queue length. Standard metrics are available for other resources as well, like memory, network and disks.

As an administrator, we could also create views for individuals or groups. All views could be configured within set time frames--even data within the last hour.

Setting up event or fault notification for system-performance problems is not as automatic as it is with OpenView and not as intuitive as it is with Foglight. In the MMC console, we set up policy groups to configure alerts that apply to all the computers in the group. We could also set alerts for each computer by right-clicking on the policy or the computer. A wizard opened up to select individual metrics grouped by platform (Windows or Unix, for example). Within metric groups, we could select specific metrics by name, as well as an application instance or the name of the resource, such as the default instance (no Instance). We could also choose a resource from a list of alternatives, including eth1 and CPU-1. This was comparable to using Performance Monitor in Windows 2000.Upon selecting the metric and instance, we set a condition on the metric. For example, we sent an alert to the console when the selected resource exceeded 95 percent CPU utilization.

After completing the alert, we needed to save it to the policy group and then activitate it. That's a lot of steps compared with our other participants. And there's no provision to add an automated action when the alert condition is triggered; that's left to the Patrol Central.

Scripts are used to manage automation in Patrol for Performance. They can also be used to generate automatic reports on the Web.

A script wizard let us define the source of data, the type of output to produce, and the time frame to produce and run it; no VB Script or PERL knowledge was required. This was short work for a long gain: It saved us time and energy in reporting data, and made it convenient to share. But the ease with which Patrol's Perform/Perceive solution applied to our review stopped when we went to install Patrol WebSphere KM 2.1.01.

KM for WebSphere required a Patrol Central console in addition to the Perform Management console. This, in turn, required a second software consultant from the Patrol line of products who specialized in WebSphere.Once the Patrol Central console was installed to the same server running the Perform/Perceive, we installed the managed objects manually on the WebSphere application server and the central console. Next, we loaded the WebSphere KM into the Patrol Console and registered the WebSphere administration account.

Patrol Console could now receive performance data from our WebSphere server. As with the products from HP, NetIQ and Quest, we could monitor and manage all the usual objects, such as server runtime resources, database connection pools, EJBs (Enterprise Java Beans) and methods.

With the addition of the Patrol Central Console, BMC provides distributed administration and lets you create and manage roles in monitoring and managing system resources. Also, event management and resource availability become much easier to configure and administer. If you plan on accelerating IT growth or consolidating resources, add Predict to your purchase. With Predict, we could analyze different system resource configurations to satisfy a theoretical demand.

Patrol and Perceive 7.1.20, Patrol Central Console 7.1.1.0, Patrol WebSphere KM 2.1.01. BMC Software, (800) 841-2031. www.bmc.comHeroix eQ 2.1.26Heroix trailed the herd of system-performance managers. But it's not all bad. If your enterprise is on a tight budget and you need something beyond Windows Performance Monitor, take a look at eQ. It builds a good set of rules with performance metrics and contains standard reporting. You can even export the raw data to other formats. But you have to get accustomed to a task-based interface that can be confusing, as well as unearth a few hidden features.

After setting up the required data repository (MS SQL 2000), we installed Heroix using a recommended Domain Administrator account to ensure permissions to write files, create services on the local host and remotely install agents. A local administrator will do if you don't plan to install clients from the central console.Out of the box, eQ supplies Windows and Unix agents, as well as a Windows and Web management console. A copy of Solution Studio, a graphical tool to modify managed objects for systems and applications is also included. This isn't a big plus. The other vendors in our review include code with an editor built in. Note that the installation also installs a user (heroix) in the administrators group to run the eQ services. But to remotely install agents, the services need to run by a Domain Administrator, just like AppManager.

The eQ management console is a Windows executable with a one-click feature to navigate. It didn't matter whether we right- or left- clicked--either brought us to the next step. And that step was to install agents on test servers.

The main window prominently featured "Remoteinstall." We started there to select a path and source folderto the installation files placed on the server duringinstallation. Then we opened that directory as a Windowsshare so other computers could install the client.Selecting that directory moved us to the next screento identify a computer by name. We entered the hostname and received a report that the target machine wasa Windows 2000 Server (SP4) with IE 6, MSI and MDAC(Microsoft Data Access Components) installed. Justbefore executing the installation, we noticed a handycheck box to execute the task in the background.As with Foglight, both the Windows and Linuxagents installed remotely. The Linux agent required anextra step following installation. Because the license forUnix computers is not attached automatically, we hadto add the license manually. If you have a lot of Unixmachines in your enterprise, this may not be the besttool to get you up and running quickly.

EQ has almost 50 rules to apply to Windows servers.On the Linux side, there are close to 100 rules, rangingfrom a count of core files on disk to Zombie Processes.With one click, we could edit the rule properties tofind an extensive graphic calendar. Using the calendar,we selected the interval for the rule and its effectivedays. This strategy allowed for planned downtimes,comparable to the blackout times in AppManager andFoglight scheduling.

The same screen gave us a way to create actions. Thedefault action, not unlike that of all the other productstested, is to generate an event. We could add otheractions, like sending an e-mail or issuing an SNMP alertor command-line instruction. No sophisticated scriptingto master here.Reporting is facilitated by Crystal Reports. eQ providessome above-average features to select and displaymultiple types of charts and export the data to the Web.And we could drill down into charts and spreadsheetsfor data in smaller time increments.Bottom line: Heroix EQ is a good entry-level product.But look to our other participants for a more fullfeaturedperformance manager.eQ 2.1.26. Heroix Corp., (800) 229-6500.www.heroix.comWe had asked the participants for this review to provide performance data for systems running in our NWC Inc. business-applications lab in Green Bay, Wis. We requested that the system-performance managers support our Windows 2000 servers running Microsoft IIS, MS SQL, Server 2000 and IBM WebSphere 4.0.1. In addition, we required support for Red Hat Linux 7.3 server running Apache Web Server 1.3.22 and MySQL 3.23.41. However, because of technical difficulties, we had to keep the tests solely in our Real-World Labs® at Syracuse University.

Because of this change, we relaxed some requirements and tightened others. Once Green Bay was out of the picture, we decided to monitor only those business applications with which we were familiar--there are too many "gotchas" in this business to work with unfamiliar applications. For example, you need to issue c:diskperf -y in a command prompt to enable logical disk object counters in Windows 2000. In addition, Apache needs to be compiled with the mod_info module that provides a comprehensive overview of server configuration. Also, WebSphere needs to be optimized for performance-counter monitoring, and the Java Heap should be increased to a maximum of 512 MB to handle monitoring. The list goes on.

We still required the products we tested to monitor and manage Windows 2000 and Red Hat Linux 7.3, but we limited the applications support to Apache 1.3.x, Microsoft SQL and IBM WebSphere Application Server. We were flexible with WAS and said the software could support version 4.02 or 5.02. HP OpenView supported 4.x but not 5.x at the time of our tests. The other participants hesitated to support 4.x, since 5.x was shipping and available.

For the test bed, we set up two Windows 2000 (SP4) servers on dual-processing Pentium III (1,400 MHz) systems. We gave Red Hat Linux 7.3 a Dell PowerEdge 1650 (Pentium III, 1,133 MHz, with 2,048 MB of RAM). On each of those servers, we ran Apache Web Server 1.3.28. We set up WAS 4.0.2 on one Windows 2000 server using the infamous Pet Store and DB2, and version 5.0.2 on another W2K server using the Plant Store. In effect, HP OpenView monitored pets while our other participants monitored plants.

Each of the participants used a Windows 2000 (SP4) server (dual Pentium III, 1,400-MHz processors and 1,024 MB of RAM) as a central console with a data repository. The console server and the monitored servers communicated with each other over a Gigabit Ethernet backbone within a Windows NT Domain/Active Directory environment. To give them something to talk about, we generated load on each of the WebSphere servers using Empirix e-Test Suite 6.71. E-Test provided scripts to run against both WebSphere servers that simulated a transaction. A transaction included a login, browsing the pet or plant store, placing an item into a shopping cart, and checking out. The scripts were distributed to 30 Celeron 500-MHz PCs equipped with e-Test virtual users in "thick client" mode. We also used Spirent Communications' WebAvalanche to generate network load and Web server activity, some Windows 2000 Resource Kit tools, and some homegrown scripts.Avalanche simulated network and Web server load with test scenarios designed to get URLs from the Apache Web servers on monitored systems. Scenarios were run at regular intervals, simulating 100 concurrent users on each system. We also used two Windows Resource Kit 2000 tools to evaluate the performance of the system managers on Windows 2000: CPU Stress (cpustres.exe) and the LeakyApp tool. CPU Stress consumes processor cycles by continuously running in an endless loop. We activated up to four threads, setting individual thread priority to high.

The LeakyApp tool simulated memory leaks to force paging during memory shortages. It allocates memory to its own process and retains it until it is stopped or reset.

Finally, to generate a consistent load on disks for both Windows and Linux, we created scripts with infinite loops. On the Windows systems, one provided a directory scan of a logical disk or partition, and another engaged file write and delete operations using 525 files (58 MB). For the Linux boxes, we put Bonnie (www.textuality.com/bonnie/ download.html) in an infinite loop to test file writes to all file systems.

During all this activity, the monitored servers were observed and managed by the monitoring servers. The data was reviewed on each of the console servers, and we judged each product's ability to detect and take corrective action on OS and application faults, performance problems, and configuration changes.Think of system-performance managers as spy masters. They deploy agents on your network nodes that feed performance data back to home base. That data can then be rolled into useful reports to pinpoint problem spots (whether you'll gather enough dirt to justify a pre-emptive war on your Windows 98 machines all depends on your spin power).

We gathered products from BMC Software, Heroix Corp., Hewlett-Packard, NetIQ and Quest Software in our Syracuse University Real-World Labs®and sent them a spying. Any of the managers we tested will help you squeeze the most out of your systems; you'll have to weigh your needs against a variety of feature sets and pricing models--some of them hefty. This time, Quest's well-rounded Foglight 4 took top spot thanks to its glowing out-of-the-box performance and Linux support. HP was hot on Foglight's trail but was thrown off by OpenView's high price. To measure response time for systems from the client side, tools like Mercury Interactive LoadRunner and Empirix e-Test Suite simulate client-side transactions, giving you a picture of system and application performance from the outside in. They have been used by testing labs, including our Syracuse University Real-World Labs®, and developers to see how products respond under load. We used e-Test (see "How We Tested," page 70) to generate load for the system-performance managers to monitor.Especially useful for e-commerce applications, these tools measure how a site conducts transactions and generates sales under load. You can then use numbers to present business-process management data that is easily understood in the boardroom.

Toward that end, new performance managers, such as Cendura's Cohesion, that focus on monitoring and managing applications are starting to appear. Also, makers of conventional system-performance management products are now promoting client-side performance managers as well; such offerings include Hewlett-Packard's OpenView Transaction Agent Server and Quest Software's PerformaSure.

In addition, performance data can be analyzed in terms of business management. BMC Software's BSM (Business Service Management) maps performance data about systems and applications to higher-level business services. For instance, it presents completed sales from an e-store or transactions from a POS (point-of-sale) kiosk through real-time views of data in dashboard GUIs and reports.

Sean Doherty is a technology editor and lawyer based at our Syracuse University Real-World Labs(R). A former project manager and IT engineer at Syracuse University, he helped develop centrally supported applications and storage systems. Write to him at [email protected].

Post a comment or question on this story.Performance Management Software

Sorry,
your browser
is not Java
enabled

Welcome to

Network Computing's Interactive Report Card, v2. To launch it, click on the Interactive Report Card ® iconabove. The program components take a few moments to load.

Once launched, enter your own product feature weights and click the Recalc button. The Interactive Report Card ® will re-sort (and re-grade!) the products based on the new category weights you entered.

Click here for more information about our Interactive Report Card ®.