Upcoming Events

Cloud Connect
Santa Clara
Feb 13-16, 2012

Cloud Connect brings together the entire cloud eco-system to better understand the transformation we're experiencing and promises to be the defining event of the cloud computing industry. Learn about the latest cloud technologies and platforms from thought leaders in Cloud Connect’s comprehensive conference.

Register Now!

F E A T U R E

Performance Monitors at Your Service

June 11, 2001
By Bruce Boardman

Mercury Interactive Corp. Active Test

Mercury Interactive is aptly named; the vendor is fast and its test more interactive than the others we reviewed. Active Test is an outgrowth of the company's long-standing enterprise test product and consulting business. Its software has a reputation of being so hard to use that consultation is required; however, this is an advantage with its load-testing service. Mercury drives this business as a consultancy from beginning to end, being careful to use clients' knowledge and goals without burdening them with project management or undue statistical analysis.

Recommendations

Mercury's careful control and focused testing of component by component determined exact cause and effect for CountryWatch's Web server. Nearly every other vendor -- Atesto is the exception -- pointed a finger at the ASP content and SQL back end, but only Mercury specifically and unequivocally found that the ASP content caused the Web servers to be CPU-bound.

Each test had very specific goals with carefully chosen options that depended on previous test outcomes. Mercury describes this as an iterative process designed to make the necessary changes to meet the desired goal. However, this does not mean that the process was rigid; in fact, just the opposite is true. Mercury tests were the most interactive in terms of monitoring interfaces and verbal feedback between the Mercury test team and VeriCenter.

Mercury Interactive Corp.'s Active Test

Click here to enlarge

The CPU diagnosis was based on three system metrics measured while CPU utilization was at 100 percent. First, plenty of memory was available, with utilization hovering between 25 percent and 30 percent for the entire test. There was no ASP queuing, indicating that there were enough threads, and the context switching was very rapid, indicating that the CPU was handling as many ASP threads as fast as it could.

Mercury did a calibration test, in which 10 users downloaded a large image, saturating the pipes to 8.8 Mbps. This test served two purposes: It determined if the network bandwidth and infrastructure could maintain the 10 percent allocation of two DS-3s VeriCenter promised CountryWatch. And the calibration test verified that the network and infrastructure were not a bottleneck. During subsequent server tests, no graphics were downloaded; thus, VeriCenter's shared infrastructure had little to do, and the Web servers did the work without the aid of client-side cache.

Results Delivered

During and after testing, Mercury delivered superior insight into the effects of the load on the Web site. We had three consoles to monitor the tests' progress, in addition to being online with the test center. One Web console showed throughput and response time statistics for the entire time and the last 180 seconds of the test, total hits and errors, and current user load being executed by the test (see screen at top right).

A second Web console, Active Watch, monitored the test from a user's point of view, and showed poorest transaction performance, sorted on geographical and transactional overlays (see screen below).

Active Test's Active Watch

Click here to enlarge

The third console was a browser-based collaboration session, which showed the NT perfmon statistics being displayed behind the firewall. This console helped VeriCenter understand the effects the load was having on the site and let VeriCenter question Mercury Interactive's conclusions as they were formed.

In preparing for the test, Mercury used the Web logs and interview process to determine what scripting and load-test sequence it should deploy. Once the scripts were created and validated, a subsequent conference with CountryWatch and VeriCenter set the stage for the test. Modifications to the test plan and the scripts were made and circulated for approval before testing. All this happened in just a couple of days for all the vendors, not just Mercury, but Mercury kept very tight control over the process documenting and following up on all project steps.

Analysis

The Mercury test comes with very complete reported results and a debriefing. Reports tend to be on the statistical side. In fact, Mercury provides all the reports along with its analysis tool, and raw results if you want to slice and dice the data in some way that Mercury hasn't considered.

However complete, though, the resulting analysis didn't have the same perspective about the available headroom compared with current load or the careful step-by-step explanation and annotation of the test as did Keynote's KeyReadiness. Regardless, the money spent on a test with Mercury will be worth every cent.

Active Test, starts at $25,500 ("six-pack" of test runs), Mercury Interactive Corp., (408) 822-5200, (800) TEST911; fax (408) 822-5300. www.mercuryinteractive.com