Performance-Monitoring Services

To test these services, we used a real site, CountryWatch, hosted by VeriCenter, an AIP (application infrastructure provider). We also coordinated each vendor's service engagement with the production control of both CountryWatch and VeriCenter. While both parties were interested and committed to participating in the test, and we have experience in project management as well as working across departmental, divisional and customer boundaries, the x-factor was each vendor's project-management service.

CountryWatch serves country-specific data, such as demographics, politics and news, for every country in the world. Use is subscriber-based and involves a secure login and shopping cart for some reports. The site runs on dual Microsoft Internet Information Server (IIS) machines, with a separate server for the secured shopping-cart transactions and a single back-end Microsoft SQL 7.0 database. All patch levels and releases were current and supported.

VeriCenter's infrastructure provides mirrored DS-3 Internet-access circuits, connected via separate carriers, into two mirrored data centers. The Web server is only two hops from the Internet and sits behind a firewall. Akamai Technologies and CacheFlow provide caching, but both were turned off for most of our tests.

The Web servers run on dual 700-MHz Intel processors with 2 GB of RAM, running Microsoft Windows 2000 and IIS 5.0 patched. The e-commerce server is a 700-MHz 1-GB Intel server running Windows 2000 and Microsoft SQL 7.0, and the database runs on a Compaq ProLiant 6400R quad processor with 1 GB of RAM, running Windows 2000 and Microsoft SQL 7.0.

We asked the vendors to treat this project as they would any engagement, driving the fact gathering, script creation and interaction during the tests, and the summarization and reporting at the end of the engagement. Each vendor could consult with VeriCenter and CountryWatch regarding the infrastructure, server hardware, application, geographical reach and transaction mix. Vendors who requested Web logs were given the prior month's logs to analyze.

Each test was conducted at night and timed to coincide with low user traffic as determined by log analysis. Representatives from VeriCenter and Network Computing were online while the tests ran. Some of the vendors checked their script logic with calibration runs. The vendors each ran a download of an image to determine the bandwidth and infrastructure's ability to deliver the 8.8 Mbps dedicated to testing.

All testing took place outside the firewall, with only Windows NT perfmon and occasional telnet access by VeriCenter to monitor servers and network infrastructure. No animals were harmed during this test; however, at least one cheese-laden pizza and a case of Red Bull were consumed.