by Jay Milne  Server Performance: Benchmarking, Monitoring and Avoiding Bottlenecks Benchmarking What are Benchmarks? Generally defined, benchmarking is the process of comparing two or more systems or components of a system to determine which can provide the faster performance or let more users access resources, such as databases or file systems. There are two general types of benchmarks: synthetic/artificial and real world/living. A synthetic benchmark does not reflect the actual use of the server in a production environment. Further, it applies a constant, artificial load on a specific component of the server to determine its maximum performance. There are synthetic benchmarks for the disk subsystem, the video subsystem, CPU, memory and network connection. Often the results of these benchmarks don't mean much except when relatively applied. A problem arises when peo ple take the results of an artificial benchmark out of context or try to extrapolate the results too far. Real world/living benchmarks attempt to measure the performance of the system under a load that would exist in a production environment. Unfortunately, the only way to do this is to put the server in production and see what happens-a luxury none of us have. The next best thing is a real-world benchmark test. These are often application-based, such as a database or messaging system, and may be customized based on your specific application and user environment. By far, the most common use of a benchmark is to measure performance. However, a benchmark also can be used to "burn in" a new piece of hardware or new application. During burn-in the server is placed under a heavy-stress level for long periods of time to see if any part of the system fails. System failures can come in the form of a system level or application software driver, faulty hardware or a misconfiguration. In performing a burn-in, you often will catch problems that would only arise after extended use or that would not turn up unless the system is under a heavy load. What's out there? A lot of benchmarks are available-too many to list. Some are commercial grade. Some are shareware. Some are worthwhile. And some are not any good at all. Here's a short list. Transaction Processing Performance Council (TPC) If you work with large transaction-oriented system, you probably have heard of TPC benchmarks. TPC benchmarks have become the de facto standard for measuring the performance of large, complex transaction-based systems. TPC tests are complicated and extremely difficult to set up and run. Some companies, including Sybase and Compaq Computer Corp., have teams dedicated to running TPC tests and tuning their products for top results. Results of TPC tests are audited by TPC: There is no such thing as an "unofficial" TPC result. There are four types of TPC tests, and you will hear them referred to by the type, such as TPCb or TPCc. TPCa First online transaction processing (OLTP) test TPCb Database server test TPCc Updated version of the OLTP test TPCd Decision support system test Standard Performance Evaluation Corporation (SPEC) SPEC is a synthetic benchmark that measures a server's integer and floating point processing capabilities. SPEC is often used by CPU manufacturers, such as Intel Corp. and Advanced Micro Devices (AMD), to show relative performance of their CPUs. SPEC is also popular in the scientific and technical computing fields. There are five major types of SPEC benchmarks. SPEC CINT92 Measures integer processing performance SPEC CFP92 Measures floating point processing performance SPEC CINT95 New version of the integer processing benchmark SPEC CFP95 New version of the floating point processing benchmark SPEC SDM Software Development Multitasking benchmark. AIM Technology AIM Technology tests measure the performance of Unix systems. Because AIM uses a combination of application-based workloads, much like the Neal Nelson RTE, to measure performance, the results are more likely to be closer to a real-world test. AIM has a suite of tests for Unix workstations and Unix servers. Suite VI Current Unix workstation benchmark Suite VII Current Unix server benchmark Suite IX Current individual subsystem tests Suites II, III and Milestone Original AIM tests Neal Nelson & Associates Neal Nelson & Associates has a suite of benchmark tests that run on both Unix and Windows NT systems. Neal Nelson tests are used widely by the government and military and in large corporate settings. The Business Benchmark is an artificial benchmark good for measuring performance of individual system components; the RTE can be customized for specific applications and user characteristics. The RTE is a complex benchmark that requires some dedicated resources. Business Benchmark Performs 33 different tests on individual subsystem components RTE Executes scripts against the server simulating client load Web Server Benchmark Uses clients running a script file to perform various actions against the Web server What to consider when running a benchmark: Make sure everything is up to date Keep a level playing field Make the test consistent Repeat the test Monitor the server, the network and the benchmark Understand what the benchmark does Review sample results before starting Computer systems are complex, even the smallest single processor server is made of many components that need to work in unison to ensure operation. When running a benchmark ensure that the operating system, application drivers, server-specific firmware and drivers are current. We have seen occasions where an updated driver has increased system performance by up to 25 percent. The goal of a benchmark is to be able to compare the performance of two or more systems. To do that, you need to keep the playing field level. Even the smallest variation in systems can distort the results. For example, if you have one system with four hard disks in a RAID 0 array and one system with six disks in a RAID 0 array, the latter would have a large read/write advantage because there are more disks to do the same amount of work. Typical problem areas include: Disk cache settings (write back versus write through) Performance characteristics of individual disks Operating system patches (different levels) Incomplete firmware on servers Application not tuned for optimal performance Just as the configuration of each server being tested should be kept the same, so should the benchmark. Many benchmarks let you fine-tune them for your needs. Although this gives you with a tremendous amount of flexibility, it also provides an opportunity for inconsistent testing. Document any changes made to the benchmark and repeat those changes for every server under test. If you keep everything consistent, the results of the test should be at least 95 percent repeatable. If they are not, something is a wrong with the benchmarking program or with a configuration on your server. The best thing to do is to run the test on single server at least three times, more if possible. There will always be some minor variation between the results. When running the benchmark, you need to monitor not only the operating system and applications running on the server but also the benchmark tool itself and the various components of the server. One small glitch in the system can skew the results. A number of tools on the market do a fair job of monitoring the operating system's performance characteristics, and the major operating systems each come with their own. In addition, after running each test be sure to check the error logs of the operating system and application to make sure no problems arose. Understanding what the benchmark does is key to successfully evaluating the server being tested. Take the time to run the benchmark on a test system several times, watching the performance statistics of your server. Doing so will let you identify bottlenecks in your system that might skew the results. For example, if you're running a test on a database that does a high level of I/O, make sure that your disk subsystem is not the bottleneck. Trying to benchmark a multiprocessor server that has only one hard disk doesn't do much good. You will never be able to see the real performance of your server. After you run the benchmark on a test server you need to scrutinize the results. If you have a server monitoring tool that records system-level statistics such as CPU utilization, disk I/O and memory cache, review those as well. Make sure the benchmark is stressing the system correctly and there is no unexpected bottleneck. The results of the test is only half the story, you also need to evaluate how the system performed, though the use of the monitoring tools. If CPU utilization was low, did the disk cause the bottleneck? You want to make sure the server is configured properly to get the results you are looking for. If you don't, you could draw incorrect conclusions about your system. Server Performance Bottlenecks Monitoring Tools Common Tips and Tricks Updated May 21, 1997 Print This Page E-mail this URL