1. Deceptive Eye Candy
"More than twice as many transactions per second."
Colorful performance comparison graphs can make bold statements about the capabilities of such networking products as firewalls, Web and application servers, load balancers and cryptographic accelerators, but they rarely are a true indicator of performance.
In our example, the chart clearly shows that Vendor B's Ethernet Thingy can perform more than twice as many transactions per second as its competition. Great, but how does the vendor define transaction? This is an important omission, as that definition varies not only from product to product, but also from test to test. An HTTP transaction could be defined as a single get request or a page that requires multiple get requests. The size of the data transferred is also an important piece of the equation. Many products are tuned to produce high-performance metrics with relatively small pieces of data (1 KB) but degrade dramatically as the size of data increases.