Right off the bat, you'll notice that our fictional vendor hasn't disclosed who tested the IT Pro or who paid for that testing. Even independent testing labs can be biased in presenting results because they're getting paid to produce marketing collateral. Vendor-funded reports rarely show that company's product anywhere but on top. Beyond that bias, beware these other common traps vendors set.
1. Deceptive Eye Candy
"More than twice as many transactions per second."
Colorful performance comparison graphs can make bold statements about the capabilities of such networking products as firewalls, Web and application servers, load balancers and cryptographic accelerators, but they rarely are a true indicator of performance.
In our example, the chart clearly shows that Vendor B's Ethernet Thingy can perform more than twice as many transactions per second as its competition. Great, but how does the vendor define transaction? This is an important omission, as that definition varies not only from product to product, but also from test to test. An HTTP transaction could be defined as a single get request or a page that requires multiple get requests. The size of the data transferred is also an important piece of the equation. Many products are tuned to produce high-performance metrics with relatively small pieces of data (1 KB) but degrade dramatically as the size of data increases.