To get meaningful results from a hybrid storage system, our benchmarks need to access the storage the way real world applications access storage. Benchmarks like TPC-C and SPECsfs are based on IO traces from real-world users and applications, so they create hot and cold areas in their test data. This means that their results should correlate more closely to real world performance than IOmeter does. The problem is that these benchmarks are expensive to acquire, and to run, so vendors tend to report results only for specially tuned high-end storage systems that use large numbers of small disk drives and other configuration options that are rare in the real world.
If that weren’t depressing enough, even the most sophisticated benchmarks write the same, or random, data to create their entire data set. While disk drives, and most SSDs, perform the same regardless of the data you write to them, the same can’t be said about storage systems that include data reduction technology such as compression or data deduplication. If we test a storage system that does inline deduplication--like the new generation of all solid state systems from Pure Storage, Nimbus Data or Solidfire--and use a benchmark that writes a constant data pattern all the time, the system will end up storing a 100-Gbyte test file in just a few megabytes of memory, eliminating pretty much all IO to the back-end disk drives, or flash, to deliver literally unreal performance numbers.
Our friendly competitors at Demartek recently posted hex dumps of the data files created by several popular benchmarks, so you can see how bad the problem is first hand.
Creating a benchmark that stores realistic data in realistic locations is a major undertaking. The benchmark would end up having to read data from a repository of some kind to write it to the system under test. To generate enough traffic to make an enterprise storage array with 500 Gbytes of flash breathe hard, we’ll need several servers working in concert and reading their source data from a storage system at least as fast at sequential IO as the system under test can run the benchmark. I sure hope someone comes up with a good one soon.
Disclaimer: Solidfire is a client of DeepStorage.net, Tom from Nimbus Data let me sit in his Lamborghini at SNW. DeepStorage.net and Demartek provide similar services, so I hate giving them a plug.
