Storage

01:00 AM
Howard Marks
Howard Marks
Commentary
Connect Directly
Twitter
RSS
E-Mail
50%
50%

VMware's VSAN Benchmarks: Under The Hood

VMware touted flashy numbers in recently published performance benchmarks, but a closer examination of its VSAN testing shows why customers shouldn't expect the same results with their real-world applications.

Comment  | 
Print  | 
Comments
Newest First  |  Oldest First  |  Threaded View
Gallifreyan
50%
50%
Gallifreyan,
User Rank: Apprentice
5/6/2014 | 1:44:14 PM
Re: Benchmarks will be benchmarks, no matter how faint the resemblance...
I agree that vendor benchmarks are going to be slanted toward the vendor. That's where independent testers come in of course. 

Maybe it's time for a venue that encourages real life benchmarks from real life end users. We used to have that on the bogomips level for Linux but that was even more useless than typical vendor benchmarks. :) 
Lorna Garey
50%
50%
Lorna Garey,
User Rank: Ninja
4/24/2014 | 4:08:25 PM
Re: Benchmarks will be benchmarks, no matter how faint the resemblance...
It seems like it matters less that the benchmark be completely real-world than that all similar products operate with comparable benchmarks, so that IT can compare apples to apples. Could a company get hold of (or replicate) these benchmarks to run them on their own setups with all short-listed products?
Howard Marks
50%
50%
Howard Marks,
User Rank: Apprentice
4/24/2014 | 3:12:37 PM
Re: Benchmarks will be benchmarks, no matter how faint the resemblance...
The sad truth is that for vendors benchmark results are frequently about marketing goals not educating the market.  Announcing that you've hit the 2 or 20 megaIOP level before any of your competitors will get you some space on websites and blogs.  Even better it gives your sales force, and fanbois, something to brag, tweet and concact customers about.

 

I just wish they would also publish some benchmarks that are closer to the way customers might actually use their products.

 

 -Howard
Andrew Binstock
50%
50%
Andrew Binstock,
User Rank: Apprentice
4/23/2014 | 5:10:30 PM
VMware Benchmarks
For many years, VMware has promulgated its own benchmarks. At times, these have been used by the larger industry, especially in the early days of benchmarking server loads. I remember tracking them when I was benchmarking servers at InfoWorld. Although we never ran VMware's benchmarks (we tended to prefer those from SPEC, which were consortium-created), we certainly heard from server vendors about their discomfort when being rated by VMware. It was difficult to know whether this was just standard vendor griping or whether the benchmarks did indeed disfavor certain designs, but my sense was that it was the latter. I don't think this was intentionally done by VMware. Rather it illustrates the difficulty of creating useful benchmarks that can be run on multiple architectures and deliver results that can be compared fairly.
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Apprentice
4/23/2014 | 4:57:45 PM
Benchmarks will be benchmarks, no matter how faint the resemblance...
I like Howard Marks' sense of realism as he inspects a vendor's benchmark. "VMware could have come up with a more realistic benchmark to run," goes without saying. So could Oracle and a dozen other vendors when it comes to benchmarking. In may experience, the benchmark is optimized to make the vendor look good, no matter how faint the resemblance to the customer's planned use of the product.
Susan Fogarty
50%
50%
Susan Fogarty,
User Rank: Strategist
4/23/2014 | 11:48:50 AM
Re: Details matter
Marcia, you're right -- customers need to read between the lines when evaluating any type of vendor claims. But I'm not sure that anyone really takes these banchmarks seriously, anyway. Readers, what do you think?
MarciaNWC
50%
50%
MarciaNWC,
User Rank: Strategist
4/23/2014 | 11:08:24 AM
Details matter
Howard's analysis shows how details really matter. It's so easy for vendors to tout great results, but you have to take a close look at what produced those results. 
Hot Topics
15
3 Signs You're Overspending On Data Storage
John Morris, President and CEO, Cleversafe,  7/24/2014
11
Fall IT Events: On The Road Again With 10 Top Picks
James M. Connolly, Editor in Chief, The Enterprise Cloud Site,  7/29/2014
White Papers
Register for Network Computing Newsletters
Cartoon
Current Issue
Video
Slideshows
Twitter Feed