Network Computing is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Sun Bit by Benchmark Bug

Earlier this month, Sun Microsystems Inc. (Nasdaq: SUNW) -- and by extension, its partner Hitachi Data Systems (HDS) -- was set to claim it had the fastest storage system in the world, until it discovered that the performance benchmark test it was using had a critical flaw that invalidated the results.

Sun's initial testing, using the Storage Performance Council (SPC)'s benchmark, showed that its StorEdge 9980, a rebranded version of HDS's Lightning 9980V storage array, performed at 67,098.02 I/O operations per second (IOPS). That would have shattered 3PARdata Inc.'s previous record of 47,001.1 IOPS by more than 30 percent (see 3PAR Claims Benchmark Title).

But before Sun had a chance to officially flaunt the results, the company retracted them due to a flaw in the test itself, according to the SPC, a vendor consortium whose sole purpose is to maintain and promote storage performance benchmarks.

A bug in the SPC-1 test's workload generator caused Sun's results to be "noncompliant," says Walter Baker, an engineer with consulting firm Gradient Systems Inc., who is contracted as the SPC's administrator and auditor. "The generator that creates the I/O stream going across the specified storage did not cover the amount of storage they configured," he says, adding, "Sun didn't do anything wrong... They configured what they said they configured."

Sun is now busy preparing for a new benchmark test, which it claims will yield results as good as -- or better than -- the original ones. Nevertheless, the episode has stirred up renewed criticism of the SPC benchmark by EMC Corp. (NYSE: EMC), the test's biggest detractor.

  • 1