NetApp, EMC in Benchmark Brawl

NAS rivals are at it again as NetApp challenges EMC's latest claims of superiority

March 17, 2004

3 Min Read
Network Computing logo

Folks at Network Appliance Inc. (Nasdaq: NTAP) are accusing EMC Corp. (NYSE: EMC)of irresponsible marketing for its latest price/performance claims about its NAS system (see NAS Gateway Boosts EMC).

The NetAppers took offense at EMCs boast that its NS700 recorded a significant price/performance advantage over the entire line of NetApp NAS filers. EMC based its claims on results it achieved using the System File Server (SFS) benchmark supplied by the Standard Performance Evaluation Corp. (SPEC).

Vendors do their own testing for SPEC, and NetApp questions the way EMC achieved its results. NetApp director of technology Keith Brown accuses EMC of using well-known testing tricks to adjust the performance of its systems, and he maintains the price/performance claims are invalid because SPEC does not verify pricing.

In the release, EMC claimed its new NAS700 system delivers a 42 to 55 percent price/performance advantage over NetApp’s systems.

“They’re embarrassing our industry,” Brown says. “That’s deception and misrepresentation of data, and it’s not good for our industry.” [Ed. note: Isn’t all marketing deception?]Brown says EMC’s tricks included turning off its disk data protection features because it drags performance, short-stroking (artificially limiting the amount of disk data the system reads during tests), and souping up the performance of the NS700 gateway by connecting it to two Clariion storage systems.

“Is Clariion performance that poor that they had to use two of them?” Brown gibes.

Does NetApp have a valid complaint? Probably more so on the price issues than the actual performance numbers. The vendors do their own testing, and then the results go through a committee of competing vendors before they are published. NetApp is a member of the committee and so could have asked for a full disclosure from EMC before the results became official. NetApp reps said they posed questions, but did not seek a full disclosure.

“Games can be played,” says analyst Arun Taneja of The Taneja Group. Taneja says EMC could have done all the things Brown charged, but probably disclosed it all in the documentation. “I’m willing to give EMC the benefit on performance from the SPEC benchmark, but the price/performance claims are up for grabs.”

The SPEC only takes performance into account, not price. SPEC president Kaivalya Dixit says that's because pricing varies from deal to deal. So EMC did its own price/performance calculations, and NetApp’s Brown questions whether the prices EMC used reflect the true configuration of the systems tested. An EMC spokesman says they used the most accurate data available.“When making price/performance comparisons, EMC uses the SPEC SFS numbers from apples-to-apples configurations along with estimated list pricing based on several readily available sources, including NetApp's competitive quotes and others,” EMC spokesman Justin Bartinoski says. “If NetApp wants to share their average street price, we'd be happy to recalculate the price/performance comparisons.”

The battle between EMC and NetApp is heating up now that EMC has reclaimed the NAS market-share lead according to the latest IDC

numbers (see Storage Sales Soar Entering '04). Yet skirmishes over benchmark performances are nothing new between these two. Both have been involved in similar spats:

  • EMC vs NetApp: NAS TCO Tussle Last September, NetApp took issue with a Meta Group Inc. report that gave EMC a total-cost-of-ownership edge, running databases based on SPEC numbers. NetApp called the analysis flawed, contending SPEC SFS does not allow oranges-to-oranges comparison of system price to performance.

  • Is NetApp SANbagging? NetApp’s numbers came under dispute in this dustup a year ago. NetApp published a white paper favorably comparing its internal SAN benchmark testing to the results published by EMC and Hewlett-Packard Co. (NYSE: HPQ) on their Websites. EMC called the claims “a marketing stunt.”

  • Does EMC's DMX Measure Up? EMC’s claim that its Symmetrix DMX was the fastest on the market 13 months ago came under fire from its SAN rivals because EMC did not use industry benchmarks to back its boast.

— Dave Raffo, Senior Editor, Byte and Switch

Stay informed! Sign up to get expert advice and insight delivered direct to your inbox

You May Also Like

More Insights