Storage

07:15 AM
Howard Marks
Howard Marks
Commentary
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Server SANs: The Dark Side

The new technology promises compelling benefits, but mixing and matching storage hardware and software can backfire, as one IT pro found out when his VMware VSAN failed.

Like many new technologies, server SANs bring a mix of promise and risk. Replacing the arcane protocols of traditional SANs -- and the associated high priesthood of storage administrators --  with software-and-server contained SSDs and disks is pretty compelling, but also has a dark side. If you can mix and match storage hardware and software to create a Server SAN, you may discover they don’t play together as well as you might like.

At the DeepStorage lab, we ran into this problem when building a cluster to test VMware’s VSAN. We bought servers with Intel ACHI SATA controllers when ACHI was on the VSAN beta hardware compatibility list, but when VSAN shipped, VMware determined that ACHI was too slow.

Last month, VMware once again declared that VSAN requires more disk controller horsepower and removed several low-end SAS host bus adapters (HBAs) from its hardware compatibility list (HCL).

This change was based in no small part on the experience of Jason Gill, who reported on Reddit that when he replaced a failed storage node in his VSAN,  the rebuild process went horribly wrong. In fact, it went so horribly wrong that all of his virtual machines went offline.

Even with his constant attention, and VMware’s level 147 support on the phone, shutting down less-critical VMs and the like to manage load, his critical servers were offline for seven hours and it was a full 24 hours before the system was fully up and running.

VMware’s root-cause analysis pointed the finger at the Dell H310 SAS HBAs in Mr. Gill’s storage nodes.  Here's the official statement on the VMware blog: "Although Virtual SAN has a built-in throttling mechanism for rebuild operations, it is designed to make minimal progress in order to avoid Virtual SAN objects from being exposed to double component failures for a long time. In configurations with low queue-depth controllers, even this minimal progress can cause the controllers to get saturated, leading to high latency and IO time outs."

In plain English that means, "Sure VSAN throttles, but we set a minimum progress level as well and that minimum progress, and the VMs doing their normal I/O, was more than the poor underpowered HBA with a maximum queue length of 25 could handle."  That’s a reasonable statement, except for the fact that the Dell H310 was at that time on the VSAN hardware compatibility list.

Like most steely eyed storage guys, I consider a vendor’s HCL as a significant endorsement, if not a holy document. When VMware said that a hardware component on their HCL was the cause of a failure it just felt wrong.  It seems obvious that VMware didn’t test the edge cases, like a storage node failing and being replaced, on lower end hardware like the H310.

While the $300 or so additional cost for a faster SAS HBA, or RAID controller, is a small fraction of the $15,000 cost of a VSAN server and its software licenses, it gets me wondering if the whole “no RAID controller required” selling point for VSAN was that good an idea. I, for one, have nothing against a good hardware RAID controller.

If the rumors are true, VMware is picking up the tab for at least some customers who have to buy faster HBAs, like Dell’s H710. I hope they’re true. It would be nice to see a vendor stand behind its HCL like that.

Howard Marks is founder and chief scientist at Deepstorage LLC, a storage consultancy and independent test lab based in Santa Fe, N.M. and concentrating on storage and data center networking. In more than 25 years of consulting, Marks has designed and implemented storage ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
ReturnoftheMus
50%
50%
ReturnoftheMus,
User Rank: Ninja
8/3/2014 | 2:14:02 AM
Fail Fast, Fail Often!
An excellent reminder and a lesson for us all.
MarciaNWC
50%
50%
MarciaNWC,
User Rank: Strategist
8/1/2014 | 3:49:12 PM
testing
For a vendor to put a product on its hardware compatability list, you'd assume it would have done some comprehensive testing.
Slideshows
Cartoon
Audio Interviews
Archived Audio Interviews
Jeremy Schulman, founder of Schprockits, a network automation startup operating in stealth mode, joins us to explore whether networking professionals all need to learn programming in order to remain employed.
White Papers
Register for Network Computing Newsletters
Current Issue
Video
Twitter Feed