VMware Gets Real About Virtual Storage

VMware has talked about software-defined storage and a virtual SAN for a long time, but the tangible software it delivered still relied on shared storage--or in the case of its Virtual Storage Appliance (VSA), it wasn't ready for prime time.

This year's VMworld in San Francisco featured real software to back up the software-defined storage story. In the near term, VMware users will have three significant software-defined storage products: vSphere Flash Read Cache for server-side caching, VSAN to create hybrid storage pools from server attached devices, and good old Virsto for better snapshots and write acceleration.

The vSphere Flash Read Cache, frequently known as vFlash after the more ambitious development project from whence it sprung, is built into the ESXi 5.5 kernel, though VMware may choose to only enable it for users with Enterprise or Enterprise Plus licenses. While it does run at the hypervisor level and is well integrated into the vCenter administration, vFlash is primitive.

Yes, the guest VMs don't need an agent, but administrators must manually configure the cache settings for each guest, including the fixed amount of flash that guests should use as cache. At least half a dozen third-party caching products can dynamically allocate cache so VMs get more cache space when they need it and give up cache space for data they still have cached from the last run on Thursday night.

I hope vFlash will show IT that server-side caching is a good idea. I also hope free evaluation software from third parties will entice IT to choose from among these better products. VMware says vFlash will be out later this year.

[Get an overview of server-side caching and the latest moves from startups in "Server-Side Caching Gets Smarter Via Startup PernixData."]

VSAN, the virtual SAN (but without the lowercase v), turns storage across a cluster of servers (up to eight in a cluster) into a shared, hybrid storage pool. Each server donating storage in a VSAN cluster must have at least one SSD and 1 to 36 spinning disks. The SSD is used as a write-through cache with synchronous replication, protecting data across multiple servers. Unlike VMware's VSA, no hardware RAID controller is required.

VSAN works much like the storage engines in Nutanix and Simplivity's hyper-converged platforms and EMC's Scale-IO do, but there are differences.

One key difference is that those products are implemented as virtual storage appliances that run as virtual machines. By contrast, VSAN is a hypervisor kernel module. As a kernel module, VSAN interacts with the hypervisor not via NFS or iSCSI as the VSAs are forced to, but at the virtual machine level. This provides a shorter, cleaner path between the storage engine and the devices it manages, and reduces the number of CPU context switches needed to process a storage I/O request. VSAN is shifting from private to public beta, but most likely won't be ready for full production until early 2014.

The third leg of VMware's software-defined storage stool is Virsto, which VMware acquired in February. VMware has pretty much left Virsto alone, so it remains same flawed gem it always was. VMware's vSphere can take advantage of Virsto's metadata-based snapshot technology when duplicating VMs or composing VDI images through the VAAI cloning feature.

The flaw is that Virsto snapshots can't completely replace vSphere's log-based snapshots, as there is no "take snapshot" primitive in VAAI. So when you back up your VMs, the vStorage API for Data Protection will still use the performance- and space-sapping log-based snapshot as the source for the backup.

Reliable sources have told me that Virsto founders Alex Miroshnichenko and Serge Pashenkov are, rather than working on a new version of Virsto, working on improvements to the VMFS snapshot mechanism. I infer from this that VMware acquired Virsto more for Alex and Serge to fix their snapshots than as a replacement for VMFS.

The one significant storage technology VMware didn't advance at VMworld is Virtual Volumes, or vVols. The vVols technology essentially micro-volumes a block storage array to allow each .VMDK or virtual disk to be managed individually. Most block storage vendors plan to use vVols to provide per-VM storage management features including snapshots and replication. Current SAN protocols from iSCSI to Fibre Channel and SCSI just can't scale to the thousands of volumes required in even a moderate sized virtualization environment.

Also curiously missing was serious discussion of VMware's VSA. While VMware spokespeople were clear to say that the VSA was not being discontinued and would still be marketed to the SMB space, VMware didn't announce any enhancements or even really discuss how customers should choose between VSAN and VSA.

My impression is that VMware plans to sell VSAN at a higher price than the $2,000/server cost of the current VSA, leaving the VSA as a low-cost alternative. No official word on pricing (or even solid rumor) for VSAN has come my way, so that's merely an educated guess.

VMware is making real progress towards the software defined data center of its dreams, despite the obvious impact on parent company EMC's array business if the world flocks to VSAN. EMC is wise enough to know that its lunch is on the table, and that EMC is better off if VMware takes big bites rather than an upstart eating it all.

[Catch Howard Marks' informative session "SSDs In the Data Center" at Interop New York, from September 30th to October 4th. Register today!]