IOV: The Final Frontier of Server Virtualization

Posted by Steven Hill on June 27, 2007

Tags: Steven Hill,  Components and Peripherals,  ESX server,  IO virtualization,  IOV,  Other,  PCI-SIG,  Virtualization,  Virtualization hardware,  Xen,  hypervisor

Channel: Other, Virtualization

The planets are finally clicking into alignment in the world of server virtualization, freeing IT to build a galaxy of high-performance and easy-to-manage virtual machines. However, before virtualized servers can become as performance-efficient as their physical counterparts, hardware and software vendors must provide sharable storage and network IO-after all, IO is the lifeblood of the data center. Development of virtualization-aware IO devices is crucial to making virtualized servers as efficient and reliable as physical servers are today.

Chip-assist technologies like Intel VT and AMD-V have opened the door to more efficient VMs (virtual machines), but keeping snorting beasts like eight-way, 1U servers fed with enough IO to keep a dozen or more VMs productive is still a challenge on a number of levels. Up-and-coming high-bandwidth fabrics like 10GbE, 8GbFC, SAS and DDR 4X InfiniBand can provide large-enough pipes to supply even the biggest racked server, but most adapter cards today are still designed to provide only one system interface and support only a single instance of an OS.

IO Virtualization Today Click to enlarge in another window

At present, disk and network IO is being accomplished by using a combination of software and CPU cycles to create a functional-if inelegant-shared IO environment, but at the cost of lower IO performance than what's achievable on a physical machine. The goal for IO vendors will be to find an efficient way of slicing up all this new bandwidth to feed those hungry masses of VMs.

That's where IOV, or IO Virtualization, comes in.

At the moment, the gatekeeper for virtual IO is the PCI-SIG (Peripheral Connect Interface Special Interest Group), the international standards body for the PCI interface. The group is more than a year into the process of standardizing IOV for the PCIe system interface, with the full set of standards expected at year's end. If the PCI-SIG can keep to its schedule, we should begin to see IOV-enabled devices at in the second half of 2008.

Tease Me, Please Me, IOV Me

Managing IO in a virtualized environment is a challenge on multiple levels. Most existing IO devices aren't designed to be shared. Memory allocation, interrupt handling and DMA addressing need to be continually assigned and renegotiated between VMs. Today this is being handled in different ways by different vendors; the two most common methods are device emulation and IO paravirtualization. VMware's extremely popular ESX server platform is based on the device emulation model, where the ESX VMM (virtual machine manager) creates a common hardware platform by emulating virtual IO devices through software.

Like everything else in this world, device emulation is both good and bad. The VMware approach makes for a very stable and standardized IO environment, with a small set of extremely common virtual interfaces that will be palatable for whatever combination of OSs and applications you may choose to run on a VM. It's also an excellent environment for migrating VMs across physical systems, because no matter where you go-there you are. Due to the enforced standardization of the hardware platform, all IO devices on the target machine are exactly the same as those of the source. Of course, the key downside is that all of that emulation at the software level comes at a performance cost-because software is never as efficient as hardware-and VM performance drops as IO utilization increases.

Perhaps the biggest challenger to VMware's IO device emulation strategy is from the approach used in the open-source, thin hypervisor technology from XenSource. Rather than completely emulating IO devices in the VMM, Xen utilizes a split-driver approach that reduces the processing requirements for administering IO by moving management of physical device drivers from the VMM to a dedicated device driver VM. In this virtual IO support model, a dedicated device driver VM-based on an unmodified physical device driver-forms a virtual device that presents the back-end driver half of the split-driver scenario.

The back-end driver then connects to what's known as a device channel, which serves to provide global access to all VMs running on that physical server. Then, each client VM-using the front-end half of the split driver-can access the device channel for shared access to the IO devices on that system. Though it sounds more complex, the Xen model has been shown to offer greater IO performance by substantially reducing the VMM's IO workload by letting the driver do all the heavy lifting. A split-driver approach with native pass-through capabilities for legacy support is shaping up to be the model of choice for the future; a key challenge in this scenario, however, is just how to orchestrate memory addressing to ensure that there are no conflicts between the dozens of DMA sessions that IOV technology will support

This particular problem will be addressed at CPU level through the introduction of the IOMMU (IO Memory Management Unit) technology scheduled to be included in next-generation processors from both Intel and AMD. An IOMMU's function is to streamline and increase memory protection for virtual-to-physical address translation of devices using DMA access, and to provide efficient interrupt translation for sharing IO. But this is also where the IO vendors need to step in and streamline the process further downstream.

IO, IO, It's Off to Work We Go

Enter the PCI-SIG and IOV. The premise behind IOV is to solve the problem by redesigning IO devices, like Ethernet NICs, Fibre Channel HBAs and even video cards, to present multiple, hardware-based virtual interfaces that can be independently addressed by the VMs themselves.

The PCI-SIG's task has been to define the nature of these virtual device host interfaces and set guidelines on how they individually assign resources, support targeted interface resets, enable VM caching of DMA address translations and provide controls for managing peer-to-peer DMA access. It's clear that these key issues need to be standardized at the level of the PCI interface, but it's worth noting that some of these capabilities are actually available today in the newest crop of Fibre Channel HBAs from companies like Emulex and QLogic. Given the absence of industry standards, these systems are vendor-specific, so if you're in the process of purchasing high-performance IO cards for virtualization, check with the vendor to ensure they can eventually be updated to the comply with the evolving IOV standard.

Long Time Coming

One reason IOV is taking so long to emerge: The PCI-SIG's focus on creating a standard that provides support for both single- and multi-root applications. Standalone servers are a single-root environment, and IO devices are accessed only by VMs residing on that machine. IOV in a multi-root context is designed for clustered systems, such as blade servers, where VMs from multiple physical machines may need to access the same high-bandwidth IO devices. Of course, this adds an understandable degree of complexity to the standardization process, but it's a problem that really needs to be adequately addressed in the early stages of IOV development.

For The Future Click to enlarge in another window

Though a single-root IOV specification will appear first in late 2007, a multi-root capability is especially important to the utility data center model, where compute, networking and storage pools can be dynamically allocated to virtual tasks. The ability to share fast IO ports across multiple compute nodes running multiple VMs-as in the case of a chassis full of blade servers-for load-balancing and failover purposes may well be the most efficient model to make best use of the upcoming, high-speed IO fabrics. Even in a single-root context, IOV offers the potential for a single IO device to provide separate interfaces for each VM with separate QoS, security and performance characteristics that can be individually tuned for that particular application.

All of the vendors we contacted for this article agreed that IOV technology is the wave of the future for dealing with the inefficiencies of existing IO in a virtualized environment. The initial sweet spot will be high-speed transports like 10GbE, InfiniBand and Fibre Channel, but there will be another surge of IOV devices for blade systems when the multi-root standards are released. The ideal IO environment for VMs of the future will include AMD-VT and Intel VT processors with IOMMU capabilities, motherboards with the PCI Express architecture, as well as new IOV-enabled cards for optimum flexibility and performance.