vMotions a Killer App for 10GbE at Vonage

Every year since the first Global 2000 company installed VMware, server virtualization has been at the core of more and more data center architectures. Vonage is one major corporation that is now making the leap, and its architects shared with me how 10 Gigabit Ethernet is playing a key role in how the organization handles vMotions.

With approximately 2.4 million subscribers, Vonage maintains and operates a VoIP network that supports more than 6 billion communications per year to more than 60 countries. Located at corporate headquarters in Holmdel, N.J., is the Vonage Development and QA Lab. It’s here that new technology and products are evaluated before they can enter a production environment. The lab also is where the company's IT architects and engineers develop and certify the standard hardware and software images that will be used in production systems ranging from front-end Web servers to back-end database servers.

Taking advantage of technologies that will allow the company to consolidate infrastructure, reduce costs and speed new server deployments, the Holmdel data center is migrating from a mostly non-virtualized, "discrete" environment to a virtualized data center. This installation features about 40 data center cabinets with 1,100 rack-mount servers, each dedicated to specific applications. The power-hungry rack servers are connected to data center LANs and SANs by more than 3,000 cables, and deploying a new server takes about two weeks from the time a user makes a request.

The migration from discrete data center to virtualized data center involves many new technologies, products and processes. Blade servers, server virtualization and network virtualization stand out as technologies that are key to slashing costs while maintaining application performance and availability.

--Blade servers: Vonage is deploying HP ProLiant BL490c G7 blade servers to perform the same function as rack servers but with far less floor space, power, cooling and cables. By sharing resources through a single enclosure, blade servers will also eliminate a great deal of management complexity and overlap.

--Server virtualization: A typical application server utilizes only 1% to 5% of available CPU. Server virtualization allows Vonage to fully utilize the compute power of each physical blade server by running multiple virtual machines (VMs) and applications. The result is another level of massive consolidation and cost savings.

--Network virtualization: To accommodate the proliferation of VMs, the HP BladeSystems feature virtual networking capabilities that allow server admins to configure unique virtual networks for each VM. Embedded on G7 servers used by Vonage are dual-port integrated HP Virtual Connect 10-Gbit FlexFabric Adapters, which are based on Emulex OneConnect technology. A single HP 10-Gbit FlexFabric Adapter port can be configured as four FlexNICs, with an option to configure one as a FlexHBA for either Fibre Channel or iSCSI storage connectivity. Each FlexNIC or FlexHBA can be configured with its own set of network policies tailored to the needs of specific VMs and applications.

Once Vonage architects got started, they realized the bar was set high. The 1,100 rack servers that form the company's discrete production environment are highly tuned for maximum performance and availability. The rest of the IT team was skeptical that a mass migration to virtual servers could be accomplished without compromising one or both. Therefore, the first step was to pilot the technology with a single blade system chassis and two server blades.It took about six weeks for company architects to learn how to efficiently configure a physical server with VMware ESXi, and HP VirtualConnect Flex-10. Then, one application at a time, the number of VMs in the Development & QA lab grew to 900. The confidence of the IT team also grew as applications running in the lab demonstrated the performance needed in a production environment.

Eventually, live migration emerged as a killer app for the 10GbE on board the HP servers. Usage in the lab also demonstrated the benefits of automated server performance optimization and maintenance. With VMware Distributed Resource Scheduler (DRS), scheduling server downtime to move applications to less utilized servers was no longer needed. Without anyone knowing, DRS was automatically moving VMs when servers became hot. This made quick evacuation of VMs during vMotions a killer application for the 10-Gbit HP Virtual Connect FlexFabric Adapters on board the G7 server blades.

Vonage engineers provisioned 4-Gbit FlexNICs to minimize the time required to move hundreds of gigabytes of data involved in live migrations, and quickly found the vMotions over-running the 4-Gbit bandwidth. Based on their experience in the lab, company system architects decided to use 10-Gbit FlexNICs in production so that ESXi has the full bandwidth of the FlexFabric Adapter available for live migrations.

The following are a few lessons the Vonage staff want to share with their peers:

--Share your system designs with your vendors: Vonage architects discovered that HP technical staff were aware of issues they encountered and would gladly have reviewed their system design.

--It takes a while to learn how the virtualization layers work together: It took six weeks to learn how to use the server and network virtualization tools efficiently. However, after learning proper configuration techniques, profiles could be cloned by Level 1 operators for rapid VM deployment.

--Take things one step at a time: 18 months ago, there was not a single VM in the Holmdel Development & QA Lab. Starting with a couple of server blades, Vonage architects slowly added application workloads that allowed the rest of the IT team to build confidence in the ability of VMs to perform.

--vMotions can use an entire 10-Gbit Link: Testing showed that vMotions were saturating the 4-Gbit FlexNICs provisioned in the lab.

Frank Berry did not receive compensation in the form of free promotional products or cash or cash equivalent for the purposes of writing this blog from (including but not limited) to Vonage, Emulex and HP.