07:18 PM
Brent Salisbury
Brent Salisbury
Repost This

Inside VMware NSX

From VTEPs to OVSDB, VMware's NSX introduces unfamiliar protocols and technologies to the data center network. Check out this overview to get up to speed.

At VMworld 2013, VMware released its much-anticipated NSX network virtualization platform. A growing number of virtual ports have appeared in the data center due to server virtualization, and NSX takes advantage of these virtual switches to create an overlay that runs over the physical network.

NSX uses SDN concepts to implement network virtualization. It uses flow-based forwarding via OpenFlow to instantiate the network flows. Flow forwarding exposes various L2-L4 header fields, along with Layer 1 logical and physical interfaces. Flow forwarding is an important concept because that combination of header attributes describes application traffic, not just two addresses in a network.

By unifying business logic and network headers, network control applications can install a simple forwarding policy or complex forwarding and service insertions.

Nested inside of the ESX hypervisor is a virtual switch (vSwitch). The vSwitch can operate as a typical flood-and-learn switch, or attach to the NSX controller. NSX builds tunnels between vSwitches using the VXLAN protocol (among others). These tunnels originate and terminate in VXLAN Tunnel Endpoints (VTEPs).

[VMware lined up an impressive number of partners for its NSX launch--except one. Find out where Cisco and VMware disagree in "Cisco Skips VMware's NSX Coming Out Party."]

A VTEP can be instantiated in servers or in a hardware switch, if it is supported in the switch's firmware and silicon. This creates a data path connecting two data planes over an L2 or L3 network. One vSwitch gives the appearance of being directly connected to another. Each vSwitch is now a VTEP in that particular virtual topology.

Underlying physical ethernet fabric
Source: Brent Salisbury

The overlay approach provides two important benefits for developers.

First, overlays bypass the constraints presented in the native physical network such as VLAN IDs, discontiguous networks or disjointed administrative network domains.

Second, tunnels remove the need for developers to extract and validate all the network paths, health and performance of the underlying network. That is, developers would, without tunnels, have to program applications to go and find out network state of each autonomous network system, which is an impractical requirement.

Flow programming is not a trivial implementation, even if focusing only on the edge. With point-to-point tunnels between every vSwitch, all network elements appears directly connected to one another.

Instead of focusing on each node in the network, network applications can focus on business logic and the task of classifying application traffic--and in turn applying policy. It begins to create a modular environment.

VTEP Edge appears directly connected
Source: Brent Salisbury

The use of VXLAN enables a greater degree of segmentation in the data center network. Traditionally, tenants are segmented from one another using VLAN IDs (VID). However, the 12-bit data structure of the VLAN ID means that a data center network is limited to 4,094 VLANs. By contrast, VXLAN includes a 24-bit VNI (VXLAN Network Identifier), which providers over 16 million available IDs.

Currently most switching hardware is constrained by the VLAN field processor embedded in the silicon. Generally speaking, the current generation of switches can process 250 to 500 VLAN IDs in hardware without affecting performance.

Don't expect hardware vendors to support millions of VXLAN VNIs right way. Vendors are presumably using the same VLAN field processors and silicon resources to process VXLAN as they would VIDs.

NSX Forwarding

In the NSX architecture, a VM boots and the host registers with the NSX controller. The controller consults a table that identifies the tenant and returns the topology the host should participate in to the vSwitch. The key identifier for virtual isolation is the VNI, which maps to a tenant's VXLAN-segmented topology.

Layer 3 forwarding between broadcast domains is supported at the edge of the NSX network in the vSwitch. This is performed by ARPs being punted to the controller and looking up the location of the destination MAC and destination VTEP in a host table in the controller.

If the host is not found, the traffic can be dropped or forwarded to a BUM traffic service node. Host discovery is forwarded to the VTEPs in the tenant tunnel overlay with multicast. VMware says it will soon support unicast, as well. Unicast is always attractive, particularly for interconnects relying on provider networks or the Internet.

Next page: Integrating Overlays

1 of 2
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
David Klebanov
David Klebanov,
User Rank: Apprentice
9/12/2013 | 6:10:59 PM
re: Inside VMware NSX
Hi Brent,

Good write up. I am trying to better understand your statements about L3 forwarding. According to NSX public material, logical routing is distributed to each hypervisor, but it would also seem that control plane for L3 functions lays in the VMs deployed on NSX edge node (called by VMware as Logical Router Control VM). I am confused here... am I to understand that L2 bridging control plane is on NSX controller and L3 routing control plane is on NSX edge node? Am I also to understand that for L2 bridging ARP traffic is terminated locally at the vSwitch, but for L3 routed traffic ARPs are punted to NSX edge Logical Router Control VM?

Thanks for clarifying.
More Blogs from Commentary
SDN: Waiting For The Trickle-Down Effect
Like server virtualization and 10 Gigabit Ethernet, SDN will eventually become a technology that small and midsized enterprises can use. But it's going to require some new packaging.
IT Certification Exam Success In 4 Steps
There are no shortcuts to obtaining passing scores, but focusing on key fundamentals of proper study and preparation will help you master the art of certification.
VMware's VSAN Benchmarks: Under The Hood
VMware touted flashy numbers in recently published performance benchmarks, but a closer examination of its VSAN testing shows why customers shouldn't expect the same results with their real-world applications.
Building an Information Security Policy Part 4: Addresses and Identifiers
Proper traffic identification through techniques such as IP addressing and VLANs are the foundation of a secure network.
SDN Strategies Part 4: Big Switch, Avaya, IBM,VMware
This series on SDN products concludes with a look at Big Switch's updated SDN strategy, VMware NSX, IBM's hybrid approach, and Avaya's focus on virtual network services.
Hot Topics
Converged Infrastructure: 3 Considerations
Bill Kleyman, National Director of Strategy & Innovation, MTM Technologies,  4/16/2014
Heartbleed's Network Effect
Kelly Jackson Higgins, Senior Editor, Dark Reading,  4/16/2014
White Papers
Register for Network Computing Newsletters
Current Issue
Twitter Feed