• 01/02/2014
    8:06 AM
  • Rating: 
    0 votes
    Vote up!
    Vote down!

Ethernet: The Datacenter's Unifying Foundation

Ethernet, now the standard for datacenter connectivity, is undergoing constant innovation from the core to the edge across the network. It's important to understand where Ethernet development is heading.

With a vast ecosystem of thousands of product suppliers committed to compatibility and the IEEE 802 family of networking standards undergoing constant, market-driven innovation, Ethernet has been taken places its inventors never could have imagined 40 years ago. Certainly, its role in the datacenter continues to expand rapidly.

What must enterprises understand about the technology's development with regard to datacenter connectivity, in order to ensure that the upgrades they initiate today position their infrastructures to take full advantage of Ethernet's evolution into the future?

Converging the datacenter
Every day, Ethernet shoulders a wider array of sophisticated datacenter services. A technology initially developed as a way to link printers and other devices seamlessly across a local area network (LAN), Ethernet now supports increasingly demanding applications within and among the world's most advanced datacenters. For example, Internet Small Computer System Interface (iSCSI) and Fibre Channel over Ethernet (FCoE) utilize the higher available bandwidths of Ethernet to consolidate storage traffic across the datacenter on to a unified transport. This is valuable because it eliminates the need to maintain separate block and file infrastructures.

In addition, Network File System and Server Message Block, which have been in place for decades, leverage the ubiquitous Ethernet transport to provide remote file access. In high-performance computing applications, Ethernet services such as Remote Direct Memory Access over Converged Ethernet (RoCE) enable a cost-effective reduction in cabling and administrative needs.

Datacenters have historically deployed various link-layer technologies to run a range of application-specific networks. Fibre Channel was deployed to support storage services, InfiniBand for high-performance computing, and Ethernet for basic LAN applications. Today, however, Ethernet is emerging as the unifying fabric for the full gamut of such applications.

Datacenter bridging (DCB) has provided a breakthrough in this area. The IEEE 802.1 Working Group's Datacenter Bridging Task Group works to satisfy the requirements of protocols and applications in the datacenter. A host of standards is intended to enable a DCB capability that eliminates packet loss due to queue overflows for selected mission-critical applications and assigns quality of service to help ensure that such high-priority traffic is allotted the bandwidth that it requires. The standards include:

  • IEEE 802.1Qbb "Priority Flow Control"
  • IEEE 802.1Qaz "Enhanced Transmission Selection and Datacenter Bridging Exchange Protocol"
  • IEEE 802.1Qau "Congestion Notification"

In this way, DCB is designed to enable lossless Ethernet networks that can cost-effectively collapse FCoE, iSCSI, and RoCE protocols on to the same physical Ethernet infrastructure. Datacenters have long sought convergence and its cost and simplicity benefits. And now the ability to converge more and more datacenter traffic on Ethernet is one of the primary drivers for ongoing innovation of the technology.

Center of the bandwidth storm
Today, we are seeing strong growth in the use of Ethernet to accommodate the exploding bandwidth needs that datacenters are under pressure to meet.

After years of 1 Gbit/s server interconnections as the mainstay in datacenters, conversion to 10G server interconnections and evolution to 40G and 100G Ethernet infrastructures are coming soon. Furthermore, the Ethernet ecosystem is already making longer-term plans for 400G Ethernet links among datacenters.

No less than a bandwidth tsunami has been unleashed with the recent proliferation of innovations such as smartphones, tablets, WiFi, 3G/4G/LTE mobile deployments, 10G servers, Internet-enabled TV, the cloud, social media, video calling, online gaming, and new database technologies. The datacenter, of course, is the epicenter of the storm.

In 2010, an amendment was made to extend the IEEE 802.3 standard for Ethernet to support operating speeds of both 40G and 100G. Thus 802.3ba became the first standard to define two new Ethernet speeds. This paved the way for a new generation of high-rate server connectivity and core switching while maintaining maximum compatibility with the installed base of interfaces.

Innovation continues with datacenters seeking higher-density, lower-cost, and lower-power solutions. IEEE P802.3bm, the draft standard for Ethernet for 40G and 100G over fiber-optic cables, is being written to define:

  • 100G physical layer specifications and management parameters, using a four-lane electrical interface for operation on multimode and single-mode fiber optic cables, reducing fibers per link, cost, and power and increasing the density of 100G applications in the datacenter
  • Optional Energy Efficient Ethernet, a technology that lowers power consumption for systems based on utilization, for 40G and 100G operation over fiber optic cables
  • 40G physical layer specifications and management parameters for operation on single-mode fiber optic cables of extended reach, which would support adoption for metropolitan applications, simplify networks, and entail a low risk of change as datacenters grow beyond their conventional brick-and-mortar structures into regional facilities

Other standards in development include a draft to expand IEEE 802.3 Ethernet to 100G operation on backplanes and copper cables and to specify optional EEE for 40G and 100G.

In April 2013, the IEEE launched a study group to explore the development of a 400G Ethernet standard to support the exponential network bandwidth growth. Again, it's a market-driven effort designed to improve the 802.3 Ethernet standard in order to stay ahead of the industry's needs. Two years of open efforts to assess the Ethernet market's emerging application requirements showed that 400G Ethernet strikes the correct balance among cost, power, density, and bandwidth demand considerations as the next speed step for IEEE-defined Ethernet applications.

The history of Ethernet has shown that standards development, technology innovation, and market growth are closely linked in a constantly spinning cycle. Understanding where the technology is headed will help datacenters make better use of Ethernet for datacenter connectivity today while keeping them well positioned for the future.

Patrick Strick is the technical chair of the Data Center Subcommittee of the Ethernet Alliance and a technical marketing engineer in NetApp's Data Center Platforms group.


Ethernet everywhere

Patrick, thanks for all of this information about Ethernet and its development. I'm sure Bob Metcalfe never imagined it would become so important to so many people! Are there any additional technologies or applications you predict will begin using Ethernet that have not traditionally been transported over the protocol in the past?

Skip 40?

Do you see datacenter engineers jumping direct from 10Mbps Ethernet to 100 Mbps, skipping 40? Seems like the price delta is small enough that it will make sense in some instances.

Two versions of RDMA over Ethenet: iWARP and RoCE

Patrick, very nice overview!  I know you were trying to be concise, but if you mention the capability of running Remote Data Memory Access (RDMA) protocols over Ethernet, you should note there are two software-compatible, but competing technologies that offer that capability, iWARP and RoCE.  Both are in production today.  In a nutshell, iWARP is built on top of TCP/IP and is thus inherently routable.  RoCE uses an InfiniBand transport over an Ethernet wire.  It cannot take advantage of TCP/IP capabilities like congestion management and in its current incarnation is resticted to an an L2 subnet in a singe data center, i.e., not routable.  There is no inherent reason in the protocols why either should have a significant performance advantage over the other.

David Fair, Intel

Re: Two versions of RDMA over Ethenet: iWARP and RoCE

Thank you David. I did not intend to leave one out in favor of the other and appreciate you filling in the gaps on them both for us!