Datacenter Ethernet: Know Your Options

Ethernet is a living, breathing architecture that is constantly evolving for developing needs and applications. In addition to its roadmap for networking data rates, the IEEE 802.3 family of standards provides a set of physical layer connectivity options to help datacenter managers leverage Ethernet as their plumbing infrastructure.

The workloads of some applications do not require high-performance central processing units (CPUs), allowing an ever-increasing number of "micro" servers to be physically small and more densely packed in a rack. On the other hand, some applications require the highest-performance CPUs possible, and innovation still occurs to maximize the compute power available in a single processor complex. 

These application extremes, and the wide variety in between, all have different physical reach needs and differing cost, power, and ease-of-use requirements. Out of this multi-dimensional diversity has developed a menu of physical layer connectivity options.

Massive datacenter buildouts by Internet and cloud service providers has driven much of the growth in 10-Gbps port speeds for server connections. Twin-axial cable (twinax) links -- in the form of SFP+ "direct attach cable" -- have provided the right balance of quick time to market and low cost for short-reach links in an environment where backward compatibility is not required. 

This may not be the majority of the server market, but it is perhaps the fastest growing. As a result, much high-end server and network architecture innovation is happening in the hyper-scale or mega datacenters that purchase equipment in such tremendous scale. To serve this market, a standard for 40-Gbps operations over twinax (40GBASE-CR4) is already in place, and a new standard to upgrade these links to 100-Gbps (100GBASE-CR4) is nearing completion.

Very-high-density computing is the new target application for electrical backplanes and micro-server chassis. The aim here revolves around the question, "How many compute sockets can physically fit into a small package?" Backplane connectivity, though first developed for blade servers, is a good fit for micro-server systems, as well. 

Fiber optics are favored for physical layer Ethernet connectivity where longer reach and higher speed are the requirements. Higher speeds tend to arrive first with optical networking; we've seen that repeated in the industry time and time again. Fiber links that run at 100 Gbps are already on the market, with an IEEE 802.3 study group working on 400 Gbps. Fiber-optic solutions are very common for backbone uplinks switching into the next network layer. In addition, there are some customers who just have faith in optics and are willing to spend extra money for optical links.

Active cable assemblies, which tend to cost less than standard modular optics, work well for medium reach, supporting Ethernet transport over longer distances than twinax cables but shorteer than fiber optics. They serve as a kind of gap filler. Datacenter managers tend to make a choice between twinax, fiber or active cable for applications with reach requirements of, say, 5 to 20 meters. The active cables can be much smaller diameter and offer higher density than twinax, and, for some applications, they are worth the extra cost to have less cable bulk.

RJ-45 connections and unshielded twisted pair cabling remain a very popular option for physical layer Ethernet in the "rest of the world." Average customers may not be pushing the performance envelope and likely don't buy an entire roomful of equipment at one time. Instead, they tend to have multiple generations of technology in the same datacenter infrastructure and expand capacity as needed. 

BASE-T has always been a good fit for these environments by providing compatible connectivity among multiple generations of speeds, from 10 Mbps to 10 Gbps. 1000BASE-T is still the highest-volume port type in servers today, with 10GBASE-T growing and 40GBASE-T on the horizon, continuing the trend of multi-generation plug compatibility.

There are always new tools coming down the pike for datacenter architects to use in their infrastructure, and different Ethernet solutions will power future generations of servers for generations of datacenters to come. Through the IEEE 802.3 standardization process, the global Ethernet ecosystem has maintained an environment in which innovation could flourish even while backward compatibility was maintained. Both have been critical to the technology's proliferation and market growth during its 40 years of growth.

David Chalupsky is the technical chair of the Ethernet Alliance BASE-T subcommittee and network hardware architect at Intel Corporation. He is also is chair of the IEEE P802.3bq task force. He has been developing Ethernet products for more than 20 years.

Adam Healey participates in the Ethernet Alliance. He is a distinguished engineer at LSI Corp., where he supports the development of high-speed serial interface products. Adam is vice chair of the IEEE 802.3 Ethernet working group and chair of the IEEE P802.3bj 100 Gbps backplane and copper cables task force.