Ethernet Innovation Inside The Datacenter

Ethernet has become a ubiquitous interconnect technology to provide communication from machines to machines and machines to humans. With its simplicity and enhanced capabilities, Ethernet provides services to different markets -- from homes, transport networks, and medical machines to wired and wireless networks, telecom networks, and datacenters.  

Within datacenters, Ethernet has emerged as the de facto interconnect fabric to connect servers and networking and storage equipment. Datacenters are quickly ratcheting through the technology’s standardized speed steps to accommodate unprecedented bandwidth demands. After years of 1 Gbit/s Ethernet (GbE) connections linking servers, datacenters are deploying 10GbE and moving to 40GbE today. And real-world deployment of 100GbE is clearly on the horizon.

Datacenter operators are finding inventive ways to cost-effectively migrate up the ladder of the IEEE 802.32012 “Standard for Ethernet” speeds and, along the way, enabling new breakthrough capabilities in datacenters around the globe.

Application-driven innovation
Traditionally, the datacenter serves clients, and most of the traffic is from client-server traffic (also called “north-south traffic”). Due to virtualization, social media, and information sharing sites, server-to-server communication traffic (“east-west traffic”) has increased multifold.

A variety of application trends is converging to drive the surge in east-west traffic -- traffic either within datacenters or from one datacenter to the next. This, of course, has dramatic ramifications for the topology of networking infrastructure and interconnect requirements. In today’s datacenter, high-speed connectivity between servers is a necessity, as evidenced in the global push toward 40GbE deployments today and burgeoning interest in 100GbE, eventually moving toward 400GbE.

There is a multitude of applications fueling demand for east-west bandwidth across datacenters:

New capabilities
Out of these market-driven application trends is growing a host of new Ethernet and other capabilities unlocked by the increased bandwidth for datacenter interconnection.

With today’s migration to 40GbE server interconnects, the industry serving datacenters has developed 4x10G Ethernet breakout cables that package within a single, short-reach module four copper or optical fiber cables, each delivering 10G, for a total of 40G. The same breakout strategy is likely to be used to eventually support 100GbE and 400GbE server interconnects, offering datacenters an appealing and cost-effective migration strategy into the higher speeds.

Another emerging capability that is gaining traction among datacenters because of the deployment of higher-speed interconnects is stateless computing. It's a concept that has been talked about for a decade or more, but never saw widespread deployment among datacenters, largely because interconnect bandwidths were lacking.

In stateless computing, the server is nothing but a central processing unit (CPU) and memory; the unique software configurations and other states of the server are stored elsewhere. The server -- a cartridge-form blade inside a core switch -- performs only computations, leveraging the software configurations and other states from other devices. Stateless computing is a scenario that enhances security and reduces maintenance burden, all predicated on very-high-speed interconnections among servers and the other devices where the other states reside.

The stateless computing model, furthermore, translates into a valuable series of related capabilities. Fabric-attached storage (FAS), for example, can deliver significant management and cost efficiencies, as CPUs are on one side of the fabric and storage is concentrated into a high-availability cluster on the other. Object-oriented storage, often implemented with FAS, also is catching on. Data is managed as objects (instead of files or blocks), with meta-data descriptions of the contents for improved indexing and management, and reduced administration complexity.

Innovation has been nonstop in the IEEE 802.3 Ethernet Working Group, as well. Key developments include: 10GBASE-T, standardized in IEEE 802.3an-2006 to support 10GbE connectivity over unshielded or shielded twisted pair cables of distances up to 100 meters; 40G and 100G backplane Ethernet, simultaneously standardized in IEEE 802.3ba-2010 to simplify complex link aggregation schema and pave the way for a new generation of high-rate server connectivity and core switching; and optional Energy Efficient Ethernet (EEE), an option for reducing power consumption in datacenters based upon systems utilization.

Market-driven Ethernet innovation never stops -- typically inside-out, from core to edge, across the network. Hallmarks of the technology’s 40-plus years of history have always been cost-effective migration paths, simplicity, and advanced capabilities as well as backwards compatibility with legacy implementations. Such characteristics are certainly in evidence in Ethernet’s deployment for server interconnection and Ethernet fabrics inside today's datacenters.

— Shreyas Shah has more than 20 years of experience in designing ASICs, FPGAs, and systems. He has held various engineering positions in design and architecture in wired communications and has focused on datacenters for the last 10+ years. Currently he is a datacenter system architect with Xilinx. Previously he was founder and CTO of Xsigo systems. Xilinx is the member of the Ethernet Alliance.