Will RDMA Over Ethernet Eclipse Infiniband?

Remote Direct Memory Access (RDMA) is a technology that allows data to be written from one machine directly into the memory of another system. This bypasses many of the operating systems and network stacks that slow down transfers. A good analogy is a delivery service that delivers from my desk to yours, versus the postal system.

RDMA has long been associated with InfiniBand, and is, in many ways, the only reason for the existence of that link type. Where InfiniBand has been deployed, performance is higher and latencies lower. This makes RDMA attractive in the high-performance computing (HPC) market and in financial services trading systems where time equals money.

Ethernet also has RDMA capability, but InfiniBand currently surpasses it in sales. However, InfiniBand's RDMA dominance may be coming to an end.

To understand the situation, one has to look at how the market for RDMA has evolved. The two major vendors in the space initially were Mellanox, driving RDMA on InfiniBand, and Chelsio pushing RDMA on Ethernet. Mellanox brought the InfiniBand approach to market well before Chelsio had product, and established a monopolistic position delivering RDMA over InfiniBand.

InfiniBand effectively saturated the market, which at the time was limited, and this locked out the competing Ethernet product to a great extent. Customer inertia and good engineering have maintained the InfiniBand status quo for a long time.

Mellanox, however, hedged its bets some years ago by designing in the ability to run either InfiniBand or Ethernet protocols with the same ASIC. This strategy has matured to the point that there are now two alternatives for Ethernet RDMA: RoCE (RDMA over Converged Ethernet) from Mellanox and Chelsio's iWARP.

Intel also entered the iWARP space with the purchase of NetEffect, but its latest "Fortville" NIC chips appear to have dropped iWARP support, leaving Chelsio as the sole supplier. RoCE has gathered up Emulex as a second source and others appear to be joining the bandwagon, while RoCE is being adopted by OEMs, including Dell.

From an industry standards viewpoint, there are recognized organizations abstracting the design from the vendors' hands in both cases. RoCE has the InfiniBand Trade Association behind it, while iWARP has IETF.

It's fair to say that RDMA over Ethernet is still in the early adopter stage, but to understand its future one has to look at some trends in the wider industry.

First, we are seeing Ethernet overtake all of the other protocols in terms of performance and options. There are 100 GbE products already available, and 10GbE, the low-cost workhorse of the data center, is poised to move to 25 GbE without extensive rewiring. Even though InfiniBand has also achieved 40 Gbps and 56 Gbps rates (and in fact, beat Ethernet to those goals) the cost of maintaining parallel support is significant.

Second, we are shifting to an era of software-defined networking. This implies a convergence to an Ethernet solution, and likely would marginalize InfiniBand in many situations.

AppliedMicro is adding RoCE to its X-Gene2 multi-core ARM CPU chip, which will make the technology much more affordable and mainstream. X-Gene2 is currently sampling, so we'll see it in volume production in 2015.

Demand for higher performance and low latency is, as Microsoft's commitment to RDMA shows, increasing fast. High-end database architectures are moving to in-memory clustered models, and this requires a close federation of memories to maintain speed. Likewise, flash acceleration of a variety of solutions needs a low-latency, low-protocol solution to moving data between server nodes.

Most storage arrays, especially the all-flash variety, are offering InfiniBand connection options. There is pressure to limit the connectivity options and have more of the same port type, rather than a variety of connectors. This tends to add interest for Ethernet RDMA.

InfiniBand has loyal supporters, and it won't disappear overnight, but Ethernet's ability to deliver almost the same performance, plus network convergence, makes RoCE the likely long-term winner. Mellanox, which provides the industry with InfiniBand switches, could facilitate the transition with a RoCE to InfiniBand router.