FCoE: It's The Network And It Ain't There Yet

Now that FCoE products are making their way to real customers, hopefully for use in dev and test environments, we're moving uphill to what Gartner's dubbed the "Peak of Inflated Expectations" where users and pundits alike imagine that the new technology will solve all their problems. While the FC part of FCoE has been adopted by T11, the underlying enhancements to Ethernet still have to make it through the notoriously ponderous IEEE.

The best example of both an over-hyped technology and the futility of competing with Ethernet is ATM.  In the 1990s, IBM, Fore and a parade of analysts predicted that asynchronous transfer mode would replace Ethernet and Token Ring in the LAN, and it would replace frame relay, X.25 (yes I'm that old), and even TDM for voice over the WAN. ATM would be the only networking technology any organization would ever need. One of my clients bought enough ATM switch gear to equip a small country, using ATM backbones to Ethernet switches connecting hotel rooms with Internet service.

ATM had its day as a WAN technology, while Ethernet remained the LAN standard. Carrier Ethernet is becoming the WAN standard. As Bob Metcalf, one of Ethernet's inventors, once said "I don't know what kind of network we'll be running in the future but we'll call it Ethernet." In fact today's 10Gig, full duplex, switched, optical Ethernet with QOS and EIEIO is a far cry from the garden hose-size coax cable we used for the first commercial 10Base-5 Ethernet installs.

Data Center Bridging, Converged Enhanced Ethernet and Cisco's Data Center Ethernet standards are in the same state the 802.11n standards were when "pre-standard" products started appearing a few years ago. The standards' conformance are close but not close enough for vendors to release products that may not be updateable to the standard via simple firmware changes. So, we have FCoE switches from Cisco and Brocade, but no downstream DCB switches from HP, Extreme or the rest of the Ethernet guys.

Today's Ethernet simply discards packets when links, or switch ports, get congested relying on higher layer protocols like TCP to retransmit the dropped packets. DCB adds several features to manage end to end congestion to eliminate this source of packet loss.  The designers of FCoE decided it was better to have a server or array hold its data if the path wasn't clear than wait for an acknowledgement of timeout and retransmission. Building a mostly lossless network will make storage access more deterministic, but it means FCoE needs switches that support DCB.From where I sit DCB isn't enough. Layer 2 Ethernet networks use the Spanning Tree Protocol to keep packets from going around a loop of interconnected switches forever.  Spanning tree does this by shutting down all but one path between any two switches. That means a switch with a pair of redundant uplinks can only use the bandwidth of one of those uplinks.

The Spanning Tree protocol, even the updated Rapid Spanning Tree version takes WAY too long to converge and select a formerly disabled path when a link or switch goes down. It also frequently results in data taking a path through several switches because a shorter, more direct, path is disabled.

We need a new bridge path management protocol that makes better use of all the available paths and reacts faster to link and switch failures. Our best hope, TRILL (Transparent Interconnection of Lots of Links), is now being developed in an IETF working group lead by Radis Perlman of Cisco fame. An earlier effort IEEE 802.1aq seems to have ground to a halt due to inter-vendor politics.

FCoE has a lot of potential. Running Fiber Channel Protocol over the same net as everything else should let storage users capitalize on the lower costs that come from the huge number of Ethernet switch ports sold each year. Realizing that promise, and not just reducing the spaghetti pile behind our servers, will require the hard work of finalizing the standards for DCB and TRILL so networking, not just storage, vendors make switches that can carry FCoE traffic.