Brocade, Cisco, End-to-End FCoE And Who's On First
November 17, 2010
I took some heat, in a good natured way, about my article Brocade First To Market With Native End-to-End FCoE . Brad Hedlund, Brian Gracely, and Stu Miniman contend that Cisco is first with end-to-end FCoE. In my original story, the headline "Brocade First To Market With Native End-to-End FCoE" was inaccurate. After talking to Brocade and Cisco in depth, the more accurate title is "Brocade First To Market With End-to-End Ethernet FCoE." I was sort of right and sort of wrong. Let me explain.
In July 2010, Cisco announced that its Nexus 5000 family was the first to support end-to-end FCoE. While Cisco uses FCoE at the edge and FC between switches, this is compliant with Fibre Channel standards because FC-BB-5, the standard defining FC over a different layer-two protocols, doesn't say that the end-to-end connection has to be FCoE. The standard only specifies how FCoE should work when it is used. Vendors are free to do whatever they want between switches, provided that FC requirements such as lossless operation are preserved and that FC frame order is maintained. In this case, Cisco's claim to be first with end-to-end FCoE is correct, but they sprinkle some FC in the middle.
In the next maintenance release of the Nexus 5000 NX-OS, Cisco will support FCoE layer 2 links between switches. The feature is available today, but it is an unsupported configuration which customers can use to trial FCoE. But even in that case, the Nexus 5000, which is a Fiber Channel Forwarder, will still have to de-capsulate the FCoE frame, process it as FC, and encapsulate it back into FCoE before sending it along to its next hop. J. Michel Metz, FCoE program manager for Cisco, says the encaps/decaps is necessary for FC processing. I am taking his word on it. I have hit the end of my understanding at the moment. Hah.
Brocade is correct in saying that VDX is the first Ethernet end to end FCoE product suite. Brocade's architecture--and some of this will have be figured out once product is shipping--works like this. Instead of processing each frame through a fiber channel forwarders (FCF) at each hop, Brocade uses a Virtual Chassis Switch (VCS) to distribute the intelligence among all the switches. Once the route decision is made at the VDX network ingress point using Fabric Shortest Path First (FSPF)--which Cisco also uses on the Nexus 5000--the frame is sent along. This appears to be standard operating mode in light of FC-BB-5 (clause 7.5 in particular).
Brocade also does some magic by using equal-cost links at a more granular level than per flow. Remember that FC is just SCSI frames running over a network and FC assumes that the fabric is lossless, has low latency and jitter, and frames always arrive in order. Ethernet is the opposite. Loss due to congestion is built in by design while handling delay, jitter, and order is left to upper layer protocols, if needed. Cisco's Nexus 5000 uses equal cost multi-pathing at the flow level, ensuring that each FC or FCoE frame in a flow--a distinct connection between two nodes--always traverses the same path, or in the event of a failure, an equal cost path. This ensures that order is maintained. Brocade's VCS uses multiple paths at the frame level, meaning the same flow can be distributed across multiple links, which Brocade does for FC as well. That means Brocade has to ensure that frames are sent to the end node in the proper order, because mis-ordered frames in FC can cause corruption.