Brocade, Cisco, End-to-End FCoE And Who's On First

I took some heat, in a good natured way, about my article Brocade First To Market With Native End-to-End FCoE . Brad Hedlund, Brian Gracely, and Stu Miniman contend that Cisco is first with end-to-end FCoE. In my original story, the headline "Brocade First To Market With Native End-to-End FCoE" was inaccurate. After talking to Brocade and Cisco in depth, the more accurate title is "Brocade First To Market With End-to-End Ethernet FCoE." I was sort of right and sort of wrong. Let me explain.

In July 2010, Cisco announced that its Nexus 5000 family was the first to support end-to-end FCoE. While Cisco uses FCoE at the edge and FC between switches, this is compliant with Fibre Channel standards because FC-BB-5, the standard defining FC over a different layer-two protocols, doesn't say that the end-to-end connection has to be FCoE. The standard only specifies how FCoE should work when it is used. Vendors are free to do whatever they want between switches, provided that FC requirements such as lossless operation are preserved and that FC frame order is maintained. In this case, Cisco's claim to be first with end-to-end FCoE is correct, but they sprinkle some FC in the middle.

In the next maintenance release of the Nexus 5000 NX-OS, Cisco will support FCoE layer 2 links between switches. The feature is available today, but it is an unsupported configuration which customers can use to trial FCoE. But even in that case, the Nexus 5000, which is a Fiber Channel Forwarder, will still have to de-capsulate the FCoE frame, process it as FC, and encapsulate it back into FCoE before sending it along to its next hop. J. Michel Metz, FCoE program manager for Cisco, says the encaps/decaps is necessary for FC processing. I am taking his word on it. I have hit the end of my understanding at the moment. Hah.

Brocade is correct in saying that VDX is the first Ethernet end to end FCoE product suite. Brocade's architecture--and some of this will have be figured out once product is shipping--works like this. Instead of processing each frame through a fiber channel forwarders (FCF) at each hop, Brocade uses a Virtual Chassis Switch (VCS) to distribute the intelligence among all the switches. Once the route decision is made at the VDX network ingress point using Fabric Shortest Path First (FSPF)--which Cisco also uses on the Nexus 5000--the frame is sent along. This appears to be standard operating mode in light of FC-BB-5 (clause 7.5 in particular).

Brocade also does some magic by using equal-cost links at a more granular level than per flow. Remember that FC is just SCSI frames running over a network and FC assumes that the fabric is lossless, has low latency and jitter, and frames always arrive in order. Ethernet is the opposite. Loss due to congestion is built in by design while handling delay, jitter, and order is left to upper layer protocols, if needed. Cisco's Nexus 5000 uses equal cost multi-pathing at the flow level, ensuring that each FC or FCoE frame in a flow--a distinct connection between two nodes--always traverses the same path, or in the event of a failure, an equal cost path. This ensures that order is maintained. Brocade's VCS uses multiple paths at the frame level, meaning the same flow can be distributed across multiple links, which Brocade does for FC as well. That means Brocade has to ensure that frames are sent to the end node in the proper order, because mis-ordered frames in FC can cause corruption.Brocade's VDX 6720-based VCS only works within its own set of products. Cisco's Nexus 5000 is in the same boat. Does it matter? It matters when you want to add other products into your FCoE fabric. What it sounds like to me is that Brocade and Cisco have different interpretations of FC-BB-5. I think standards should be adopted in good faith and that vendors come to a common interpretation of the standard and should prove meaningful inter-operation amongst themselves or to quote the IETF credo, "rough consensus and running code". Regardless, standards compliance opens the market place and gives customers choice. But standards purists don't carry a lot of weight with companies that are #1 and #2 in the SAN space. Storage vendors and customers have lived with qualified equipment lists for years. FCoE isn't going to be the leveler.

One of the main problems Brocade has with the VDX 6720 is that it loses visibility into the FC network, which SAN administrators rely on for management and troubleshooting. Brocade admits this is a big miss, and says it will address it in upcoming versions of either SAN Health or LAN Health products which report on traffic statistics and provide troubleshooting tools. Also, the VDX doesn't have any FC ports, nor does the VDX integrate with Brocade's other FCoE switch, the 8000.

If you want to do FCoE the Brocade way, it's all or nothing. Cisco is happy to point this out, although they are in a similar position. With all the finger pointing, someone is going to lose an eye.

Just to put a bow on this monster: two nodes using FCoE connected to a Nexus 2000 Fabric Extender, which is connected to a Nexus 5000, does not constitute end-to-end Ethernet FCoE because the Nexus 2000 Fabric Extender is just a bump in the wire and switching occurs on the Nexus 5000. If you want to call the Nexus 2000 a hop (and you know who you are, Brad), you might as well call the CAT6 cable between them a hop as well. So there. LOL