The 10 Gigabit Ethernet Cable Conundrum

As we enter the 10 Gigabit Ethernet era in the data center, we're facing some difficulties regarding cables and optics. While I'm happy all the vendors have settled on SFP+, eliminating the nightmare of figuring out if a given device uses XFP, Xenpack or X2 optics to make 10 Gigabit Ethernet affordable, we're going to have to use copper cables for the 10-Mbps or shorter connections within the data center. I just wish that was as easy as grabbing a cable that meets industry standard specs and plugging it in.

As I brought my Brocade 8000 switch up and started using it, I’ve had a few hiccups that led me to believe things weren’t as simple as I hoped. Frankly, vendors are at fault. 10-Gbps twinax Direct Attach Copper (DAC) cables seemed to be the answer. When Brocade so kindly lent me the switch, it provided two Brocade-branded cables and a handful of 10G-SR optics. Not wanting to be a piker, I ordered additional cables from Amphenol Cables On Demand, a major vendor.

When I tried using said cables, the switch said they were invalid SFPs. A little research (through a whitepaper) showed that Brocade supports only Active DAC. I had bought passive. After all, Passive DAC cables were $100 each and the Active DAC cables were $150 each. I then read the Brocade 8000 FAQ, which says that the 8000 supports only Brocade-branded SFPs. I read this to mean that if I bought Molex active cables--which by the way are what Brocade sells, though Brocade has Molex change the identifier to Brocade--they wouldn’t work, either.

Further research has shown Emulex and Qlogic CNAs will make a best effort with whatever you plug into them, but my Intel X520 NIC’s driver crashes if I plug a Brocade optic in it. EMC seems to also enforce cable branding, as blogs indicate you can’t connect an Fibre Channel over Ethernet Clariion to a Cisco switch with Cisco’s passive cable. Things have gotten so complicated that vendors like EMC are reduced to building compatibility matrices for their products and the switches and CNAs they OEM from other vendors, like this one created by Erik Smith of EMC.

If switch vendors enforce using their own cables, building any multivendor network will be an expensive proposition. Connecting the Dell, HP or IBM switch built into your blade chassis to your top-of-row switch might require not a $150 DAC cable but a $300 to $1,200 optic at each end, plus a $100 OM4 fiber cable.

Even if vendors' switches complain but send data anyway, as Cisco switches apparently do, users that, like my friend Stephen Foskett, stick to the HCL will be spending thousands of dollars on fiber transceivers that could go to SSDs, server memory and other things that will actually matter. Once they decide to use optics, users will end up with little boxes of spare optics for each of their switches, occasionally delaying projects or repairs to order the right brand of optics when they have competitors' modules in the drawer.

All of which is even more annoying when you consider that most vendors don’t actually make optics or SFP+ DAC cables. They buy them from OEM vendors like JDSU and Finisar. The ultimate answer may be to live with 10Gbase-T’s higher power consumption and pray vendors can’t figure out a way to identify twisted pair cable by brand.

Disclaimer: Brocade and Emulex are clients of DeepStorage.net, though that might change after they read this blog post. Intel, Qlogic and, frankly, most 10-Gbps Ethernet NIC/CNA vendorshave provided cards for use in the DeepStorage lab free of charge.