All I Want For Christmas Is 10Gbase-T

Dear Santa: While I wouldn’t be upset to find a Mercedes SLK350 with a big ribbon on it parked in front of my apartment on Christmas morning, the best present you, and the networking industry, could give me is general availability of 10GBase-T across switches and NIC/CNAs. Once I can just order my 10gig network gear with 10Gbase-T, I can stop worrying about how to pay for optical transceivers at $350 to $1,200 a pop, as well as about the never ending compatibility problems with both optics and twin-ax direct connect cables.

Since 10Gbase-T runs over plan old twisted pair cable, there’s no way for a switch or CNA to interrogate the cable and reject it because it wasn’t blessed. Sure, you have to make sure your cable plant is up to the higher data rate. In general, that means Cat6a unshielded twisted pair or Cat 7 shielded cable, though you can use Cat 6, or even Cat 5E, patch cables for short distances (30 meters or less).

Just as I was feeling that I had found a solution to the 10 Gigabit Ethernet cable conundrum, my friend Greg Ferro blogged that he thought twisted pair cable was a mistake for 10 Gigabit Ethernet. He pointed out four problems he had with Cat 6 for 10 Gigabit Ethernet traffic.

Greg’s first complaint and the biggest real down side to 10GBase-T is power consumption. Even with today’s 40nm PHY chips, driving a 10Gbase-T port will take 2 to 5 watts, where a twin-ax DAC cable or 10Gbase-SR transceiver draws less than a watt. Since I’ve spent much of my career working in data centers on the 34th floor of New York office buildings, I’m sensitive to the power limitations many data centers face.

But before you write off 10Gbase-T as a major cause of global climate change since it uses four to five times as much power, remember that we’re talking about 5 watts in the context of a server that’s probably drawing 300 to 500 watts. Unless you’re designing a system for a submarine, or one of my client’s over-populated Manhattan data centers, 10GBase-T isn’t going to bust your power--or your financial--budget.

Assuming a worst-case situation, where a 10GBase-T link would use 10 watts more than a fiber link, that’s only an additional 7.3 KWh per month. We use about as much power each year to cool our lab as the servers and storage inside the lab use and pay about 18 cents per KWh. At those rates, a 10-watt link will cost us about $30 more a year to run while costing $700 less. Since most 10Gbase-T equipment will run in a roughly 2-watt mode for short cables (up to 30M, which should cover most server-switch connections), we’re talking about a few dollars a year to run 10Gbase-T on a server.

Greg then objects to the size of Cat 6A cables. He has a point if we’re comparing Cat 6A to fiber. After all, the 48-strand fiber cable that would serve a full rack of servers is just 0.6 inches in diameter, where a 48-cable bundle of Cat6A cables could be over 3 inches in diameter. From where I sit, it makes more sense to compare Cat 6A not to fiber but to the Cat 5E that it’s probably going to replace. Most Cat 5E cable is between 0.2 and 0.25 inches in diameter, where the Cat 6A standard allows cable to be up to 0.35 inches. However, vendors including Panduit, which Greg references in his post, and Belden are now making Cat6A cables that are less than 0.3 inches, so that a 48- cable bundle will be about 2.5 inches. Yes, bigger, but considering that we’re taking a typical server from 5-9 Cat 5E cables for 1GBase-T to three--two data and one IPMI management--with cables in the 10 Gigabit era, the total space twisted pair cables will take up will be less than we’ve used in the recent past.

Greg says, "This leaves the problems of mechanical and electrical performance over time. The basic problem I have is that Cat6A is close to the limit of what can be achieved with copper in terms of electrical performance. It was originally believed that 10GBaseT wouldn’t even be possible."

Now, Greg has a point that eventually we just won’t be able to squeeze higher and higher data rates through a twisted pair cable and 8 by 8 modular jacks. I myself once predicted that we’d never see Gigabit Ethernet. Since collision detection requires that the first bits of a packet reach the most distant node before the last bits are transmitted, a gigabit network could only be a few meters end to end. I was of course proven wrong as we adopted switching, and collision detection became a historical artifact.

However, I remember hearing almost exactly that same "twisted pair isn’t up to carrying our really cool new fast tech" line when Synoptics introduced LattisNet Ethernet over twisted pair. People were so concerned that two twisted pairs couldn’t securely carry 100Mbps Ethernet that a four-pair standard 100Base-T4 was ratified just in case. The pattern during the past several Ethernet speed upgrades has been for fiber to come first and for truly widespread acceptance to occur when twisted pair made it cost-effective.

Greg worries that twisted pair cable is delicate, that the cable is weak where it enters a modular plug (more frequently referred to as RJ-45), that over-insertion of cables during cable assembly can cause problems, and that a kinked cable can cause data reflections. Having spent 20 years in data centers where twisted pair was the norm, I’ve always had the impression it was fiber, not twisted pair, that was fragile and hard to terminate. Maybe Greg thinks it’s better that the fiber in an optical cable snaps when slammed in a cabinet door so the problem is permanent not intermittent. I’ll just dress my cables so they can’t slam in the door.

Finally, Greg quotes a no-longer-available blog post that implied that 10Gbase-T has lower bit error rate specs than optical or DAC connections. I spent some time researching this and found not only that all 10 Gigabit Ethernet PHYs (physical layer implementations), and Fibre Channel have the same worst case BER of 1 in 10^12. My research shows real world BERs of more like one in 10^15 in the field. While some Fibre Channel folks have lobbied for higher standards, they’ve never made it into the spec.

When 10 Gigabit Ethernet becomes standard on servers, it will almost certainly be 10Gbase-T because 10GBase-T ports can auto-negotiate down to 1GBase-T. Server vendors can sell 10GBase-T servers into data centers that haven’t yet upgraded their network gear but would have a smaller market for SFP+ on the motherboard. Now, if there were 10Gbase-T SFP+ modules like there are for 1GBase-T ... But that would just get us into SFP+ checking again.

Disclaimer: I really do believe in fat guys with beards that give my kids presents.