• 06/18/2014
    3:46 PM
  • Rating: 
    0 votes
    Vote up!
    Vote down!

Facebook Unveils Open TOR Switch

Social networking giant builds its own switch that's designed to operate more like a server for better flexibility.

Facebook took another step towards what it calls a "disaggregated network," introducing an open, top-of-rack switch code-named Wedge running a Linux-based operating system code-named FBOSS.

"What we've done with our switch is take the software and split it into modules and take the hardware and split it into modules," said Jay Parikh, VP of infrastructure engineering at Facebook, in unveiling the switch Wednesday at Gigaom Structure in San Francisco.

The switch, he said, will provide Facebook with lower costs, but most importantly, flexibility. "We will be able innovate on these components and change things at a more rapid rate," he said.

Last year, Facebook launched an effort to develop an open, OS-agnostic top-of-rack switch through the Open Compute Project. The social networking giant started OCP in 2011, focusing on developing open designs for data center infrastructure components such as racks, servers and storage. The switch effort received open switch specifications from Broadcom, Intel, Mellanox, and Accton, while Cumulus Networks and Big Switch Networks contributed software.

Conventional networking based on proprietary hardware and software presented an obstacle for Facebook, Parikh said. "We were running into things that slowed us down and didn't give us the flexibility we wanted," he said.

Parikh said the switch extends Facebook's focus on disaggregration -- breaking down data center technologies into smaller parts that provide more flexibility in handling workloads. Wedge, which is embedded with a microserver module, operates more as a server, he said.



Figure 1:



In a blog post, Facebook engineers Yuval Bachar and Adman Simpkins described Wedge and FBOSS -- which are being tested in the company's network -- as allowing Facebook to leverage its experience in running thousands of servers in its data centers. "In other words, our goal with these projects was to make our network look, feel, and operate more like the OCP servers we've already deployed, both in terms of hardware and software," they wrote.

"By using a real server module in the switch, we're able to bring switches into our distributed fleet management systems and provision them with our standard Linux-based operating environment," they added. "This enables us to deploy, monitor, and control these systems alongside our servers and storage -- which in turn allows our engineers to focus more on bringing new capabilities to our network and less on managing the existing systems."

Facebook plans to propose Wedge and FBOSS as contributions to OCP. Parikh said the design makes it easy for others to change it up as suits their needs. For example, Facebook's initial design has an Intel chip, but someone could use an ARM chip instead, he said.

"We expect lots of people to have ideas on how to make this thing better," he said. "We're not wedded to this, but we need to solve our problems. We can't let our business slow down because we can't find the things we need."



Interesting comment in the Facebook engineering blog post by Bachar and Simpkins: "With FBOSS, all our infrastructure software engineers instantly become network engineers."


Yes, I have heard similar comments about VMware NSX -- that it allows the server guys to do things they used to need to ask the networking team to do. That's probably a good thing from a "breaking down silos" and efficiency perspective. If i were a network engineer, I might find it a bit threatening, though. Any thoughts from readers?


I'm not sure I buy the "all our software engineers become network engineers" line. Do they have a management tool to let them configure network connections and apply policies, or are they just doing things like installing and updating the OS software?


According to that blog post, FBOSS uses the same software libraries Facebook uses to manage its servers: "By controlling the programming of the switch hardware, we can implement our own forwarding software much faster. We also added a Thrift-based abstraction layer on top of the switch ASIC APIs, which will enable our engineers to treat Wedge like any other service in Facebook."


Another thing, to Drew's point, is that even if this is as momentous as Linux emerging against Windows, the network operating system in these cases is a much bigger deal than a desktop or server OS in a typical business environment. Facebook can pull this technology off because they have thousands of network architects and engineers who have been doing this for years. Very few other companies can or would even try to replicate that.


Definitely, this is something only a huge company like Facebook with tons of resources can pull off. Not sure what it will take for the open networking model to work in the enterprise space. 


It will be interesting to see how functional FBOSS itself is, when Facebook releases it via OCP as an open product (if i understand correctly).

If it's an OS with a Thrift API for the network ASIC and not much more, then it's not a lot of use. Or will it come with controller software too?

I'm also assuming that in terms of Wedge itself, Facebook is releasing the design specifications rather than planning to enter the white-box switching market, so even once released, somebody now has to build and sell it (unless you are in the mood to build your own switches as well as code for them?)

It does sound very cool though. If we can understand what this really means to use as consumers of network hardware, that would be nice.


I would also assume that Facebook isn't planning on entering the white-box switching market, but some of the PR language that went with this announcement was interesting: "While existing network switches are currently a 'black box' of proprietary hardware and software, Facebook has separated the two, to bring more choice into the market."


I actually liked the line "all our software engineers become network engineers", although I don't think it would have been intended to mean advanced network engineers.

A somewhat similar comparison might be how someone can build a respectable website without knowing any HTML, CSS, etc.  

independent network operating systems

In his blog post about Facebook's switch, Greg Ferro said the arrival of independent network operating system (NOS) products has parallels with Windows/Linux.

Re: independent network operating systems

Yeah, there is also a lot of bluster out there about Cisco's customers now "suddenly" becoming their competitors. Yes, that is happening, but I think there are not that many companies that have the capabilities to do this type of networking on their own, so Cisco has a lot of the market lefyt to play in. That may change over time, but for the near future it's really the cutting edge, and the action has been shifting toward Google and Facebook for a while.