Special Coverage Series

Network Computing

Special Coverage Series

Commentary

Brent Salisbury
Brent Salisbury

Inside Google's Software-Defined Network

Google shared details on its production use of OpenFlow in its SDN network at this spring's Open Networking Summit.

The concept of software-defined networking has captured the attention of network engineers and the trade press, but very few examples of a live SDN implementation exist. One of those few is Google. The search giant presented details about its SDN network at the 2013 Open Networking Summit. Let's take a look.

Obviously Google operates a massive network. It's so large that a 2010 study by Arbor Networks concluded, "If Google were an ISP, it would be the fastest growing and third largest global carrier. Only two other providers (both of whom carry significant volumes of Google transit) contribute more inter-domain traffic."

More Insights

Webcasts

More >>

White Papers

More >>

Reports

More >>

At the Open Networking Summit, Google Distinguished Engineer Amin Vahdat presented "SDN@Google: Why and How." The talk shared some details about how Google uses a combination of Quagga open source software along with OpenFlow to optimize its data center interconnects. He also shared details about Google's use of OpenFlow within its own data centers. Google calls its SDN network "B4."

Vahdat noted that the growth of Google's back-end (east-west) network is quickly surpassing its front-end user-facing network. This growth is expensive because the network doesn't scale economically like storage and compute do. The operating expense of compute and storage becomes cheaper per unit as scale increases, but this is not the case with the network.

Vahdat laid out Google's rationale for software-defined networking. First, by separating hardware from software, the company can choose hardware based on required features while being able to innovate and deploy on software timelines. Second, it provides logically centralized control that will be more deterministic, more efficient and more fault-tolerant. Third, automation allows Google to separate monitoring, management and operation from individual boxes. All of these elements provide flexibility and an environment for innovation.

At the start of the project, Google built its own switches (see image) using merchant silicon. Google built its own hardware because there wasn't any hardware on the market to fulfill its needs at the time it kicked off the project.

B4 Network Hardware Source: Google
(click image for larger view)
B4 Network Hardware
Source: Google

Vahdat didn't mention if any of the SDN-enabled switches now on the market, which typically involve a vendor firmware upgrade incorporating an OpenFlow agent, would satisfy Google's hardware needs.

However, Vahdat noted that while OpenFlow was not and is still not a perfect protocol, Google will continue to use it for flow instantiation because it will be supported by a variety of vendors. I think this implies Google isn't interested in continuing to roll its own hardware, and that OpenFlow support will be a requirement for future hardware purchases.

Vahdat then went on to discuss Google's SDN migration path, which moved in stages from a fully distributed monolithic control and data plane hardware architecture to a physically decentralized (though logically centralized) control plane architecture.

The hybrid migration for the Google B4 network proceeded in three general stages:

Legacy:

Legacy Hybrid SDN Deployment Source: Google
(click image for larger view)
Legacy Hybrid SDN Deployment
Source: Google

Mixed:

Mixed Hybrid SDN Deployment Source: Google
(click image for larger view)
Mixed Hybrid SDN Deployment
Source: Google

Final:

Hybrid SDN Deployment Source: Google
(click image for larger view)
Hybrid SDN Deployment
Source: Google

In the next image, you can see that Google has deployed sets of Network Controller Servers (NCSs) alongside the switches. The NCSs contain the extracted control plane for some number of network elements. The switches run an OpenFlow agent with a "thin level of control with all of the real smarts running on a set of controllers on an external server but still co-located," said Vahdat. The NCSs are 32-core servers.

B4 Architecture Source: Google
(click image for larger view)
B4 Architecture
Source: Google

On top of the NCSs are OpenFlow controllers running leader election for high-availability failover. The primary application Vahdat discussed was a traffic engineering application that instantiates policy into control protocols, including BGP, ISIS and OpenFlow.

My Thoughts

I was interested to hear about Google's hybrid implementation strategy, as I am involved with a 1,000-port production implementation in the enterprise.

It seems to me that Google is fairly limited in how it can react to the loss of network state and mechanically continue forwarding packets. Presumably the network falls back to low-priority, proactively instantiated flow rules. A packet that did not match a flow rule in the flow tables (e.g. table-miss event) could be drained into either a normal forwarding pipeline or a set of pre-installed, catch-all flow rules to reach an egress gateway. While custom forwarding would fall back to a shortest path or a static path, it would keep traffic forwarding until the control elements recovered.

The diagrams imply control elements arranged in a multilayer hierarchy. Hierarchy and modularity is how we scale a large network today. By chunking portions of the NIB (network information base) into modules, Google's approach resembles today's network architectures minus a dedicated control plane per data plane. These modules can then distribute the hashing tables and NIB throughout all the modules and provide global views at the aggregations.

Unfortunately, Vahdat was limited on time and did not go into details around state distribution and redistribution between the control protocols. If you'd like to see the talk for yourself, it's available on the Open Networking Summit website. It's free to watch, but registration is required to view the recorded sessions.

Brent Salisbury, CCIE#11972, is a network architect at a state university. You can follow him on Twitter at @networkstatic.



Related Reading



Network Computing encourages readers to engage in spirited, healthy debate, including taking us to task. However, Network Computing moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing/SPAM. Network Computing further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | Please read our commenting policy.
 

Editor's Choice

Research: 2014 State of Server Technology

Research: 2014 State of Server Technology

Buying power and influence are rapidly shifting to service providers. Where does that leave enterprise IT? Not at the cutting edge, thatís for sure: Only 19% are increasing both the number and capability of servers, budgets are level or down for 60% and just 12% are using new micro technology.
Get full survey results now! »

Vendor Turf Wars

Vendor Turf Wars

The enterprise tech market used to be an orderly place, where vendors had clearly defined markets. No more. Driven both by increasing complexity and Wall Street demands for growth, big vendors are duking it out for primacy -- and refusing to work together for IT's benefit. Must we now pick a side, or is neutrality an option?
Get the Digital Issue »

WEBCAST: Software Defined Networking (SDN) First Steps

WEBCAST: Software Defined Networking (SDN) First Steps


Software defined networking encompasses several emerging technologies that bring programmable interfaces to data center networks and promise to make networks more observable and automated, as well as better suited to the specific needs of large virtualized data centers. Attend this webcast to learn the overall concept of SDN and its benefits, describe the different conceptual approaches to SDN, and examine the various technologies, both proprietary and open source, that are emerging.
Register Today »

Related Content

From Our Sponsor

How Data Center Infrastructure Management Software Improves Planning and Cuts Operational Cost

How Data Center Infrastructure Management Software Improves Planning and Cuts Operational Cost

Business executives are challenging their IT staffs to convert data centers from cost centers into producers of business value. Data centers can make a significant impact to the bottom line by enabling the business to respond more quickly to market demands. This paper demonstrates, through a series of examples, how data center infrastructure management software tools can simplify operational processes, cut costs, and speed up information delivery.

Impact of Hot and Cold Aisle Containment on Data Center Temperature and Efficiency

Impact of Hot and Cold Aisle Containment on Data Center Temperature and Efficiency

Both hot-air and cold-air containment can improve the predictability and efficiency of traditional data center cooling systems. While both approaches minimize the mixing of hot and cold air, there are practical differences in implementation and operation that have significant consequences on work environment conditions, PUE, and economizer mode hours. The choice of hot-aisle containment over cold-aisle containment can save 43% in annual cooling system energy cost, corresponding to a 15% reduction in annualized PUE. This paper examines both methodologies and highlights the reasons why hot-aisle containment emerges as the preferred best practice for new data centers.

Monitoring Physical Threats in the Data Center

Monitoring Physical Threats in the Data Center

Traditional methodologies for monitoring the data center environment are no longer sufficient. With technologies such as blade servers driving up cooling demands and regulations such as Sarbanes-Oxley driving up data security requirements, the physical environment in the data center must be watched more closely. While well understood protocols exist for monitoring physical devices such as UPS systems, computer room air conditioners, and fire suppression systems, there is a class of distributed monitoring points that is often ignored. This paper describes this class of threats, suggests approaches to deploying monitoring devices, and provides best practices in leveraging the collected data to reduce downtime.

Cooling Strategies for Ultra-High Density Racks and Blade Servers

Cooling Strategies for Ultra-High Density Racks and Blade Servers

Rack power of 10 kW per rack or more can result from the deployment of high density information technology equipment such as blade servers. This creates difficult cooling challenges in a data center environment where the industry average rack power consumption is under 2 kW. Five strategies for deploying ultra-high power racks are described, covering practical solutions for both new and existing data centers.

Power and Cooling Capacity Management for Data Centers

Power and Cooling Capacity Management for Data Centers

High density IT equipment stresses the power density capability of modern data centers. Installation and unmanaged proliferation of this equipment can lead to unexpected problems with power and cooling infrastructure including overheating, overloads, and loss of redundancy. The ability to measure and predict power and cooling capability at the rack enclosure level is required to ensure predictable performance and optimize use of the physical infrastructure resource. This paper describes the principles for achieving power and cooling capacity management.