Data centers

10:00 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Facebook's Data Center: Where Likes Live

Welcome to the Oregon high desert, where Facebook stores all of your likes while pursuing data center energy efficiency on a new scale. Coming soon to the neighborhood: Apple.

To highlight this openness, Facebook sponsored a tour of Building Two recently, led by Crass. DPR Construction, which builds data centers and other advanced buildings for financial services, healthcare and technology companies, said the Facebook example is having an impact. "Facebook has taken the lid off the secrecy about how to bring power and cooling into a modern data center," Andres said. Its example is being copied by other leading data center builders.

Who, you might ask? Standing on the roof of the Prineville facility, Crass looks to the south and can see another mega data center going up next door. At 338,000 square feet, it appears to resemble Facebook's. It's being built by Apple.

One of the keys to power conservation in the mega data center is cooling by evaporation, rather than air conditioner compressor. There are no industrial air conditioners -- chillers, they're called in data center circles -- in the Prineville facility. Some advanced data centers, such as those built by Vantage in Silicon Valley, install chillers just in case for hot spells. But in Prineville, the dry desert air is a boon to evaporation, and Facebook has built what has to be one of the most massive air-movement systems in the world to get its data center servers to the temperature that it wants.

The air warmed by running disks and servers rises into a plenum above the equipment. It's siphoned off to mix with cooler outside air, pushed through a deep set of air filters that look like square versions of the round air filters that used to sit atop a car engine's carburetor -- a thick, porous, soft corrugated paper.

The air is then pushed through something called Munters Media, an absorbent cellulose material that absorbs a small amount of water trickling down its vertical, wavy slats. The dry air cools as it absorbs some of the moisture, and Facebook's sensitive equipment faces a reduced risk of static electrical shock as the outside air is cleansed, cooled and humidified. A product of a Swedish company, Munters Media was used by European farmers for simple cooling systems based on fans and air flow to cool poultry and cattle before being drafted into use in advanced data centers.

Server fans draw the cool air over server motherboards that have been designed to be long and narrow rather than the common rectangular shape. Components are arranged to encourage air flow, and unnecessary components, like video and graphic processors, are stripped away. Instead of memory chips acting as dams to the air flowing over the hot CPU, they are aligned in parallel with the direction of air flow. Warm air from two rows of servers, standing back to back, is exhausted into a shared hot aisle, rises to the plenum, and is either pushed out of the building or sent back to the ambient air mixing room to restart its journey.

A Facebook Server Suite

Crass, pictured here in one of his "server suites," refused to pump up the processes that he supervises. At the heart of it, he said the building management system is "deciding how much to open the dampers to bring in the cool air, putting it in the data center and getting rid of the hot air." It's also pushing air through banks of filters, controlling the valves that let water flow to Munters Media, monitoring the humidity as well as temperature of the data center air, and adjusting the speeds of six-foot fan blades for exhausting air from the last of a row of rooftop chambers.

The building's design lets fans and passive systems accomplish most of the work. Once humidified and cooled, the air flows down a nine-by-nine foot chute built between the filters and Munters Media rooms to the cold aisle of data hall below. As it falls, it hits a big deflector plate at ceiling level of the cold aisle and is spread out over the tops of running servers, where it's drawn across the warm server motherboards.

In earlier data center design, "the cool air flowed down to the floor of the data center. That didn't work as well. It didn't scatter as far and you need it at the top of the server rack, not just the bottom," said Crass.

The air-flow cooling also works because of the cool temperatures in the high desert. The hottest days of July and August will not go above 86 degrees, on average. As the Prineville complex was being designed, the American Society of Heating, Refrigeration and Air Conditioning Engineering raised the acceptable temperature for air used to cool computer equipment to 80.6 degrees. The conventional wisdom held that it was better to run computer equipment with air-conditioned air that might be in the low 60s or even 50s as it emanated from the chiller.

The Facebook data center runs hotter than many enterprise data centers at the peak of the summer, but its servers are not enclosed in metal cases; rather they sit on simple, pull-out trays in server racks that maximize the air flow over their components.

Crass thinks electrical equipment in his data hall can be adequately cooled with 85-degree air, given the powerful air flows possible with the building's design, and Facebook engineers have raised the upper margin of the cold aisle to that limit without ill effects. The higher the operating temperature, the less energy that needs to be poured into pumps for moving water and cooling fans.

Previous
2 of 3
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
timwessels
50%
50%
timwessels,
User Rank: Apprentice
3/7/2013 | 3:11:39 AM
re: Facebook's Data Center: Where Likes Live
Well, another good article by Mr. Babcock, who is one of the best tech writers in the business. If you want to see the inside of facebook's Prineville, Oregon, data center, there are a number of videos about it on YouTube. http://tinyurl.com/cg9rmgy
John Foley
50%
50%
John Foley,
User Rank: Apprentice
3/6/2013 | 6:15:31 PM
re: Facebook's Data Center: Where Likes Live
If you like data centers, a trip to Oregon is recommended. I've been to Google's data center in The Dalles and to Amazon's facility in Boardman, though I observed them from the outside not inside. Flying into Portland, driving along the Columbia River Valley where the wind turbines are spinning, and a side trip to Mt. Hood make for a great few days. I also rafted and fished in the Deschutes River. Land, sun, water, and wind -- you can see why they build data centers there.
fmann-craik950
50%
50%
fmann-craik950,
User Rank: Apprentice
3/6/2013 | 5:47:31 PM
re: Facebook's Data Center: Where Likes Live
Interesting article that gives Facebook an environmentally responsible "face." Thanks for sharing this in-depth info Charles.
D. Henschen
50%
50%
D. Henschen,
User Rank: Apprentice
3/6/2013 | 5:26:47 PM
re: Facebook's Data Center: Where Likes Live
This account really demonstrates where the rubber of our digital lives meets the road. Great to see so many green initiatives coming together. It's pretty obvious that environmentalism and good old-fashioned capitalism -- the drive for efficiency and competitive advantage -- go hand in hand. If Facebook is as willing to share its data center breakthroughs as it has been its Open Compute Project server breakthroughs, the entire industry will benefit.
Hot Topics
7
Have You Hugged Your Sysadmin Today?
Susan Fogarty, Editor in Chief,  7/25/2014
5
Fall IT Events: On The Road Again With 10 Top Picks
James M. Connolly, Editor in Chief, The Enterprise Cloud Site,  7/29/2014
4
Where Is Your Cloud?
Andrew Froehlich, President & Lead Network Architect, West Gate Networks,  7/23/2014
White Papers
Register for Network Computing Newsletters
Cartoon
Current Issue
2014 Private Cloud Survey
2014 Private Cloud Survey
Respondents are on a roll: 53% brought their private clouds from concept to production in less than one year, and 60% ­extend their clouds across multiple datacenters. But expertise is scarce, with 51% saying acquiring skilled employees is a roadblock.
Video
Slideshows
Twitter Feed