MITRE Data Center Retrofit Is a Model of Efficiency

MITRE Corporation wants other organizations to know that retrofitting their existing data centers to handle higher densities might well be cheaper than building a new data center or using commercial colocation or cloud computing.

The company conducted a thorough study of all the options before choosing to deploy a refrigerant-based cooling system, 415-volt power and other upgrades in one of its data centers. The resulting High Performance Compute Center (HPCC) has operational costs 40% to 60% lower than commercial colocation services. MITRE's innovative solution won second place in the InformationWeek IT Excellence infrastructure category.

MITRE describes itself as nonprofit  that operates multiple federally funded research and development centers. It works closely with government agencies related to defense, intelligence, aviation, homeland security, healthcare, and other areas.

A few years ago, the company ran into a problem: it was running out of capacity in its data centers.

"The nature of the work MITRE does over the past six or seven years has increasingly been driven by higher performance analytic capabilities," Jim Treadway, chief engineer of infrastructure, at MITRE, told me in an interview. "That influx of higher performance created a set of conditions where we had exceeded and outgrown the capacity of our legacy, traditional air-cooled data centers."

The initial plan was to consider building a new data center. But with price tags in the tens of millions, that option seemed out of reach for the nonprofit.

Next, Treadway and his team considered what it would cost to run MITRE's workloads in the cloud or a colocation facility. "Neither of those options were particularly cost effective, either," Treadway said.

He explained that while the cloud is good for "bursty" workloads, it was not well-suited to the sorts of round-the-clock analytics that MITRE was running.

With no other attractive options, MITRE began considering how it could retrofit its existing data center in a Bedford, Mass., building that was originally constructed in the 1980s.

After an examination of some high-density computing operations in the telecommunications industry, MITRE decided to deploy a long list of infrastructure upgrades. Two in particular -- refrigerant based cooling and 415-volt power -- proved critical.

MITRE HPCC.jpg

MITRE moved its higher-powered servers, storage, and other computing hardware into new "quasi-custom" cabinets with pumped refrigerant rear door active heat-extraction systems. That system was much more effective at cooling the hardware that could draw 40 to 60 amps of power per cabinet. To deliver that level of power at lower cost, the company made an unusual decision. It switched from the 208-volt power standard that is common in the United States to the 415-volt power standard, "which is very common in Europe and Japan," Treadway said.

In addition, MITRE also made some changes related to its cable management and the fiber-optic standard it was using. "We really focused heavily on cost containment and driving our capacities and densities up and driving our costs down," Treadway said.

MITRE's HPCC became operational in August 2015, and the results have been impressive. The company reduced its cooling costs by 90% and its overall costs by 20%. At the same time, it increased its compute capacity by a factor of ten and reduced floor space by 40%. It also avoided the high capital expenditures that would have been associated with building a new facility.

Impressed by these results, more than 40 companies have toured the HPCC, and according to Treadway, at least three have deployed similar technologies in their own data centers. MITRE hopes to serve as a proof of concept for federal agencies and other organizations looking to reduce their data center costs.