US Navy Bolsters Mega-SAN

How would you like to develop a disaster recovery plan for one of the world's largest SANs? With multiple Pbytes of storage, a hundred server farms, and half a million users, this is the challenge facing the IT team behind the Navy and Marine Corps Intranet (NMCI).

"It's probably the largest storage system in the world," explained Akira Robinson, a computer scientist working as a civilian consultant on the project, during a presentation at the Storage Decisions event in New York today. "Each user has 100 Mbytes of storage [and] they also have 50 Mbytes of Exchange space."

The massive Intranet provides a Web-based communications infrastructure for 450,000 Navy and Marine personnel, as well as civilians working within the two branches of the U.S. military.

EDS is the primary contractor for the $9 billion project, which it describes on its Website as "the world's largest, most secure private network". Despite a series of performance and payment disputes, the Department of the Navy extended EDS's contract last year, taking the deal through to 2010.

Dell provided the initial 2-Pbyte storage infrastructure for NMCI back in 2000, although NetApp and EMC have since got involved with the project.Precise details of the NMCI project are hard to come by, although the Intranet is said to account for a sizeable chunk of EDS's managed storage infrastructure, which is more than 12 Pbytes.

"There's a SAN that supports this and we make pretty good use of NAS," explained Robinson, adding that this hardware is spread across three network operation centers (NOCs) in Norfolk, San Diego, and Pearl Harbor, as well as more than a hundred server farms dotted around the world.

Just a year into the project, Robinson and his team received a wake-up call when a thunderstorm knocked out the power at one of their server farms in Norfolk, Virginia. "When this happened, I really found myself deep in disaster recovery," he said.

Although only a tiny percentage of users were affected by the outage, the exec realized he had to review the organization's DR plans when an electrician arrived to get the power back on. Because the power panel was also attached to the farm's air conditioning unit, the electrician refused to touch it.

"We were out for another six hours while the air conditioning guy came out," said Robinson. "It really brought home the message that you have got to know what you have got."In an attempt to tighten up its disaster recovery, the NMCI project now relies heavily on the IT Infrastructure Library (ITIL) framework. ITIL, which is growing in popularity, is a set of best practices covering application management, security, and IT service delivery.

"There's a real need to control IT environment complexity and change, and this gives you a framework," explained Robinson, adding that ITIL can also help improve storage management.

To illustrate his point, Robinson explained how the NMCI project is using ITIL to plot its future storage needs. "We have had a lot of users calling up the helpdesk, saying that they are running out of storage," he said, explaining that staff within the organization's 25 "echelon commands" are currently deciding whether a distributed file share (DFS), centralized storage, or a more network-centric approach can solve this problem.

Officials will use the ITIL criteria to check how effectively the chosen solution works with the Navy and Marine Corps' existing infrastructures. "It's a framework that has been successful in the U.S. Army and at Procter & Gamble," said Robinson, adding that a technology can only be chosen when it has unanimous approval.

Although developed in the U.K, a number of other high-profile U.S. organizations have also deployed ITIL. These include General Motors (GM), which is currently in the midst of a $15 billion overhaul of its IT infrastructure, and the U.S. Pacific Air Force (PACAF) Directorate of Communications and Information, which is using ITIL as part of an effort to restructure its technology service desk.Developing a disaster recovery plan for the U.S. military is all about paying attention to the smallest details, according to Robinson. "When you have marines sitting on your change control board and they know that the lives of their brothers in arms are in danger because somebody didn't document something, they are not going to let that happen," said Robinson. "They have taken their responsibility for change control very seriously."

Because of this attention to detail, there has only been one successful external attack on the NMCI. "We have only had one attack in seven years, and we would like to keep it that way," said Robinson, alluding to a highly publicized hacking incident in late 2005.

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Byte and Switch's editors directly, send us a message.