The World's Biggest SANs: Part Two

We at Byte and Switch are still on a quest to unearth the world's largest SAN.

With users across the globe wrestling with phenomenal rates of data growth, it makes sense to see how some of the world's largest storage deployments are meeting demand. Clearly, for the organizations on this list, size matters.

From the the U.S. military to supercomputing sites and Internet service providers, the organizations featured here either have deployed, or are planning to deploy, monster SANs.

As with the roll-call of mega-SANs we published earlier this year, this list is part of a long-term effort to sniff out super-SANs. If you've got a big SAN story to tell, we'd love to hear it. Hit that message board, call us, or send us a message.

So, without further ado, we present five of the world's biggest SANs:

Next Page: US Navy

Storage snapshot: The U.S. military is building a huge multi-Pbyte SAN to support its critical communications infrastructure.

Key suppliers: Dell, EMC, NetApp

The U.S. Navy is currently building a multi-Pbyte monster SAN to support the Navy and Marine Corps Intranet (NMCI). It's potentially the largest storage network in the world.

The Navy's massive Intranet provides a Web-based communications infrastructure for more than 450,000 Navy and Marine personnel, as well as civilians working within the two branches of the U.S. military.NMCI aims to provide Intranet users with 100 Mbytes of storage and 50 Mbytes of Exchange space. Earlier this year, the Navy started to roll out a "single mailbox restore" service, which aims to restore users' mailboxes in 20 minutes in the event of a data loss.

EDS is the primary contractor for the $9 billion NMCI project, which it describes on its Website as "the world's largest, most secure private network." Despite a series of performance and payment disputes, the Department of the Navy extended EDS's contract last year, taking the deal through to 2010.

Dell provided the initial 2-Pbyte storage infrastructure for NMCI back in 2000, although NetApp and EMC have since gotten involved with the project.

"There's a SAN that supports this, and we make pretty good use of NAS," explained Akira Robinson, a consultant on the project, earlier this year, adding that this hardware is spread across three network operation centers (NOCs) in Norfolk, Va.; San Diego, Calif.; and Pearl Harbor, Hawaii, as well as more than a hundred server farms dotted around the world.

Precise details of the NMCI project are hard to come by, although the Intranet is said to account for a sizeable chunk of EDS's managed storage infrastructure, which is more than 12 Pbytes.Despite the ambitious scope of NCMI, the Department of Defense (DoD) continues to come under fire for its handling of the project. In October, for example, the Government Accountability Office (GAO) warned that the Navy needed to keep closer tabs on NMCI performance management.

The U.S. Navy is already using the IT Infrastructure Library (ITIL) to tighten its control of the NMCI project, following in the footsteps of General Motors and the U.S. Pacific Air Force (PACAF) Directorate of Communications and Information, which is using ITIL as part of an effort to restructure its technology service desk.

The ITIL framework, which has become widely accepted as a reference point, is a set of best-practices covering application management, security, and IT service delivery.

Next Page: AOL

Storage snapshot: Virtualization is at the heart of the Internet firm's plans to manage its rapidly growing SAN.Key suppliers: Brocade, EMC, Yotta Yotta

Internet giant AOL is currently building out its 4-Pbyte SAN as part of a long-term project to deal with the firm's massive data growth. "I can conceive of it doubling," says Daniel Pollack, AOL's operations architect, explaining that the SAN could easily reach 8 Pbytes within the next few years.

The SAN, which is built from Brocade Fibre Channel switches and EMC hardware, supports both AOL's internal and external systems. "It's primarily [EMC] DMX 1000s; we have some 950s as well, and Clariion CX3-80s and 700s," says Pollack.

The network provides storage to around 3,000 systems within AOL, although Pollack tells Byte and Switch that the firm is now looking to virtualize large chunks of its data.

"We're about 10 percent virtual block storage," he says, explaining that AOL deployed virtualization devices from Yotta Yotta earlier this year to support its data migration efforts."We were just struggling with the continuous data migration workload," says the exec, adding that, prior to virtualization, he had to speak to as many as 80 internal users when he wanted to move an array. "It was pretty bad."

By deploying Yotta Yotta's GSX 3000 devices in front of his SAN, Pollack says he has revolutionized AOL's ability to shift data around.

"This abstracts the front end from the back end so that we can change the back end as often as we want to."

AOL is now planning to extend the block virtualization solution across its SAN. "By the end of 2008, we should be 90 percent done with virtualization -- we should finish out the rest of it in early '09," Pollack says.

Next Page: National Center for Supercomputing Applications (NCSA)Storage snapshot: This supercomputing site is bracing itself for a data explosion over the next few years, which will more than quadruple the size of its SAN.

Key suppliers: Brocade, EMC, DataDirect Networks, Engenio, Finisar, Nagios

Located at the University of Illinois in Urbana-Champaign, the NCSA designs software for some of the world's largest supercomputers, as well as performing its own research in areas such as DNA sequencing and astrophysics.

The Center currently has around 1.4 Pbytes of disk-based storage in its SAN infrastructure, although this figure is expected to grow to a hulking 10 Pbytes by 2012. Built largely from Brocade Silkworm switches and hardware from DataDirect Networks and Engenio, the SAN is supported by a 4-Pbyte DiskXtender archive system from EMC.

"Under its current license, the archive could grow to 10 Pbytes," says Michelle Butler, the NCSA's technical program manager. "That archive almost doubles every year, so we will get there."The SAN is linked using Fibre Channel, although the NCSA is currently looking to upgrade this infrastructure. "It's two and four gig at the moment," says Butler, "but we're doing a lot of work with ten-gig and trying to figure out where it fits."

With such rapid data growth, the NCSA has to think very seriously about how it monitors its infrastructure.

"There's way too much stuff to keep track of manually," says Andy Loftus, the NCSA's technical manager, explaining that the Center uses a set of automated software tools to keep track of everything.

These include an open-source tool called LogSurfer, Nagios Web interface software, and Finisar's NetWisdom product.

"LogSurfer monitors the logs and sends reports to Nagios, whereas Finisar monitors traffic on the SAN itself, and we use that for health analysis and performance base-lining," explains Loftus.At this stage, it is still relatively early days for the NCSA SAN, which must support five separate supercomputers, including the recently launched 90-Tflop Abe system.

Next Page: Snapfish

Storage snapshot: The digital photo store is laying the foundations for a whopping 22-Pbyte SAN

Key suppliers: HP, Brocade, DataDirect Networks, PolyServe, QLogic, SGI

Online photo service Snapfish foresees phenomenal growth for its SAN over the next few years, as more and more people look to share their digital images over the Internet.The firm, which was bought by HP for a rumored $300 million in April 2005, started building its SAN infrastructure almost three years ago and is currently planning a massive upgrade to it.

"More or less, we're adding storage every quarter," says David Lee, Snapfish's storage architect. "We have grown approximately 50 to 60 percent a year."

The San Francisco-based firm is currently using around 5 Pbytes of its 7.6 Pbyte capacity, although the exec foresees a SAN explosion over the next year or two.

Snapfish's SAN could easily reach 10 Pbytes by fall 2008 and then some, according to Lee: "We have just finished building a new data center and, at full capacity, we expect to have 22 Pbytes."

The firm already has two data centers in San Francisco, but was forced to build a third 25,000-square-foot facility in Texas to support its growth. "We were running out of floor space and power."Snapfish has around 22 HP EVA devices in its SAN, and more than 100 of that vendor's Modular Storage Arrays (MSAs). The photo service, which uses PolyServe for clustering, also has 12 Brocade 48000 Directors deployed across its three data centers, as well as 42 Brocade 4-Gbit/s switches. "We use the directors at the core and the standalone switches at the edge," notes Lee.

Snapfish has also deployed around 300 Qlogic HBAs across its data centers, and it is also using some "midsize" arrays from DataDirect and SGI to supplement the EVAs and MSAs.

Snapfish's growth is all the more remarkable considering that the firm's SAN infrastructure is still in its infancy. "Just before we got acquired by HP, we started to look at SAN," Lee says, explaining that Snapfish was previously a NetApp NAS shop.

"Once you have the [SAN] infrastructure built, it makes scaling disk storage fairly easy. If we tried to scale with NetApp, it was intensely expensive."

There are still around eight R200 NetApp nearline appliances at Snapfish, although Lee tells Byte and Switch his next big project is IP over Fibre Channel."We're talking about using this because our Fibre Channel infrastructure is much larger than our IP infrastructure in terms of available throughput," he says. "We do a lot of data moving -- we move Tbytes in hours [so] we were hoping that trunked four-gig Fibre Channel will get us massive amounts of throughput."

Next Page: The Internet Archive

Storage snapshot: Non-profit organization takes a leaf from Google's book to harness the power of its storage network.

Key suppliers: Capricorn Technologies

This San Francisco-based organization is faced with the Herculean chore of recording all the Web pages ever produced, a mind-boggling task that requires some serious storage backbone."We're also digitizing hundreds of thousands of books," explains Brewster Kahle, the Internet Archive's founder, adding that the organization is looking to store music and video data as well.

"There are a couple of other companies that do things on this scale -- Google and the hotmail application, but there are not many more that I am aware of."

Like Google, the Internet Archive is also harnessing the power of thousands of Linux machines.

Founded in 1996, the Internet Archive is essentially a massive digital library and currently contains around 2 Pbytes of data in its storage network. This network consists of around 2,000 Linux boxes from Capricorn Technologies, linked via Ethernet, each of which contains around 4 Tbytes.

These "Petaboxes," as they are called, enable the Internet Archive to quickly scale up its operation without breaking the bank, according to Kahle: "Its low-cost, easy to maintain, low power, and high density."Archived Web-page data is accessed via the Internet using a portal called "Wayback" built by Kahle and his team. The name recalls a segment of the Rocky & Bullwinkle cartoon show of the 1960s.

For the Archive's future storage needs, Kahle tells Byte and Switch that he is looking at Sun's recently launched Project Blackbox, a mobile data center shipped in a container to users' sites.

"This is the first major step forward I have seen that would be interesting to someone on the scale of the Internet Archive," he says. "You can put three Petabytes in a shipping container. That, we think, is a real contender."

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Byte and Switch's editors directly, send us a message.