Petaflop Supercomputer's Storage Exposed

The Roadrunner supercomputer announced today by the U.S. Department of Energy and IBM supports massive speed and legendary proportions, but its consumption of shared storage is relatively puny.

The supercomputer, housed at Los Alamos National Laboratory, doubles the speed of the previously largest system included in the soon-to-be-updated world's Top 500. That one, IBM's BlueGene/L system in Lawrence Livermore National Lab, has a theoretical peak performance limit of 596 teraflops. Roadrunner's is 1 petaflop (over 1,000 teraflops, or one thousand trillion calculations per second).

Roadrunner has 80 Tbytes of internal memory. It couples 12,960 hybrid "Cell engines" originally made for commercial video games with 6,948 AMD Opteron chips on blade servers housed on 288 IBM BladeCenter racks. Over 10,000 InfiniBand and Gigabit Ethernet connections (more on those momentarily) requiring 57 miles of fiber optic cable interconnect the racks and link out to external storage.

Considering the scale of all this -- not to mention Roadrunner's weight of 500,000 pounds and cost of over $100 million -- IBM confirms a fairly modest amount of external storage, at least by enterprise standards: The system's builders plan to deploy 1.5 Pbytes of Panasas storage, via 200 of the vendor's ActiveStor 3000 shelves, linked via 10-Gbit/s Ethernet switches from Force10. The 10-Gbit/s switches will hook up with InfiniBand switches from Voltaire (more on those momentarily). Roadrunner deploys 10-Gbit/s Ethernet adapters from MyriNet.

That hardly compares with the 14 Pbytes claimed by JPMorganChase ITers, along with other humongous SANs.In a prepared statement, Gary Grider, deputy division leader of Los Alamos's HPC division, says Panasas is the current platform for the agency's file system: "We will be connecting [Roadrunner] to our site-wide file system solution to enable global parallel access to our existing Panasas storage. Additionally, we will be adding significant Panasas storage resources to the global file system service to handle the additional parallel I/O load Roadrunner will generate."

The file data on the Panasas shared storage will consist of scientific data, mostly for nuclear weapons applications.

One vendor who's lobbying for credit, however, is Voltaire, whose Grid Director 2012 switches, based on Mellanox silicon, are being used to interconnect compute and I/O nodes in Roadrunner. Los Alamos has purchased 26 of the big switches, each supporting 20-Gbit/s DDR InfiniBand.

Mellanox, Voltaire's chip supplier, announced today that it's shipping 40-Gbit/s switch chips. Will Roadrunner get an upgrade?

"We're looking at the technology and may do something in that area by the end of the year," says Patrick Guay, EVP of global sales and GM, U.S. operations, for Voltaire. "We will wait for the maturity of silicon and customer demand."Stay tuned for more information on Roadrunner's storage profile as it becomes available.Have a comment on this story? Please click "Discuss" below. If you'd like to contact Byte and Switch's editors directly, send us a message.