Network Computing is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

File Systems That Fly

Building powerful supercomputers from off-the-shelf PCs, disk drives, and Ethernet cables running on the open-source Linux operating system has become more than a way to get high-performance computing on the cheap. Those clusters have upended the market for large systems over the last decade. But the ability to shuttle data between the computers and disks hasn't kept pace with advances in microprocessor and memory speeds, adding time and costs to important projects. Now an emerging class of file-system software for clusters stands to change the way companies buy storage.

Cluster file systems, including the open-source Lustre technology developed by the Department of Energy and commercially backed by Hewlett-Packard, speed input-output operations. The technology already is making a difference at universities, national labs, and supercomputing research centers, and it could make inroads into general business computing in coming years.

Cluster file systems are

Cluster file systems are "incredibly fast," says Oak Ridge National Labs CTO Studham.

Photo by Brad Jones

"In terms of raw performance, it's incredibly fast," says Scott Studham, chief technology officer for the National Center For Computational Sciences at Oak Ridge National Laboratory and president of a Lustre user group. With Lustre, I/O speeds range from hundreds of megabytes of data per second to or from disk to 2 Gbytes per second per computer. And since results increase nearly in lockstep with the number of workstations attached, aggregate speeds in a cluster can reach dozens of gigabytes per second while reading from disk.

"Enterprise-class file systems won't do this," says Greg Brandeau, VP of technology at Pixar Animation Studios, which runs a cluster file system from startup Ibrix Inc. The system serves up 240 billion data requests a day from Pixar's 2,400-CPU rendering farm for the computer-animated film Cars, due next year. Pixar is for the first time using "ray tracing" techniques that lend its characters reflective chrome and more realistic shadows, but which place massive demands on CPUs and networks. "We've realized over the past six months that we're not doing enterprise-class computing anymore--we're a high-performance computing shop," Brandeau says.

This week, HP plans to release a second version of its Scalable File Share, a server and software package launched in December that uses Lustre to distribute storage serving in a cluster, much as IT shops have been doing with computing servers for the better part of a decade. Scalable File Share lets Linux machines in a cluster read data at up to 35 Gbytes per second and allows for up to 512 terabytes of total storage, double its previous capacity. "One of the keys is you now build the storage system using cluster technology," says Kent Koeninger, HP's high-performance computing products marketing manager.

  • 1