Storage Gets Scattered

Everybody wants more storage, from consumers to IT managers -- that's what utility computing is all about. And the trend toward storage-on-demand is throwing light on a hitherto esoteric technique called dispersed storage.

Dispersed storage describes a method of disassembling data into slices, which are then compressed, encrypted, and stored across multiple computers, disk arrays, or workstations. Only specialized client software can retrieve the data, and it can do so using fragments. If one system in the dispersal network goes down, the data can still be regenerated from the remaining parts.

While dispersed storage has been used by government labs for nearly twenty years, it's gotten wider market attention in the last few months, thanks to a company called Cleversafe Inc., whose founder, Chris Gladwin, is hoping to create a new "storage Internet" based on open-source software for use by service providers.

Gladwin, an MIT graduate, inventor, and entrepreneur with expertise in wireless thin clients, broadband audio, and other in-the-now technologies (click to see his photo below), was looking for a way to store his own media files when he decided that dispersed storage was the best way to do so reliably and securely.

Figure 1: S. Christopher Gladwin, president, CEO, and co-founder, Cleversafe Inc.

According to Gladwin, the safest way to use dispersed storage is to divvy up slices of data across disparate geographic locations and even organizations. That way, there aren't enough bits in any single location to crack the data, and no one knows enough to try and put it together.

"With dispersed storage, there isn't any sequential data. You store and recover without a copy anywhere," he says. "If a server is cracked, there's no need to make a disclosure announcement."

Cleversafe is still in the development stage, but it's working on two levels toward commercializing its technology. First, the company offers software on its site for anyone looking to create their own dispersed storage grid. So far, Gladwin says Cleversafe's seen at least 6,000 downloads.

In a second project, Cleversafe is building its own storage grid, using techniques that combine Gladwin's information dispersal algorithms with a grid architecture developed by another Cleversafe co-founder, Matt England, who is now the VP of engineering.

This so-called Cleversafe Dispersed Storage Project is an 11-node storage grid across worldwide locations that uses the Internet for connectivity. It's meant to be a proof of concept, particularly for would-be service providers. "Our focus is on getting the technology started," Gladwin says.Eventually, Cleversafe would like hosting providers to take up this idea and develop their own grids, which in turn would interact with grids from other providers to create a kind of storage Internet. Cleversafe would make money through services to help providers get going.

One thing: It's clear that Cleversafe's technology isn't for transactional storage applications. "Dispersed storage is best for any kind of secondary storage, such as backup and archiving," Gladwin says.

While no service providers have expressed interest in Cleversafe publicly, there's no doubt that several companies are interested in the concept of selling storage-on-demand to consumers, SMBs, and enterprises. Among these is Amazon, whose S3 service has enabled some firms to lower costs and increase efficiency. (See SmugMug.)

But sources question whether dispersed storage can really get off the ground in a big way. "I can see the appeal of this kind of storage service," says analyst Simon Robinson of the 451 Group. "But we all liked the SSP [storage service provider] model before and we saw what happened... The business model is the biggest challenge."

There's plenty of evidence that big companies are interested in the larger concept of grid-based storage, though that doesn't mean they're after dispersed storage. IBM, HP, and Microsoft have all been involved with big storage research. Microsoft's Farsite project is ongoing. And Google's interest in big, distributed storage technologies is well known, if not well understood, thanks to the company's notorious caginess. (See Google's Space Oddity.)Is it likely any of the big storage vendors, including these and EMC, would resist dispersed storage as a potential threat to installed hardware? No, says Gladwin. "EMC already recognizes their future in, to oversimplify it, software-oriented solutions. Half their revenue comes not from hardware. And you can put all kinds of interesting solutions on top of dispersed storage."

Cleversafe's isn't the first effort to widen the scope of dispersed storage. The University of California at Berkeley's OceanStore project, which also includes a storage grid, has been going on for years. But the Website hasn't been updated for over a year, and the director, John D. Kubiatowicz, did not respond to inquiries at press time about its chances of going commercial.

Separately, Oak Ridge National Laboratory and North Carolina State University jointly developed the FreeLoader project, which deploys data dispersal on a smaller scale than Cleversafe.

FreeLoader splits data into chunks or "morsels" that can then be striped across desktop machines. "If I stripe data across ten workstations I can not only aggregate storage, but also network bandwidth and I/O," says Sudharshan Vazhkudai, an ORNL researcher

He's not sure what the commercial implications of the technique might be, but he sees the potential for use in SMBs and departments where there are huge files that may be written to once but read many times -- a scenario familiar to the scientific community.Mary Jander, Site Editor, Byte and Switch