Seagate Boosts Disk Drive Intelligence

Face it: disk drives have been boring for a long time. Once we crossed the 1TB barrier, even the periodic announcements of bigger capacities stopped being especially interesting. The excitement is all about SSDs, even as disk drives hold the exabytes of content that are flooding both cloud and corporate data centers. However, Seagate is making an interesting move in the disk drive space with its Kinetic Open Storage platform, a new storage architecture that shifts much of the intelligence in a modern storage system to new disk drives with Ethernet interfaces.

My friend Stephen Foskett has been saying for the past few years that if we were to invent storage from the ground up today, we wouldn’t be relying on ancient abstractions like blocks and LUNs. While today’s file and object storage systems do create more useful layers of abstraction, those storage systems eventually have to map their files and objects to blocks on conventional disk or tape drives.

The last time disk drives got smarter was in the 1980s as SCSI and ATA drives, which integrated the disk controller with logical block addressing into each drive, replaced earlier interfaces like ST-506, SMD and ESDI. With logical block addressing, operating systems no longer had to keep track of data locations by head, track and sector, and drive vendors could implement tricks like zoned recording, which puts more blocks on the longer outer tracks of the disk. However, the long and slightly bumpy transition from 512 byte to 4KB blocks showed that block interfaces are still rather primitive and inflexible.

Seagate’s Kinetic Open Storage platform replaces that primitive block IO interface with a key/value API and the traditional SAS or SATA interfaces with a pair of gigabit Ethernet ports. This allows the new drives to offload tasks such as keeping the object to block storage mappings and a list of free storage blocks on each device. Now, an object storage system like OpenStack’s Swift or distributed file system like GlusterFS can put and get data from the drive more like applications get and put data from object stores.

The key in the pair can be up to 4KB in length and divided into fields for access control tokens, hash values and the like. Seagate recommends that values be up to 1MB in size. The company has designed the whole thing so an object store can write shards of an object across a sea of drives as the values using the URI of the object plus a shard identifier as the value. Other obvious use cases would include storage for NoSQL databases like Cassandra.

[While some vendors won't fare well if software-defined storage takes off, disk drive companies will reap benefits. Read how in "Why Software-Defined Storage Is Good For Seagate."]

While the initial drives will be capacity oriented devices spinning at 7200 RPM or less, Seagate’s hinted at flash and/or hybrid versions. I could see some vendors building large deduplicating data stores by using the hash of 4KB data chunk values. Each drive in a very large array could then be responsible for holding values across a range of keys, creating a content addressable backend similar to what Exablox or SolidFire does.

The drives are also smart enough to play an active role in functions like data protection. Today, an object storage system has to write a copy of each object individually to several disk drives around the data center. The Kinetic drives can perform third-party copies, so an application could write a key value pair to drive A and tell it to copy the object to drives C and D. They’re also smart enough to perform all the housekeeping tasks associated with maintaining the key value store. In fact, as part of the housekeeping, they end up storing the data on their platters sequentially by key value, which maximizes performance reading multiple shards of an object.

Seagate’s lined up an interesting group of “fellow travelers,” including Dell and Xyratex to build Ethernet JBODs for the new drives SwiftStack and Basho for object stores, and customers Yahoo and Rackspace for the voice of reality.

It was time for Seagate to make a bold move and Kinetic Open Storage, while a clumsy name, is a bold move. Only time will tell if the market accepts smart disk drives, but it’s certainly time to give the idea a try.