Performance Vs. Capacity: A Fork In The Road

During my recent panel session with Howard Marks and Vanessa Alvarez at Interop, a question came up regarding the "big data" concept: How is it that storage capacity is so important to so many companies when performance has never been so critical? How can these two diverging requirements coexist in the same market, solutions, and devices?

The storage industry has long focused on exactly these two requirements: Devices must meet the performance demands of applications while accommodating an ever-increasing demand for capacity. Most advancements in the industry have focused on one of these areas, often enabled by a companion advancement in the other.

It is interesting to consider that RAID was originally developed with performance, rather than capacity, in mind. Combining multiple disks in a RAID set delivered theinput/output operations per second (IOPS) of a mainframe direct access storage device (DASD) at a fraction of the cost. This is the reason for the initial acronym, a redundant array of inexpensive disks.

But something happened along the way: Enterprise storage makers began building centralized storage devices based on RAID, and realized that the same technology could deliver massive capacity, as well. Soon, the race was joined to develop gigabyte-, terabyte- and petabyte-scale storage arrays.

It is challenging to deliver performance as well as capacity. Massive storage arrays have often suffered in terms of responsiveness, so much so that vendors rarelyfill a device with disks. Typical storage array configurations are usually an order of magnitude smaller than the theoretical maximum, and capacity is oftensacrificed in the name of performance.So-called short-stroking of disks has given way to solid state drives and caches, and automatic tiering software helps move "hot" data to better-performing locations. But all this is done in the name of making up for the lack of performance inherent in the disk system to begin with.

Massive drives are a serious problem, as well, since I/O interface performance is accelerating more gradually than disk capacity. It can take hours or even daysto "flush" the data from a modern hard disk drive. Vendors are responding again with clever data placement and caching algorithms that mask but do not eliminatethe problem.

A novel solution to the "performance or capacity" question has recently appeared. Applications are increasingly focused on one area or the other, rather than demanding both. Today’s enterprise applications often pair a performance-critical database for processing and a capacity-filling repository of data. The demands of the applications map to different storage devices, each focused more on one area than the other.

This is the answer to the puzzling query at Interop: "Big data" applications exist, but so do "big performance" applications. Some environments have more of oneor the other, but both are critical for the IT industry as a whole. We must be sure that, when we evaluate new solutions, we keep the intended use in mind. And anapplication that demands both performance and capacity may need reworking, rather than searching for a storage system that can deliver both.