The Truth About Storage Reliability

A pair of studies on storage technology reveal some disturbing facts about disk drive performance and reliability.

Howard Marks

March 30, 2007

3 Min Read
Network Computing logo

Two papers presented at the Usenix File and Storage Technology conference in February challenge many of the assumptions we've long used as the basis of our storage-system designs, most significantly the 1 million hour or higher MTBF found on the spec sheet of almost all disk drives sold for server use.

In both "Disk Failures in the Real World: What Does an MTTF of 1 Million Hours Mean to You," by Bianca Schroeder and Garth A. Gibson of Carnegie Mellon University, and "Failure Trends in a Large Disk Drive Population," by Eduardo Pinherio, Wolf-Deitrich Weber and Luis André Barroso of Google, the actual failure rate of drives was typically more than four times the 0.88 percent AFR (Annual Failure Rate) that a million-hour MTBF represents.

Each group studied the replacement history of more than 100,000 disk drives over data center lifetimes. CMU's samples included both enterprise (SCSI and Fibre Channel) and high-capacity drives with SATA interfaces. Google used desktop style ATA and SATA drives with spec-sheet MTBFs of 400,000 hours in their custom servers. Both studies used the same definition of a drive failure that you or I would use: If a drive had to be replaced by the data center maintenance team for any reason, it was declared a failed drive.

As a charter member of the "you can tell a vendor is lying if his lips are moving" club, I wasn't all that surprised that drives fail more than once every million hours. I was a bit surprised, though, by some of the studies' other findings. In the CMU study, SATA drives failed at about the same rate as the enterprise SCSI and Fibre Channel (FC) drives, contrary to the conventional wisdom that enterprise drives are 50 percent to 100 percent more reliable than their SATA counterparts.

Even more surprising was that drive-failure rates increased as drives aged, even within the five years most of us consider the reasonable service life of a disk drive, and there was no observed peak in drive failures at the beginning of the drives' lives due to infant mortality. In fact drive failures in years 4 and 5 were up to 10 times the rate predicted by the vendor spec sheets.And get this, the studies found that drive failures in individual systems come in clusters. The CMU group found that drives were significantly more likely to fail within hours when another drive in the same server cluster failed. That's bad news for RAID 5 fans, since drive capacity has grown much faster than data-transfer rates, so even in systems with hot spares, it could be several hours, or longer, before the array is rebuilt. A second drive failure, or unrecoverable read error, during that time would mean lost data.

Although SATA reliability appears to rival that of SCSI and FC, enterprise-class drives--due to their higher rotational speed and faster positioners--provide better performance in random I/O applications. SAS (Serial-Attached SCSI) and FC drives are dual-ported, giving redundant RAID controllers independent paths to the drive so a controller or data-path failure won't result in data loss.

So what can we storage managers take from all this academic research? First, choose your drives based on capacity and performance rather than vendor claims. If your application needs lots of space and will primarily do sequential I/O, go with lower cost SATA drives without feeling guilty about giving up reliability--because you haven't. Second, since the odds of multiple drives failing in an array before the array can be re-built are higher than we imagined, you should investigate RAID-6, data replication or other arrangements that can survive multiple drive failures. Now that vendors from Network Appliance to AMCC are speeding up the parity calculations on their RAID 6 controllers, I'm recommending RAID 6 for all the arrays we build with 500-GB and bigger drives.

Howard Marks is founder and chief scientist at Networks Are Our Lives, a network design and consulting firm in Hoboken, N.J. Write to him at [email protected].

About the Author(s)

Howard Marks

Network Computing Blogger

Howard Marks</strong>&nbsp;is founder and chief scientist at Deepstorage LLC, a storage consultancy and independent test lab based in Santa Fe, N.M. and concentrating on storage and data center networking. In more than 25 years of consulting, Marks has designed and implemented storage systems, networks, management systems and Internet strategies at organizations including American Express, J.P. Morgan, Borden Foods, U.S. Tobacco, BBDO Worldwide, Foxwoods Resort Casino and the State University of New York at Purchase. The testing at DeepStorage Labs is informed by that real world experience.</p><p>He has been a frequent contributor to <em>Network Computing</em>&nbsp;and&nbsp;<em>InformationWeek</em>&nbsp;since 1999 and a speaker at industry conferences including Comnet, PC Expo, Interop and Microsoft's TechEd since 1990. He is the author of&nbsp;<em>Networking Windows</em>&nbsp;and co-author of&nbsp;<em>Windows NT Unleashed</em>&nbsp;(Sams).</p><p>He is co-host, with Ray Lucchesi of the monthly Greybeards on Storage podcast where the voices of experience discuss the latest issues in the storage world with industry leaders.&nbsp; You can find the podcast at: http://www.deepstorage.net/NEW/GBoS

SUBSCRIBE TO OUR NEWSLETTER
Stay informed! Sign up to get expert advice and insight delivered direct to your inbox

You May Also Like


More Insights