Comments
RAID Vs. Erasure Coding
Newest First  |  Oldest First  |  Threaded View
timwessels
50%
50%
timwessels,
User Rank: Apprentice
7/16/2014 | 9:26:39 AM
Re: Good short explanation of RAID vs. Erasure Coding
Object storage is considered secondary storage and therefore performance has not been a major criteria.  One object storage software, Scality, claims that tests run on their RING storage clusters equal the performance of primary data storage systems.  The object storage software vendors who offer erasure coding in addition to replication include Caringo, Cloudian and Scality.  Ceph's commercial sponsor, InkTank, was recently purchased by Red Hat.  Sage Weil, who developed Ceph as part of his PhD work, is a genius kind of guy, but Ceph has not seen wide spread deployment in commercial environments yet.  Ditto for Swift.  Other object storage vendors, like Amplidata and Cleversafe, base their object storage solely on the use of erasure codes.  While some erasure codes are proprietary, many are based on or derived from Reed-Solomon, which has been around since the days of X.25 packet switching networks.  I recall back in the day that a lot of 1/4-inch cartridge tape drives and 4mm DAT drives used Reed-Solomon ECC to reliably write data to tape. 
Brian.Dean
50%
50%
Brian.Dean,
User Rank: Ninja
7/16/2014 | 6:17:19 AM
Re: Good short explanation of RAID vs. Erasure Coding
Erasure coding is an excellent frontier, in large setups it can create a load on CPU power, but I feel that this is minimal considering the rate of CPU advancement.

As the price war for Cloud Storage seems to be slowing down, this would be a good opportunity to offer Cloud data protection services to datacenters. However, considering that some enterprises only have an in-house datacenter, because of the efforts from intelligences services and other security needs -- encryption requirements for data in transit would create a higher CPU load.
joreilly925
50%
50%
joreilly925,
User Rank: Ninja
7/15/2014 | 2:36:59 PM
Re: Good short explanation of RAID vs. Erasure Coding
Aditshah, I think the hardest part was finding a mathematical structure that reflects the erasure code calculation! It's amenable to some hardware assist logic. Roll on a few SoCs?
joreilly925
50%
50%
joreilly925,
User Rank: Ninja
7/15/2014 | 2:34:57 PM
Re: Good short explanation of RAID vs. Erasure Coding
Tim,

I agree with your assessment, but there are plenty of people still in denial. I cringe, for instance, every time am SSD is compared price-wise with a $59 bulk SATA terabyte drive. It's the wrong comparison. MLC SSD is now cheaper than "enterprise" HDD, and there is no excuse for not moving to solid-state primary storage.

The migration to object storage is a bit more complex, since the vendors didn't pay enough attention until recently to performance and features. Opensource code like Ceph does now support erasure coding. We need the Chinese ODMs to hit their stride in the US channel market, and object storage will take over the secondary, bulk-storage, tier pretty quickly.
aditshar1
50%
50%
aditshar1,
User Rank: Ninja
7/15/2014 | 1:43:56 PM
Re: Good short explanation of RAID vs. Erasure Coding
When i think about use cases for erasure coding first thing comes to my mind is object-based cloud storage although understading its mathematical calculation has always been puzzle for me
timwessels
50%
50%
timwessels,
User Rank: Apprentice
7/15/2014 | 11:37:51 AM
Good short explanation of RAID vs. Erasure Coding
Well, I agree that hardware RAID as we have known it for over 20 years is dead, especially RAID levels that use parity to protect data.  Mirroring is still viable as there is no parity calculation needed.  In the near future there are only going to two types of storage...flash and object.  All primary or "hot" data will be stored in flash and everything else, which is 80% unstructured data, will be in object storage with the "warmer" data stored using replication for durability and and faster reads while the "colder" data will be stored using erasure codes for durability as it will seldom, if ever, be read.


Cartoon
Slideshows
Audio Interviews
Archived Audio Interviews
Jeremy Schulman, founder of Schprockits, a network automation startup operating in stealth mode, joins us to explore whether networking professionals all need to learn programming in order to remain employed.
White Papers
Register for Network Computing Newsletters
Current Issue
Research: 2014 State of the Data Center
Research: 2014 State of the Data Center
Our latest survey shows growing demand, fixed budgets, and good reason why resellers and vendors must fight to remain relevant. One thing's for sure: The data center is poised for a wild ride, and no one wants to be left behind.
Video
Twitter Feed