Storage

03:27 PM
George Crump
George Crump
Commentary
50%
50%

Measuring The State Of Primary Storage Deduplication

Before you can assess a vendor's storage deduplication ability, it's important to understand the challenges of the deduplication process.

I've often been asked to rank various vendors' primary storage deduplication capabilities. This is a dangerous prospect as it is clearly subjective. But I can provide some ideas on how to measure each vendor's primary storage abilities so you can weigh those ideas in order of importance to your data center.

First, however, we need to discuss the risks of deduplication.

All deduplicated data has some risk -- after all, you can't get storage for nothing. Deduplication works by segmenting incoming data and creating a unique ID for each segment. These IDs are then compared to other IDs. If there is a redundancy, the redundant data is not stored -- instead, a link is established to the original segment, thereby saving you capacity.

All the IDs are stored in a meta-data table. This table is essentially a roadmap showing what segments belong to which data so that data can be reassembled when requested. If this table is somehow corrupted, you have more than likely lost the map to your data. Even though the map is still there, you can't access it, at least not easily.

[ Keep your VM data local and protect it too. Here's how. Are SAN-Less Storage Architectures Faster? ]

The size of the meta-data table is a concern for deduplication systems. Each new segment represents an entry into that table, and each redundant segment represents a branch to the tree. The size of the table can cause problems, especially when you consider the speed of accessing and updating it.

Think of the meta-data table as a relatively simple database that needs to be able to be updated and searched quickly. This is especially important in primary storage because you don't want write performance to be impacted while the table is searched for redundancy. To avoid this problem, most vendors place their table in RAM. However, in the case of a large primary storage system that houses dozens -- or even hundreds -- of TBs of information, the entire table can't fit in memory. To get around this problem, the table is split between RAM and disk. The problem there is that a deduplication process is not cache-friendly, where a first-in first-out method of using RAM would generate viable hit rates. To overcome this, some vendors deploy their tables in flash; others process deduplication as part of an off-hour process rather than performing the function in real time.

Understanding deduplication meta-data is an important first step in assigning a grade to a vendor's deduplication efforts. Most of the problems described here can be overcome, as discussed in this recent webinar.

In my next column, I'll discuss what to look for in a vendor to ensure that your primary storage deduplication technology is safe, fast and scalable.

The next steps in going virtual up and down the stack, from network to desktop: Automation and finally taking hypervisor security seriously. The two go together, because if you're going to trust production systems to run without human intervention -- a must for delivering IT services on demand -- you'd better be darn sure attackers can't gain control. Get our 2013 Virtualization Management Survey report now. (Free registration required.)

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
PJS880
50%
50%
PJS880,
User Rank: Apprentice
12/3/2012 | 2:35:40 AM
re: Measuring The State Of Primary Storage Deduplication
This is obviously a process where one must weigh the benefits vs the risk to determine if it is worth it to go with data deduplication. The amount of space saved by going this route when you have tremendous amounts of data to be stored, the savings has to add up fairly quickly. Is it possible to determine the size of the table prior to implementation or is the table size unknown or unable to be calculated?

Paul Sprague
InformationWeek Contributor
Hot Topics
4
Stand-Alone SSD Vendors: A Vanishing Breed
Howard Marks, Network Computing Blogger,  7/10/2014
3
Data Backup: Beyond Band-Aids
Tony Kontzer 7/11/2014
White Papers
Register for Network Computing Newsletters
Cartoon
Current Issue
Video
Slideshows
Twitter Feed