Primary Storage Data Reduction Heats Up

Things have been heating up in the primary storage data reduction market as a management change at Storwize has made them a lot more aggressive, leading to a whole new round of tweets, white papers and the like arguing the merits of fast LZW type compression vs. the hot and sexy technology of data deduplication.  While they've been fighting with Ocarina, EMC and NetApp for the enterprise Nexenta, GreenBytes and HiFn have brought primary storage deduplication to the masses.

GreenBytes and Nexenta both base their systems on Sun/Oracle's Open Solaris and ZFS. GreenBytes delivers ready-to-use storage appliances complete with a management GUI and GreenBytes' own inline data deduplication. They recently announced a new entry-level system with eight 500GB small form-factor drives and 32GB of flash that speeds up ZFS and hash table lookups for the deduplication all for $10,000.

Last year Sun and Greenbytes had their attorneys bringing suits over GreenByte calling their file system ZFSplus, with features curiously appearing in one or the other projects and similar annoyances.  Greenbyte's CEO Bob Petrocelli recently blogged that "I am happy to report that the matter is now behind us and has been settled in a mutually beneficial fashion."

Nexenta is sticking to software putting a BASH/Linux CLI and of course management GUI on OpenSolaris to create NexentaStor which they then sell direct to users that want to roll their own unified storage systems and to OEMs like OnStor and PogoLinux who'll sell fully packaged NAS appliances.  A SuperMicro server configured with 4TB of storage and a 4TB NexentaStor license is in the same $10,000 neighborhood as the GreenBytes GB-1000.

I'm a big fan of how ZFS handles RAID like data protection, snapshots and replication, and especially how it can use flash for log and read caches. I am somewhat concerned about how Oracle will continue to offer, support and extend these open source technologies. I'm therefore glad to say that SpectraLogic is working on a port of ZFS to FreeBSD. If Oracle decides to kill OpenSolaris, the open source community can continue to enhance ZFS without the rest of the OpenSolaris baggage. Some have argued for a Linux ZFS port, but ZFS breaks some of Linux's rules and would have to be run with FUSE  which is a performance killer. BSD and Solaris are more alike under the covers than Solaris and Linux.HiFn's has just started shipping samples of their BitWackr, which combines a PCIe card with HiFn's compression, encryption and hash-generation chip, which has software for Windows Server or Linux to create a data volume that's deduped, compressed and optionally encrypted in real time. Since BitWackr is designed to be installed in systems that aren't dedicated to providing storage services performing hash generation and compression in hardware leaves more CPU available for other things.

It's important that a system using hashes for Deduplication has the ability to do hash-table lookups quickly, and BitWackr can use an SSD to store the hash tables. BitWackr installs as a disk/RAID controller so data is still stored in the OS's native file system, so it can support applications like Exchange that insist on running with NTFS.

HiFn's parent company Exar, who recently snapped up 10Gbps Ethernet NIC maker Neterion, is peddling BitWackr as an OEM product for around a kilobuck, so while you might not see it on the store shelves any time soon, it just might be included as part of some of your favorite products. I've received an evaluation kit, complete with SSD for hash tables, from HiFn and will be setting it up in the lab after Interop. Watch this space for an update. Disclosure: At the time of publication, I am working for EMC and NetApp, but I'm not working for any other mentioned vendors.