Hard Disk Drives Cling to Life as SSDs Take Off

Despite recent developments in hard-disk drive technology, solid-state drives are on the way to becoming the solution of choice for enterprise storage. They have cost more than HDDs, which held back adoption considerably for the last few years, but the situation changed this year, as I predicted.

The secondary storage market is characterized as using high-capacity 3.5 inch hard drives. The segment is price-sensitive rather than performance oriented and, due to hard-drive capacity growth over the last half-decade, remains the primary market for HDDs.

The HDD capacity growth curve has, however, stalled out. Moving beyond 14TB – the capacity Western Digital announced in October and the largest HDD to date -- will prove difficult technically. The common approach to larger capacity has been Heat-Assisted Magnetic Recording (HAMR), where a laser is used to magnetically “soften” the area to write a bit. Vendors promise to reach 100TB capacity in eight years, for example, but the technology is very difficult to get right, never mind produce in volume.

WD said it's using an alternative approach, MAMR, which uses microwaves to achieve softening. The company promises to ship its first MAMR product in 2019, which may prove a bit optimistic. The problem with either approach is that we already have 32TB SSDs, which fit in a smaller 2.5 inch footprint, effectively half the size of these bulk HDDs.

The evolution of SSD capacity is moving rapidly along with 3D NAND, die stacking, and QLC cell architectures all poised to drop prices rapidly in 2018 and to make even larger capacities available. We can certainly expect 64TB SSDs in 2018 and perhaps even see the 100TB units several vendors have promised.

SSD.jpg

These large SSDs will likely be at a dollars/terabyte premium over HDDs for a while, but the capacity increase and reduced size means many fewer appliances will be needed to store secondary, cold, data. Moreover, non-traditional drive packaging such as Intel's elongated M2 blades, with 32 blades of 32TB each in a 1U appliance, promise 5PB in a 1U appliance when combined with compression.

Already this year, perceptions around SSD costs have been impacted by a realization that the much greater throughput possible with SSD primary storage means that fewer servers are needed to run a given workload, with the resulting savings more than offsetting extra pricing for the drives. Also, the cost of NVMe flash drives has moved close to SAS and SATA pricing, resulting in NVMe now being the interface of choice in servers and even desktops.

At the same time, SSD-based all-flash arrays (AFAs) have displaced the RAID array as the preferred approach in networked storage. Here, because of the high bandwidth available, AFAs support compression of both the primary data they store and the secondary data stream being offloaded to other storage appliances. For most applications, compression results in a 5:1 effective multiplication of the raw capacity. Because HDD primary storage is way too slow, compression is not a viable option for RAID arrays.

A third trend benefiting SSDs is the growth of hyperconverged infrastructure. Based originally on SSDs, HCI has migrated to NVMe SSDs to obtain the response times and throughput those drives bring. Led by Excelero, the next step in HCI is direct connection of NVMe drives to the RDMA Ethernet fabric of the cluster of nodes, removing latency and providing very high throughput. Excelero’s approach opens up directly connecting future NVMe Ethernet drives to the cluster fabric, allowing a great deal of parallelism in the cluster storage design.

The result of all of these trends is that this year the battle for market share is heating up, with SSD pulling ahead strongly in the enterprise drive class. At the same time, SAN-based primary storage is in decline, with RAID array sales falling quarter by quarter.

Can desktops hold the HDD market up for a while? I just bought a new system, with a mid-market motherboard. It has four slots for M2 NVMe SSDs! Gamers will generally go for speed, especially more so when the price of the NVMe drive is identical to the SATA equivalent!

One bump in the road for SSDs is flash die production capacity. The conversion to 3D NAND was more difficult for suppliers than expected, causing shortages in the first half of this year. With that problem moving into the history column and real capacity gains from the recent innovation of die stacking  for 3D NAND, 2018 will see supply moving close to demand. Still, demand will be high for NAND die so shortages may persist during the transition from HDDs.

With few factors in their favor, hard drives are looking to go the way of the Dodo. This isn’t going to be an overnight phenomenon. Radical changes like this take half a decade or more to complete and even at the end there will be a market for legacy systems.