Tech Alone Not Enough To Manage Big Data

Big Data Talent War: 10 Analytics Job Trends

Most people who have been in IT for a while have a favorite story about the staggering growth of data storage demands over the years, and the equally staggering advances in the capacities offered by storage technologies. My own is a marquee sale I made in the mid 1980's.

It was a huge deal both financially and in terms of operational significance; the data center floor was swept of one vendor's equipment and replaced by the new, cool stuff that I was representing. Delivery necessitated the closure of a London street on a Saturday morning in order to have a crane lift the equipment to its appointed floor. And the punch line? The total capacity was 10 GB. It is amazing when you think that today I can get a mobile electronics flash card that's about the size of a fingernail and holds 64 GB!

My point is not just about the world moving on in absolute capacity; there's a relative story here, too. After all, at the time, that 10 GB represented "big data". Those of you who were around then will recall a serious debate within the industry about how many gigabytes a single-capacity planner could realistically manage. Maybe two, perhaps four? But, to repeat, the world has moved on. Or has it?

Today, there's lots of talk about big data. It is typically associated with, or followed by, a word such as "analytics" and its precise meaning is all rather mysterious--but it definitely refers to processing and storage needs that exceed the abilities of standard platforms. Partly it exists because--at least according to conventional wisdom--many of the issues associated with "regular" large-scale storage management apparently have been solved by modern storage software: thin provisioning, automation, what-have-you.

[ Read Big Data Projects: 6 Ways To Start Smart. ]

Of course, not everyone has the skills, budget, or scale to afford or deploy such functionality. But one could surely be reasonably safe in assuming that "regular big data"--just the amounts, without the analytics--is no longer much of an issue in the largest IT enterprises.

Our research shows that, surprisingly, this is not the case. Instead, it looks as if data growth still is outpacing the ability of advanced storage to manage and tame it. This is because organizations with higher amounts of storage capacity not only still rank "managing data growth" (in a tie with "improving backup and recovery") as their top overall IT priority, but they also are significantly more likely than smaller operations to be hiring storage administrators. For comparison, in smaller IT operations--those 50 TB and under--"managing data growth" is only sixth on the IT priorities list. Does this mean that all those advanced storage features that have been designed to optimize the management and protection of data are not living up to their billing? Or does it perhaps mean that these larger organizations simply aren't using them effectively, or at all?

Every year, ESG conducts an IT Spending Intentions survey. As one might expect, the top IT priorities for 2012 include--as they have for three straight years--such things as the increased use of server virtualization, improved backup and recovery, major application deployments or upgrades, managing data growth, and information security. However, for 2012, "managing data growth" rose to the top overall spot among organizations that have at least 500 TB of storage capacity under management. There is also a greater emphasis on improving backup and recovery processes among these same organizations. These two factors suggest that, despite a plethora of advanced storage and data protection features and tools, organizations with significantly higher storage capacities are still struggling to keep everything in check.For perspective, data growth and data protection were not merely singled out as top storage priorities, but rather were identified as the top overall IT priorities among those organizations that have more storage capacity to manage. The fact that these larger organizations rank managing data growth ahead of trendier initiatives--such as server virtualization and private cloud technology--makes the insight all the more noteworthy and intriguing. After all, aren't these the places that should have their fingers on the pulse of the problem?

Of course, you might say, data growth will invariably result in a need for more overall storage capacity, so--while it's interesting that it's important to people--it's no big deal. However, another consideration when managing such data growth, and the associated, supposedly palliative, modern storage functionalities, is the degree to which increased skilled human interaction is required. Certainly the ability of administrators--supported, of course, by advanced, and often automated, functions within storage systems--to manage far more data than was ever considered possible decades ago, has helped to mitigate the personnel needs. Yet the link between overall storage capacity managed and the number of people required for the task still exists. In other words, it remains a matter of scale more than simply a matter of skill and functionality.

Specifically, the fact that organizations with at least 500 TB of total storage capacity are twice as likely as those with less than 50 TB (42% vs. 21%) to be looking to hire additional storage administrators in 2012 clearly reinforces the idea that data growth is still outpacing the abilities of the technology deployed to manage it. Yes, the amount of storage an individual can manage these days far exceeds what once was the case, but the fact remains that it is still hard to avoid increasing headcount as a method for managing data growth.

What are we to make of this? Either advanced automated storage features are not delivering as much as we'd like, or perhaps companies are not using these technologies effectively, or even at all. The takeaway: both the end-user community and storage vendors must continue to focus more than ever on storage management and efficiency. The beast of data growth has not yet been completely tamed by all the recent technological advances for storage, so vendors must continue to push the development of features that allow storage infrastructure to automatically keep pace with managing growing data volumes, especially in today's increasingly dynamic IT environments.

This is counter to the common belief that storage is storage, and that it is in many respects--at least from a "command and control" perspective--a done deal whereby you add performance or capacity as needed and the rest happens auto-magically.

Likewise, users must take the time to learn about the various capabilities of today's systems and commit to leveraging them whenever possible. Vendors can help through education and training.

Otherwise, as these research findings show, until the direct linkage between data volumes and cost can be broken, storage has an unsustainable IT future.