The Storage Problem You Can't Ignore

It's not news that storage is swamping IT budgets. Our 2011 InformationWeek Analytics State of Storage Survey shows the amount of actively managed storage expanding at around 20% per year. In our practice, we work with a few companies dealing with growth levels in excess of 50%. At this rate, most data centers double storage capacity requirements every two to three years. And as employees start using multiple mobile devices and consumer applications for work, that estimate could be conservative.

In our first InformationWeek Analytics Public Cloud Storage Survey, fielded in April, 59% of respondents using, planning to adopt, or assessing public cloud storage services called out email as the application most responsible for storage growth, followed by increasing demand from new or planned applications (58%). Seventy-six percent said they're somewhat or very concerned about storage costs, and most CIOs we speak with insist they're actively seeking to reduce those expenditures while still keeping data available. So you can imagine our surprise that, when asked exactly what they're spending per gigabyte, nearly half our survey respondents said they have no clue. They have data retention policies, but enforcement is all over the map. When we asked about strategies that could lower storage costs, we got a virtual yawn: Just 10% plan to use external storage services within the next two years. Only half are taking advantage of storage virtualization. Sixty-one percent either make do with the management tools provided by their storage vendors (53%) or don't actively manage storage resources at all (8%).

Let's be clear: No vendor is going to show up with a multifunction storage uber-solution that will save us from ourselves. Among our enterprise clients, storage volume growth is outstripping the dropping price of physical media--limiting IT innovation as costs go through the roof.

We must do the hard work of figuring out the right mix of policies and technologies to balance access, performance vs. capacity, security, and short- and long-term costs. We must figure out what, exactly, we're paying per gigabyte for internal storage so we can do an intelligent total-cost-of-ownership analysis of not only various cloud service alternatives, but also new technologies like solid-state drives, deduplication, and storage virtualization. Is Fibre Channel your future, or is it time to move to Ethernet? Do you need to revisit your tiering strategy? CIOs must look three to five years out and make sure their purchases today align with their long-term strategies. And clearly, you can't do that if you don't have a strategy.

Research: Cloud Storage

Changing Dynamics Beyond Services
Become an InformationWeek Analytics subscriber and get our full report on cloud storage.

This report includes 36 pages of action-oriented analysis packed with 19 charts. What you'll find:

Get This And All Our Reports

Illustration by Nick Rotondo

Lay Down The Law

The forces of cloud, mobility, and consumerization are hastening the demise of the fat corporate desktop. In our InformationWeek Analytics 2011 End User Device Management Survey, 67% of 551 respondents said their companies allow email to be accessed over employee-owned gear, and in our cloud storage poll, file storage services are highly popular. What that means for storage admins is that business content is now located on personal laptops, smartphones, and tablets, as well as on Dropbox accounts--a real problem for shops that need to exercise a high level of backup, analysis, and compliance control over data. In response, vendors like Asigra and SugarSync have launched business-class off-site backup protection for mobile devices, and Apple's iCloud can help some shops (see Mobile Device Backup--Still A Work In Progress). But you need policies, not just technology, to solve this problem.

Many CIOs are thinking, Do we really need all this stuff? In most cases, the answer is yes. Analyzing massive amounts of data is how mature companies spot customer trends, dive into analytics, and leverage institutional knowledge. Aggressive data deletion isn't the answer; a comprehensive data retention policy is. It's the foundation of storage governance and for defining technology requirements. Without that baseline, you're throwing dollars at storage willy nilly, and the result is a depleted budget, underutilized technology, and zero ability to plan for the future.

It's encouraging that 89% of respondents to our public cloud survey have formal data retention policies. The bad news? Just 22% say employees comply with it extremely well. Drilling deeper, we see that policies vary by the type of application. Companies are most likely to apply definitive retention and deletion policies to enterprise data warehouses and email; least likely to R&D data sets and Web content.

And having a policy doesn't necessarily equate to a defined deletion date. We were surprised at the number of companies with indefinite data retention requirements for enterprise database/data warehouse and Office documents/SharePoint--even though we included up to 10 years as an option. These applications generate a lot of data, and we expect that these policies will be revisited as full costs become apparent.

Writing data retention policies is a report unto itself, but at minimum, consider how data will be used, searched, and obtained. That will drive the selection of storage technology. Tools like EMC Centera and IBM Tivoli Storage Manager can help enforce policies and enable governance. With Centera, for example, you can preserve original content and provide a higher degree of integrity during the life of your archived information by using application-based record retention and disposition.

Before contracting for external storage services, make sure your data governance tools can extend into the provider's network and manage a mixed environment. Most survey respondents, 53%, use vendor-provided utilities to manage their storage systems. In our experience, they're unlikely to have the depth of visibility needed to make informed decisions. Better are tools such as Nexenta Systems' NexentaStor, Quantum's StorNext, and Quest's Foglight Storage, which can provide a unified view of storage allocation and utilization.

The Economics Of Storage

Our InformationWeek Analytics Cloud ROI Survey revealed the top three reasons companies are using or evaluating public cloud services of all kinds: the ability to quickly roll out business technology, the expectation that they'll lower long-term expenses, and the need to reduce the number of activities that require in-house IT expertise. Replacing capital expenses with operational expenses came in No. 4.

With respect to storage, however, these considerations are of less concern. Many companies have the mind-set that, like gravity, storage prices always fall. In our public cloud storage survey, 21% said they're not at all concerned with costs. Half said they're somewhat concerned but see costs as manageable. "Buying more storage is a lot easier than trying to get a handle on how it's used," says the IT director of a global financial institution.

This attitude, while it makes storage vendors happy, isn't sustainable. When nearly half (44%) of respondents don't even know the per-gigabyte cost of their on-premises storage, how can they say expenditures are manageable? Now, figuring out how much any IT service costs is tough. We weren't surprised that few of the 551 respondents to our End User Device Management Survey said their IT organizations charge back or do cost allocations. Of the 122 that do, 83% charge only for the purchase price of the device and OS plus software required for the employee's duties. Network and server assets? Forget it, and this for something as relatively simple as a PC. It's no wonder that breaking down storage is a tall order. Do you include power? Maintenance and support? Data center floor space?

Tom Scroggins, an IT architect with a financial services group, pegs his organization's cost per gigabyte at the high end of our respondents' range but goes beyond just raw disk to include RAID levels and arrive at a per-usable-gigabyte price, which is allocated to business units. "Currently, the main benefit of cloud-based storage is variability," Scroggins says. "We can provide our own geographic diversity and can purchase storage as cheap, or cheaper, than the cloud providers deliver." He says his organization is also constrained by regulatory requirements and believes storage services with sufficient controls for financial and healthcare institutions are a few years away.

If you're moving toward an internal IT service management structure, you, like Scroggins, have a better than average understanding of your environment, but our survey found some knowledge gaps. And our experience shows that a lack of visibility into elements of a service, like cost and utilization, leads IT to take the path of least resistance--in this case, buy more disk.

Why Not Go Virtual?

From a technology perspective, 95% of respondents use hard-disk-based systems for their on-premises storage. More than half use magnetic tape, with optical disks (28%) and solid-state drives (25%) filling in at the high end to create a tiered environment that can be tailored to the access demands on various classes of data. Given widespread server virtualization, we expected to see movement toward on-premises virtualized storage systems. After all, vendors such as DataCore, FalconStor, and StarWind offer flexible, standards-based storage software that will run on any x86 platform and easily add a software virtualization layer.

Nope. Over the next two years, 60% of respondents expect virtualized storage to make up less than 25% of their environments. Of that, nearly 20% said they won't use virtualized storage at all. Unlike server virtualization, few in our survey see the value of storage virtualization and remain content with the status quo of on-premises, Tier 1 SANs.

Why the lack of interest? A key benefit of outsourcing storage is shifting the burden of capacity and performance management, not to mention operations and maintenance, to the provider. Not so with internally virtualized environments, where CIOs may not think the benefits outweigh the cost and effort to deploy and manage. This attitude reflects what we're seeing in our practice: Only extremely large enterprises can justify the manpower that virtualized storage requires.

32 Flavors

The term "cloud storage" is imprecise, so in our survey we drilled down into various aspects, like backup and email archiving, and extending on-premises raw storage. Public off-site services can provide file-system access for enterprise applications, creating a hybrid on- and off-premises storage environment.Vendors such as GoGrid, Peer1, and Rackspace also offer managed on-site services that make it easier for IT shops to virtualize storage within their data centers and provide for scalability and management in a much denser environment. Hybrid approaches combine public and private, allowing for bursting into the provider's data center when on-premises capacity is exceeded. Running applications in a mixed environment can improve performance across a highly distributed geographical area.

Even with all these options, our survey found very low adoption rates. A mere 15% said they're using public storage services. Only 10% said they're planning to adopt them within the next two years. While 38% of respondents are still considering how to proceed, 37% have decided they have no desire to look outside for storage. While marketing of public clouds is hot, potential buyers are cautious. So what's the holdup?

As in other cloud surveys, security, privacy, reliability, performance, potential for data loss, and availability lead the list of inhibitors among those with no plans to use public cloud storage. Regulatory and legal constraints are also of great concern to 39% of respondents. Those are some pretty big barriers to overcome.

The willingness to store data off site also varies greatly by industry--healthcare, financial, and legal firms may never move to off-site storage, except for peripheral services. "There's too much risk relying on the Internet and other networks/data centers to store medical imaging and reporting there," says Eric Nied, CIO with Radiology Ltd.

"Our information is sensitive and highly regulated," says Billy McDonald, VP of IT with E Federal Credit Union. "I can keep it locally, have more control, more security, for less cost, and simply feel better about it."

Among those companies open to public storage services, they're most likely to consider these services for backup, disaster recovery, and file and e-mail archiving. When it comes to primary data storage, the numbers fall off dramatically. Scroggins says he sees small and midsize businesses as the sweet spot for storage service providers, whose data protection practices may be an improvement over what an SMB could afford on its own. But as we hear about successful breaches of corporations from Citi and Nasdaq to PBS and Sony, we can't help but think large enterprises should compare their security with what a storage provider like EMC or HP could provide.

Bells And Whistles

Survey respondents moving toward outsourced storage named more than 30 vendors they're using or considering; we discuss them in more depth in our full InformationWeek Analytics report. Meantime, when considering storage services:

>> Drill into integration capabilities. No one wants a new silo--creating a unified environment and ensuring you have the ability to freely move data between off- and on-premises storage is the most critical feature.

>> Insist on real-time, on-demand access to data that enhances the business. This includes unified search.

>> Be realistic about regulatory requirements. Using FUD to avoid considering storage services is a good way to lose credibility. "Our biggest concern with public cloud storage is strict laws on retention polices," says one respondent. "A simple policy from the cloud vendor to use deduplication could break state laws regarding original file source." Now, we're not providing legal advice, but right now a government agency could ask any online service provider to give the names of all users who have a particular file, whether or not the provider employs deduplication. So check with your legal counsel before issuing blanket statements.

>> Still, your IT organization must master encryption and other security elements--no matter where you keep your data. The current standard is AES 256-bit encryption, but that will add a level of latency when data is unencrypted back within your network. As we've seen, "on premises" doesn't always equate to "strong security." Wherever sensitive data lives, make sure it's encrypted.

>> Ensure that retention and deletion policies will be carried over and enforced. Understand if a record or backup of data is kept when information is removed. Otherwise, e-discovery could land you in hot water.

>> Monitoring tools should clearly show usage and performance statistics and cost breakdowns, as well as provide a way to export these stats for reporting purposes. Monitoring systems should also provide visibility into SLAs. Since there's a lot to the management aspect (SLAs, usage, compliance, integrity reporting, integration), make sure that the interface is easy to use--during an outage isn't the time to find out that you can't get insight.

>> The importance of geographic redundancy was exposed by the recent Amazon data center outage. You must understand how a storage service provider will deal with major disruptions. Ensure you have a formal notification mechanism that kicks in if the provider changes its architecture, and negotiate the right to conduct regular reviews. The smaller your company, the less willing the vendor will be to accommodate such requests, but hold out for key considerations, including the location of your data, plans for backup and continuity of operations, speed to get your data, application access mechanisms, encryption at rest and in flight, physical and data security policies, and fees and triggers for adding more space and network bandwidth.

>> You never know which applications will need storage resources in the future, so put a variety of communications mechanisms in place. Vendors should support a variety of Web services APIs to enable a service-oriented architecture as well as transfer protocols such as SCP, SAMBA, and CIFS.

>> If you expect a lot of bursting, spell out the cost of quickly and seamlessly increasing the amount of data stored with the provider, along with any caps or advance notice required. And be sure that if you scale up, you can scale back down. Many vendors make adding capacity easy; removing it, another story.

Your Strategy

Eventually, we expect storage services will become just another tier, used for specific needs. How extensive that use will be depends on how well vendors reassure CIOs on security and availability. Another roadblock: We've yet to see a product that does a good job managing a hybrid, multivendor set of enterprise storage as a unified environment.

Still, we have to do something. It was only a few years ago that we started talking about terabytes. How long before we start thinking in terms of exabytes, zettabytes, and yottabytes? A few years? A few months? At one client, we saw 80% growth per year.

Unfortunately, instead of seeing legacy devices inherit cloud-like attributes, we see companies treating the cloud as just another silo. Seek to create a single pool of virtualized storage that leverages existing IT investments. Doing that might involve using private, public, and hybrid systems. Watchwords are reliability, scalability, multitenancy, and multitiering. Bake search, migration, and archiving in--and develop requirements before considering vendors.

Continue to the sidebar:
Mobile Device Backup--Still A Work In ProgressContinue to the sidebar:
Following Uncle Sam Into The Cloud

InformationWeek: Jun. 27, 2011 Issue

Download a free PDF of InformationWeek magazine
(registration required)