Storage Pipeline: A Look at the Storage Professional

Networked storage has raised the level of responsibility for the storage manager. If one segment of your direct-attached storage infrastructure goes down, you lose access to a slice of your corporate data. But if you lose your SAN, most or all of your data becomes inaccessible.

In addition, new industry rules and government regulations have placed heavy demands on IT to ensure that business records are retrievable nearly on demand, even if the records are years old.

Meantime, data volumes are soaring. Just a few years ago, a 1-TB data store was considered big. Now, some 20 percent of the ASNP's 1,600 members manage data stores larger than 100 TB, and Fortune 500 companies, such as Best Buy, are said to be approaching 1 petabyte (1,024 TB) of data, putting their storage needs on a par with that of the Pentagon. The amount of disk storage system capacity shipped worldwide in the first quarter of 2004 hit 247 petabytes, up nearly 40 percent from the same period a year earlier, IDC said.

There are many reasons for the data explosion. E-mail messages are multiplying by the day, and IT organizations often must archive all of them to comply with business and government rules.

Along industry lines, health-care facilities are digitizing X-rays and other patient records to reduce errors and physical storage space, while the federal government is pressuring the industry to digitize nearly all patient records in the next decade. Financial services firms are reorganizing how they store and share customer records in light of new privacy and disclosure regulations. Automobile and aircraft manufacturers are collaborating more closely with their suppliers on product design, forcing them to swap and store huge engineering files. And the threat of terrorism has everyone preoccupied with disaster-recovery planning.But just as the data explosion was occurring, IT budgets stopped growing. So throwing more capacity at the problem isn't impractical.

"Just because you froze the budget doesn't mean you stop generating data," says Steve Duplessie, an analyst at the Enterprise Strategy Group. "Managers started telling the storage guy, 'You're not getting any more money, but the data store is growing 30 percent a year.' "

Storage administrators had to work smarter with their infrastructure. The emphasis moved from adding capacity to identifying file redundancies and other inefficiencies.

The storage manager's role in an enterprise depends on the company's size. Big organizations are adding specialists with "storage" in their job titles, suggesting that they spend most or all of their time on storage tasks. In small and midsize companies, the storage guru is a server or network admin who manages storage as part of the job, says Steve Schuchart, an analyst at Current Analysis and a contributing editor to sister magazine Network Computing.

John Halamka, CIO of Care Group, a collection of hospitals affiliated with Harvard University, recently brought in two full-time storage staffers as part of a transition to a HSM (hierarchical storage management) strategy, under which data is shifted to less-expensive storage media as it ages. The transition cost about $2 million, chiefly for the purchase of EMC Symmetrix DMX and Clariion equipment to juggle 100 TB of data.In 1998, the medical group stopped using film for X-rays and went 100 percent digital. With each X-ray image taking up roughly 100 MB of storage space, demand for storage increased 250 percent in just the first couple of years. This forced Halamka to rethink his storage strategy for the $1.4 billion, 12,000-employee organization, where physicians make up a quarter of the work force. He and his staff came up with the hierarchical approach.

"If an X-ray is taken today, you need immediate access, so we put it on the highest-end storage device we have: the EMC Symmetrix DMX," says Halamka, a former emergency-room physician who doubles as CIO of Harvard Medical School. "After a year, it's not as vital, so we move it to an EMC Clariion, which is a little cheaper and less reliable--it has four nines of availability instead of five nines. After five years, you can afford to put it on a NAS (network-attached storage), for which we use EMC Celerra equipment. Beyond that, Massachusetts requires we keep things for 30 years. But after 15 years, what's the likelihood that someone will want a chest X-ray? So we put it on tape."

Medical images aren't the only data treated this way. "If an e-mail attachment hasn't been read in a week, we move it to cheaper storage," Halamka says.

The difference in the cost of media is substantial--some $14 per gigabyte for the DMX, compared with as little as 14 cents per gigabyte for tape. All data is centralized; user PCs don't even have local storage anymore.

Halamka hand-picked his two storage managers from within--one experienced with Windows Server, the other with Unix server architecture. Both managers have deep knowledge of backup configurations and Fibre Channel. They're responsible for overseeing the data store and educating others on sound storage practices. Halamka expects to hire several staff-level storage administrators to work under the two managers.Optimists might tell you that staff levels will decrease as enterprises adopt ILM (information life-cycle management), which automates the movement of data to different storage media based on business value. ILM expands on the concept of HSM by establishing a business priority for all data. Whereas HSM, a mainframe-era concept, judges the value of data only by its age, ILM takes advantage of metadata to distinguish, say, an e-mail about 2005 budget forecasts from an e-mail notifying the staff that bagels are available in the conference room.

But Schuchart isn't convinced that ILM will reduce the need for storage labor. "If anything, your staff levels will increase," he says. "ILM will require an incredibly complex storage environment. It won't alleviate complexity. The savings associated with ILM are in the use of transactional storage, not reduced staff levels." (For more on ILM, see "ILM: Panacea or Proprietary Poison?,".)

That said, midsize companies may see a consolidation, rather than an expansion, of their departments. James Taylor, a systems networking engineer at New Process Steel Co., Houston, became the resident storage guru when his shop started moving from a departmental, direct-attached storage architecture to a SAN architecture last year. Now, the company's storage is centralized on a SAN using EMC's Clariion CX 400 with Cisco Systems MDS 9000 switches.

The SAN replaced even the primary storage disks in the steel manufacturer's farm of Hewlett-Packard servers running Citrix MetaFrame. The servers no longer contain any resident storage. The company's 450 employees access their data and applications over a Citrix interface attached to the SAN.

With this centralization Taylor can manage the storage infrastructure almost entirely by himself. And he can provision new users quickly. When we spoke, he was about to add a Citrix server for 50 users, which he said would take all of 15 minutes."It's far more effective to build a SAN and have a couple of people on staff who are crackerjack on it, versus 10 people running around putting out NAS fires all day," Taylor says.

Taylor says his background as a network and server admin gives him a holistic understanding of data bottlenecks. This knowledge will serve any storage professional well, especially when you consider the differences between the storage management and systems administration disciplines, some managers say.

"Admins are into fire fighting; they like the adrenalin rush," says LeRoy Budnik, a managing partner at Knowledge Transfer, a systems integrator in the Chicago suburbs that also offers courses on storage management. As a result, he says, they tend to wing it when it comes to configuring equipment rather than adhere to a standard set of parameters.

When Budnik hires storage managers, he looks for people with a history of documenting best practices in an organization. "If you document what you do, someone else can step into that job if you're not around," he says. This is important if you want to relieve your storage staff from having to answer every call about a lost file. The helpdesk should be able to take care of such calls, Budnik says.

He also looks for a history of crisis management. He doesn't want someone who boasts that he has never lost data. "You don't have a full appreciation for it unless you have experienced it," Budnik says. "You're always just a few commands away from wiping out the whole Hitachi frame. If you don't respect what you could do, how can I trust you? It's like electrical work--tell me about a time you were shocked."Storage management also requires a better conceptual understanding of networking than most network admins have today, Budnik says. The storage role is perhaps closer to that of a network or systems architect.

"It's like the TV show 'America's Test Kitchen,' " he says. "They teach you how to make the food, but they also say, 'Here's the science underneath and why it tastes good.' "

Experts agree that the storage pros with business acumen will make the best livings as the specialty matures. It should come as no surprise, then, that one of the Big Three auto manufacturers with which Budnik does business made a finance person the head of storage strategy, with six technical managers reporting to him. "They're treating data as an asset, so they brought in a guy who understands what you do with assets," Budnik says.

It's also important to understand an industry's regulatory requirements. If you're bound by the Sarbanes-Oxley Act, which tightened disclosure and privacy requirements for financial services firms, you may be required to furnish data in as little as 48 hours--which means handing over the data to your lawyers in half that time so they can review it first.

Salary OutlookExperts and storage pros take an optimistic view of the salary prospects for storage specialists. Many liken the storage role to that of a security or database specialist, whose salaries tend to be higher than average. Because storage is an emerging specialty, experts are in short supply, the thinking goes, so companies must pay top dollar if they want a true storage guru.

A survey of more than 10,000 IT professionals conducted by sister magazine InformationWeek tells a slightly different story. The 5,321 staff-level IT personnel who listed their incomes reported a median salary of $71,000. For storage pros alone (2,263 in all), the median salary was $68,000. Yet, storage pros with particular expertise with EMC, Hitachi, IBM and Sun Microsystems equipment reported salaries above the overall median--as high as $82,000 for EMC gurus (see charts).

As certifications take hold, the role of the storage admin will become commoditized, and salaries will remain on a par with other admin roles, Budnik says. Only storage experts who can provide business as well as technical expertise can expect to earn up to 20 percent more than other admins, he predicts.

Training and Certification

As the storage industry matures, a battle is brewing to establish a standard set of certifications, much as a CISSP certification demonstrates expertise in security.Today, the vast majority of storage training is offered by vendors, chiefly EMC, HP, Hitachi, Sun and, increasingly, Dell. It's a hallmark of an emerging specialty: The equipment and many of the management controls are still largely proprietary, and IT shops tend to need specialists in one platform or another.

But the nonvendor options are expanding. For example, the ASNP has begun offering courses focused on specific storage technologies and products. Its officials argue that as a user organization, the ASNP is closer to the real problems that enterprises face and isn't tainted by vendor interests.

The Storage Networking Industry Association, a vendor-led group, says it's in a better position to establish standard practices because it's closer to the technology. "By the time the ASNP turns something into a certification, it's a year and a half later, and the problems have changed dramatically," says Budnik, who serves on SNIA's user council.

Opponents of vendor-led certification say that's precisely the problem. Such programs aren't practical because they focus on emerging technologies. What these managers say they really want is training on their equipment.

SNIA started a new certification program in June. The program is separated into four categories: concepts (basics), standards (technical standards), solutions (assessment and planning) and products (vendor offerings). Topics within these areas include storage management, virtualization, IP storage and disk technology. The program is available in any of 3,500 testing centers worldwide.The goal is to standardize much of the vendor training that's available, to reduce redundancy and take a load off the vendor programs, says Peter Manijak, SNIA's director of education. McData has agreed to make a portion of SNIA's program a prerequisite for its own education programs.

Other organizations vying for the certification lead include two geared toward data center pros: AFCOM and the Institute for Data Center Professionals.

In the absence of a certification that everyone can agree on, experience counts most. "For me, certs don't do it," says Lev Katz, data center operations manager at Mid-America Bank in Naperville, Ill. Katz is looking for new hires to help him manage a 30-TB data store that's growing at a rate of 100 percent annually. "It's hands-on," he says. "I want to see what they know."

Other hiring managers agree. Calpine's Zitterkopf looks for an engineering background. He's in the process of rolling out an enterprise storage plan for some 70 TB of data.

"I want people who understand the systems that are interfacing with the storage systems: Unix, NT," Zitterkopf says. "I want people who can engineer solutions."Others like to see a more diversified background. Jerome Wendt, a senior storage manager at a Fortune 500 financial services company, wants three or more years of sys admin experience, and perhaps another year or more of database experience. But he also favors someone who has a liberal arts background like him.

"I want someone who can see things beyond black and white and who realizes that there are many ways to solve a problem, and you have to find the one that makes the most business sense," Wendt says.

To Wendt, you need a multidisciplinary approach to advance as a storage guru. You need to know storage arrays, the OS, the network and how each application interacts with the hardware.

"So it's really more of a discipline than an art or science," he says. "You have to pick someplace to start and work outward."

Career OutlookThe best opportunities for storage pros, says ASNP chairman Delshad, are in small and midsize companies, where data stores are expected to double in the next two years, with some having grown from 2 TB to 10 TB in just a year or two. Such companies will require new data centers and backup strategies, Delshad says--along with people to do the job.

That said, you'll be hard-pressed to find storage job titles among small and midsize IT shops. Whereas large companies have the budgets to create dedicated positions for specialties like storage, security and network administration, small companies require jacks of all trades.

Either way, the career prospects for storage pros will be commensurate with the value of data.

"The data is more important than the server," Delshad says. "You can replace the server but not the data."

New and growing technologies like Infiniband and iSCSI will create roles for people inside storage, practitioners say. And even before those technologies become mainstream, there's plenty of work to be done. Duplessie, the Enterprise Strategy Group analyst, estimates that roughly half the data managed by corporations today is stored in direct-attached equipment that is not networked."This niche is changing so fast," says Brian Miller, a storage architect at M&T Bank in Amherst, N.Y., who started working with storage in 2000. "I haven't seen any other area of IT change as quickly as this has, from the size of the drives to the amount of data that needs to be stored to what the backup software can do. Storage is the place to be right now."

David Joachim is group editor of the Network Computing Enterprise Architecture Group. Write to him at [email protected].