When Slaying Legacy IT Costs, The Data Center Is A Good Place To Start

Is your IT department a center of innovation? Or is it a cost center or, worse, a money pit that requires a growing budget just to keep the lights on and the hard drives spinning?

The answer may hinge on whether you've cracked the 80-20 conundrum of IT spending--that around 80% of the typical IT budget is spent on management, operations, and maintenance, and only 20% goes to new technology and innovation. For companies serious about improving that ratio, one of the first places to look is the data center.

The business need is obvious. When $8 out of every $10 goes to routine, day-to-day operations, it's easy to understand why many businesses view IT as a support service and not an innovation engine. IT managers who aren't actively working to change their spending patterns shouldn't be surprised when they're not invited to C-level strategy sessions and instead are relegated to the kiddie table.

The tactics to change that 80-20 ratio in the data center are well known: inventory, standardize, consolidate, virtualize, automate, and enforce measurement-driven best practices. But they're hard to accomplish, and there haven't been many good technology tools to help--until now. Most of the major systems management vendors, including BMC Software, CA, Hewlett-Packard, IBM, and Opsware, offer products that reduce the drudge work that consumes a lot of IT staff time and much of the IT budget.

Many of these management systems have tools that automatically inventory IT assets and map them to critical applications and business processes. Change or configuration management databases, for example, track each server and PC and their applications, configurations, access rights, and other key characteristics. If a problem occurs, say with a software update or security patch, the database can be used to roll back a server or PC to an earlier and stable configuration. Once a company settles on a standard configuration for servers and PCs, these databases also can be used to quickly set up and configure dozens of computers, saving the IT staff from doing it manually.Companies have been able to create virtual servers for years, but new management tools make it far more practical even for some very small businesses to virtualize computing and storage resources and dynamically allocate them as needed, so financial systems get more horsepower at the end of a quarter and retail systems get more juice during the back-to-school selling season. They can automatically enforce security policies and monitor network and application access. And they can let you know when and where you need to add more resources by doing trend analysis of usage rates. The goal is not only to cut IT infrastructure costs but also to provide guaranteed service levels to the entire business.

Space Cadets
Many companies are consolidating and virtualizing their servers simply because they've run out of space as they've followed the old one server, one application approach. Michael Shimmin, data center coordinator for University Health Care at the University of Utah, says 80-20 is a pretty good estimate of how his IT budget is divided between maintenance and new projects. But it would be much worse if the IT department hadn't consolidated around HP blade servers and virtualization software from VMware.

The university's health care IT department a year ago faced multiple problems in its data center, a 4,000-square-foot facility that serves the university hospital and clinics, five regional hospitals, and two dozen outpatient clients. It had no room to grow, and its electrical system couldn't handle much more capacity, yet it needed 80 new rack-mounted servers. Shimmin knew the building couldn't handle that, and it would have taken five years just to secure the funding to build a new consolidated data center.

Shimmin developed a plan with HP, VMware, and IT reseller Avnet to move the data center to blade servers powered by dual-core processors. The university funded an update of the facility's electrical system to allow more power. And Shimmin began consolidating around HP blades, which let him pack more servers in a smaller space. He also used VMware software to create about 20 virtual servers on each physical server.

The department also bought HP's OpenView Performance Insight, a central console that lets the IT department monitor usage rates on each physical server to identify the best candidates for virtualization.There's more work to do. Shimmin's investigating new cooling options from American Power Conversion, Liebert, and other vendors to handle the heat generated by the densely packed blade servers, as the department tries to keep this data center in operation until the university funds a major overhaul.Inventory Time
Before companies can consider consolidation and virtualization, they must make sure they have a handle on what IT resources they have and how they're being used. Which server is running the business-critical ERP system and which is being used for an internal test bed? Which server is operating at a low usage rate, making the application running on it a candidate for moving to a virtual server on another physical machine?

For many companies, consolidating data center resources is the first major step toward bringing costs under control. In an IDC survey of 400 senior IT executives, 60% view consolidation as an important step in improving operational efficiency, and 80% are engaged in a consolidation project. The goal is to to free up dollars and personnel from maintenance and management tasks and create an infrastructure flexible enough to react to business changes and problems, IDC analyst Matthew Eastwood says.

The language of data-center management also has evolved. Companies use Six Sigma or similar business process improvement programs, but the Information Technology Infrastructure Library increasingly is the framework for creating auto-mated processes. ITIL includes definitions and workflows for operations such as change management, configuration management, and software release management.

A major commitment of time to ITIL helped Eric Martineau, VP of Web operations for MyFamily.com, and his IT staff of 30 deal with the company's fast growth. In the past two years, MyFamily, whose main Web site is Ancestry.com, a genealogy service, grew from 750,000 subscribers to 1.6 million, from 103 terabytes of storage to 600, and from 1,200 servers to more than 4,000.

Martineau bought ITIL service support and delivery manuals, as well as the Microsoft Operations Framework manuals, for each IT staffer. For five months, the team spent lunches working through the manuals and brainstorming how to best implement the practices within their IT environments. "Implementing the processes and restructuring the department around those processes really didn't cost a dime," he says. "In the end, we were able to get a better definition of what responsibilities were to be assigned within each department."MyFamily was able to cut equipment costs by implementing a strategy around building its own blade servers, and it's now adding about 250 blades per month. Its biggest hardware outlay is for storage, which it mainly buys from Network Appliance. The final piece of the effort was to deploy CA's Unicenter system management suite to ensure that the ITIL and Microsoft Operations Framework and processes were implemented and enforced. The payoff is the company now manages more than 4,000 servers with only a half-dozen system administrators, Martineau says.

Small-Biz Frustration
GHY International shows how companies don't have to be big to use advanced data center tools to get IT maintenance costs under control. GHY, an import/export broker, has a small IT staff that was spending 98% of its budget on operations and maintenance, making it impossible to launch new technology initiatives, VP of IT Nigel Fortlage says. "We got to a point of total frustration, where we were doing nothing as an IT department," Fortlage says. It was costly to the company--and killing morale.

GHY got into that jam by dealing with IT problems on a piecemeal, reactive basis, he says. Every time a problem cropped up, the easy answer was to throw a new server or software program into the mix. The result was a hodgepodge of systems that required lots of time to manage. Options available at that point included doubling the number of servers or doubling the size of the IT team, but there wasn't money available to do either.

Fortlage consolidated seven Linux, Windows, and Unix servers onto one four-processor IBM iSeries 550 server, which was partitioned into 17 virtualized environments. Half of the virtual servers run Novell's Suse Linux Enterprise Server 9; the other half run Windows 2000, AIX, and IBM's i5 operating system. The consolidation helped GHY cut its IT budget by 14%, giving the IT team credibility to add a person to the help desk. "It wasn't a hard sell," Fortlage says.

Another way companies are knocking down data center costs is by using fewer vendors and standardizing hardware and applications to cut management problems. With more than 1,600 computers, ranging from mainframes to AS/400s to Unix and Windows servers, YRC Worldwide, a large transportation services company whose brands include Yellow Transportation, Roadway, and Reimer Express, has embraced consolidation and virtualization. YRC calls itself an ITIL shop. "It gets things done a lot quicker and faster," says Kim Galway, team leader for the company's performance management and capacity planning team.Galway's team uses BMC's systems management tools to monitor system performance and plan for additional capacity. The same BMC tools monitor all the types of hardware that YRC uses, so a small set of people can monitor operations rather than having a specialist for each system. Plus, YRC has automated most of the performance monitoring. Staff is alerted only when problems occur that the system can't take care of. That setup has let YRC cut the seven-person team to five even though the company made two acquisitions during that time.

IDC estimates that 70% of the average IT budget is spent on people. So companies that want to change their maintenance-to-innovation ratio must be ready to ask people to do new kinds of work and to let go of those who can't make the transition.The Outsourcer's view
IT outsourcing vendors are particularly sensitive to their operations and maintenance costs, since their contracts provide a fixed income, says Shahin Pirooz, CTO at CenterBeam, which manages IT infrastructure operations for small and midsize businesses with 100 to 2,000 servers. "The only way I make more money is to increase margins by increasing operational efficiency," Pirooz says.

CenterBeam focuses on key metrics such as the ratio of administrators to servers, networked devices, PCs, and e-mail accounts served. Working with HP to move to a virtualized blade server environment, CenterBeam has improved its admin-to-server ratio from 1-to-20 three years ago to 1-to-150 today. Pirooz is shooting for a 1-to-200 ratio by year's end, with a long-term goal of 1-to-400.

Most IT leaders can relate to living on a fixed income. Their companies aren't keen to pump a lot of new dollars into technology, especially if it isn't producing much of a return. So the pressure is on to lower IT maintenance costs in order to free up funds for innovation.

The data center's a critical place to start; the tools exist to gain efficiencies by consolidating, virtualizing, and automating the management of servers and extending and automating the management of desktops and remote users. Most IT departments are spending around 70% of their budgets on personnel, and that probably won't change. But, at many companies, the work those IT staffers are doing can and should change. Instead of simply keeping the computer lights blinking, they need to be developing and deploying new features and capabilities, providing new tools and resources to the company, and--ultimately--generating new revenue. Once that happens, IT will get the respect it deserves.-- with Paul Travis

Continue to the sidebar:
Systems Management Vendors Add Features To IT Toolbox