A Virtual Lifesaver

In 2006, Christus Health reached a painful peak in its server-processing capacity within our eight data centers and their associated space, electrical power consumption, and cooling capability. Within these data centers, nearly 500 IT associates were managing more than 600 host-based applications, 15,000 desktop devices, and 2,000 servers (70% of which were in the primary data center in San Antonio, Texas) for 30,000 users. We also had a rapidly growing server and application base and rising implementation costs.

Christus Health is a Catholic, faith-based, not-for-profit health system comprised of 380 services and facilities, including more than 50 hospitals and long-term care facilities, 175 clinics and outpatient centers, and dozens of other health ministries and ventures. Christus services can be found in 60 cities in Arkansas, Georgia, Louisiana, Missouri, Texas, and Utah, and in Chihuahua, Chiapas, Coahuila, Nuevo León, and Tamaulipas, Mexico.

As a result of the IT capacity crunch, it was getting difficult to deploy new applications and upgrade hardware without impacting business. Also, as more of our clinical care depends on our IT systems, service outages would affect more than just the business of health care operations--they would affect clinical care itself.

One of the key challenges hospitals face is the need to provide high-quality, efficient patient care while keeping hospitals and the rest of the health care system operating smoothly. Consider, for example, the variety of information that affects a patient's record: In addition to critical care information, there are financial records, demographic records, insurance claims, and inventory accountability. Behind all of these touch points are innumerable workflows and processes.At Christus, we decided to conduct an assessment of the potential for server virtualization. By virtualizing a large deployment on a few highly scalable, highly reliable, enterprise-class servers, an organization can substantially reduce costs related to hardware and application purchases, services, and maintenance.

That assessment included an ROI analysis in our Microsoft operating system environment, where the majority of our applications reside and our processing takes place. The assessment revealed some interesting facts about our data center usage:

The results told us that we had a big IT environment with lots of systems, many of which we weren't using efficiently.

So virtualization appeared to make a lot of sense for Christus. To start the process, we selected the virtualization software: VMware ESX Server. Because of the number and diversity of applications we support, we selected an enterprise-class vendor for the hardware: Hewlett-Packard's ProLiant BL20P and BL25P blade servers. We then assessed what our optimum processing ratio would be. After studying seven to eight different models and configurations--from four-way processors with 4 GB of memory to eight-way processors with 128 GB of memory--we ended with a dual-core processor configuration with 10 to 12 GB of RAM, depending on the application. We estimated a minimum consolidation rate of eight processors to one blade to allow for spikes in processing and memory usage.

In addition, we added an HP EVA 8000 midtier storage system with 10 TB of initial storage to accommodate the pooling of resources and the virtual environment. Mission-critical services are run on an EMC Symmetric frame.After about 16 weeks, we had established three distinct VMware farms that supported many of our important functions, including our Meditech Health Information System application servers, support servers, core infrastructure servers, and servers for enterprise applications such as the Ansos nursing scheduling system, printing, and directory services. We chose not to migrate several applications that were deemed weak candidates for virtualization such as servers regulated by Food and Drug Administration requirements, applications with unique hardware requirements such as the Kronos TeleTime system, and systems that maintain high CPU or memory utilization and enterprise databases.

ADDED BENEFIT

On one hand, virtualization has been a way to make better use of our processors and improve system availability. It's also been a way to reduce the co-location space we've had to rent while we're building a new data center. We found it would cost approximately $3 million to upgrade our hospital facilities to get the power we needed in the San Antonio data center, and since we were physically limited to obtaining only another 2,000 square feet from the San Antonio site, we decided to build a new data center, which we hope to open by the end of this year or in early 2009.

Christus Health,
By The Numbers

For fiscal year 2007,
ended June 30, 2007

Revenue:$2.8 billion
Income:

$154 million
Data: Christus Health

Over the last year and a half, Christus Health has used virtualization to reduce 824 servers to 83 blade servers with only one application problem. We also decreased the San Antonio data center's electrical/cooling load by 202 kilowatts, even after we had redeployed some of the savings. For every kilowatt that we save in electricity directly related to server consolidation, we save another 80% related to cooling. We translated that to $125,000 per year savings in electrical costs (San Antonio rates). What we aren't spending on co-location space averages $200,000 in cost avoidance.

Christus has spent about $975,000 to put all the technology in place, including blade servers and virtualization software. We redeployed $1.5 million worth of hardware harvested from the consolidation. With our total cost avoidance near $3 million, our net cost avoidance is $1.8 million.

Our total cost avoidance encompassed three years worth of electrical source costs, depreciation avoidance of a data center build cost, and server reallocation and reserve capacity. The difference between the total cost avoidance and the hardware, software, and implementation costs equals $1.8 million (rounded off). In other words, we would have spent $1.8 million more than we did to provide the same level of processing and IT capabilities.NEED FOR THE HUMAN TOUCH
One factor to consider when performing a major technology migration is the human resources that will be required. Concurrent to rolling out these new technologies, we were adding many new systems and dramatically extending the reach of IT in clinical and nonclinical areas, both inside and outside of our hospitals and in new locations. The virtualization technologies don't decrease staffing because applications are still present--they just run more efficiently. If anything, we've needed more competent and specialized people to manage these new technologies in addition to training our existing staff.What we're doing to increase our availability across our entire processor complex has a direct impact on patient care. Now that we have a VMware farm established, we don't have to run through the time-consuming due diligence for additional hardware and application space when a new project comes in. We're more agile, can turn on demand faster, and see ROI more quickly. Our staff can access patient data on a moment's notice, and therefore offer better quality of care.

After having implemented server virtualization, we found an excellent opportunity to boost capacity with it for our workflow automation projects. The information management staff handles myriad tasks such as printing reports, coordinating the input of outside agencies,, and data entry across all departments, including admissions, billing, emergency department management, lab, medical records, pharmacy, and quality management. The group uses a scripting tool called Boston WorkStation to help automate those time-consuming and repetitive tasks. We're not talking a few scripts here, but rather 400 scripts throughout the organization.

Efficiency comes with a price. As requests for automating tasks poured in, server space dwindled. Eventually, there were 200 scripts on one server, which filled that server's schedule and prevented staff from filling requests for additional scripts. To solve this problem, Christus placed the scripting projects in the virtualized environment and realized an immediate boost to the capacity.

Having a single virtual server per region has enabled Christus to spread out its scripting tasks, making them more manageable. Taking the "many hands makes light work" approach, an agency update that we need to download and disperse into our health information systems across all regions, which would have taken eight hours to complete with a typical standalone script, now takes a fraction of the time when split up among several virtual servers.

One of the most common data entry tasks that the information management department is asked to complete is to script in annual raises. With anywhere from 1,000 to 5,000 employees in any one region, such a job would require four to five people over a week of data entry work to accomplish. Using a script, we can accomplish this task in much less time. Often a single computer working approximately 24 hours can script the raises for an entire region's employees, ensuring that the most important resource of a hospital, the staff, is rewarded in a timely manner.Being able to load additional scripts onto the virtual servers creates more opportunities for efficiencies with our workflow automation projects, as well as the system of shared functions within the projects themselves. We also have improved staff efficiency, data accuracy, and overall cost savings.

Although the Christus virtualization project went smoothly for a project of this complexity and size, there are still opportunities to share key lessons. These lessons can be categorized in the project life-cycle stages of planning, design, testing, migration, and transition to operations.

First and foremost, the time we spent planning and testing paid off. The study allowed us to glean key information regarding our biggest opportunities: payback and the ease to achieve the payback. As for design, the time we took to calculate the optimum virtualization farm configuration with regard to cost and performance also paid off.

Another success factor was spending the appropriate amount of time in the lab validating study findings and assumptions on migration and application performance, and ensuring that there was a quick way to swing production systems back to physical hardware in the event of a performance issue in order to isolate and troubleshoot the environment.

Achieving a mind-set with regard to application vendor support was important. In the early stages, we let a few vendors rattle our cages with statements regarding their lack of support for VMware. As we continued to push forward, we realized that most vendors didn't understand the technology, and the expertise actually lay with ourselves and our project implementation partner.We pushed the vendors to provide evidence of why they couldn't support a virtualized environment. Most of the time, the vendors didn't have the experience with virtualization. That gave us an opportunity to prove the technology for them. After migrating more than 800 servers to the VMware farm, we've had only one issue that forced us to reverse the decision to virtualize the application.

Finally, don't under- or overestimate the consolidation ratios that can be achieved. Conduct careful observation of application performance during testing and deployment, and build flexibility into the deployment and migration plan to accommodate variances.

George Conklin is CIO, Mitch Lawrence is lead application analyst, and Mark Middleton is system director of IT architecture for Christus Health.

Continue to the sidebar:

Action Plan: Putting Virtualization In Project Life-Cycle Stages Of Planning