To Your Health: A Healthcare Organization's IT Transformation

In general, today's IT organizations face a daunting litany of challenges, including what seems to be insatiable growth of demand for three principal hardware components -- servers, storage and network bandwidth. At the same time, data center environmental constraints in power, cooling and footprint make that growth of demand difficult if not impossible to accommodate. And all this comes at a time when budgetary constraints -- how to pay for that growth -- has become very difficult, and headcount constraints -- asking fewer people to do more -- can only go so far.Healthcare IT organizations are not immune to these general trends; in fact, they face even greater challenges, among which are: -- The diversity of both clinical and non-clinical existing applications is enormously complex. -- The digitization of medical information (such as medical images) is still ongoing, and huge growth demands (think storage) present enormous challenge. -- Process workflows still require tremendous work, in many cases for not only technical but also behavioral reasons. On top of all this, the microscope of public attention is now focusing on the healthcare industry as never before. So understanding how a particular health systems provider is dealing with IT transformation can serve as an illustration of how that process can really be accomplished To this point, IBM asked Virginia Commonwealth University's Health System (VCUHS) Information Systems organization to describe how VCUHS is going about the transformation of their IT infrastructure. As VCUHS is an IBM customer, the company is very familiar with what VCUHS is doing, and IBM felt that their story was worth further visibility. Overall, I have to agree that the VCUHS story offers worthwhile insight into how such a transformation is really possible.Virginia Commonwealth University Health System Works to Transform Its Data Centers The Richmond-based VCU health system supports five health science schools, as well as hospitals, including a teaching hospital and a critical care hospital, a 550-physician, faculty group practice and an 110,000 member Medicaid HMO. According to Greg Johnson, Director & CTO, VCU Health System Information Systems, the VCU Medical Center has two data centers 1.5 miles apart, providing a complex networking strategy, including Cisco wireless coverage supporting data and voice. The IT infrastructure includes over 200 TB of data, more than 700 Windows-based servers and almost 8,000 fixed and mobile devices (with over 500 computers on wheels). The VCU Medical Center supports hundreds of clinical and non-clinical systems, including electronic medical records (Cerner), enterprise resource planning (Lawson), scheduling (IDX), a PACS system, pathology and pharmacy. As with most IT organizations, the VCU Medical Center information systems grew organically over the years, following the common principle that new application requirements when fulfilled are simply incorporated into the existing IT infrastructure. That left the IT organization with a huge challenge: Although medical research and medical facilities are state-of-the art, resources for IT have been constrained. For example, a tarp has been used to cover some systems to protect against roof leaks. The biggest IT issues VCUHS faced are data center environmental constraints, constant infrastructure growth requirements and a small technical staff (the staffing ratio for servers is more than 140 servers per one IT administrator). Clinicians and patients understand neither data centers nor the problems that can occur, such as unwelcome outages. Simply put, there is no tolerance for downtime at VCUHS. Not only does that negatively impact clinical workflow but can also affect patient care and safety. No wonder then that the VCU health system describes its IT mission as "ensuring uninterrupted access to data."The challenge, then, is how to meet current and future growth needs while maintaining high availability to data and systems. A solid IT infrastructure was considered the foundation to this effort, and this required going from a reactive to a proactive approach. Two key strategies were to virtualize everything possible and standardize on core technologies. The VCU Health System: Meeting the Challenge of Growth

What are the steps required to meet these goals? Start with Networks -- The VCU Health System views the network as the backbone to its effort, which created some inherent challenges. Cisco stated that VCUHS has one of the most complex networks anywhere, going from zero to over 1 million square feet of Cisco-enabled wireless in three years in order to provide the mobility that clinicians depend upon such as for paging, cellular, telemetry, nurse call system and infant monitoring. Several vendors' solutions are included in this infrastructure, with Cisco and InnerWireless providing key technologies. Continue on with Storage -- Storage is a good example of the way VCUHS is leveraging traditional IT solutions in new ways, first with virtualization and then with standardization. Because of what it considered extreme environmental pressures, the VCU Health System felt they could not move gradually but had to become extremely aggressive in their adoption. This seems to be a good example of what Dr. Samuel Johnson said, "When a man knows he is to be hanged in a fortnight, it concentrates his mind wonderfully." Finish off with Servers -- Server proliferation was another problem at VCUHS with the growth resulting in the usual green computing challenges of power and cooling, as well as straining already overburdened staff with long installation time and no unified administrative system. Starting out, the VCU Health System considered its storage to be in disarray due to increased demand, multiple outdated systems, lower than desired utilization, and time-consuming, complex management processes. They needed an easy-to-manage, scalable, highly available storage solution that would fit in a reasonable footprint. The first step was virtualization, for which they chose IBM's SAN Volume Controller (SVC), a solution that provided a single, administrative interface for mixed storage arrays, improved storage utilization and complemented the group's server virtualization solutions. In practical terms, this new approach dramatically reduced the constant firefighting mode in which the three storage engineers previously spent most of their time. VCUHS's second step was to standardize on the IBM XIV disk system with one 80 TB XIV array at each of the primary and secondary data center sites. IBM's XIV virtualizes within the disk system, but the VCU Health System used IBM SVC for cross-platform virtualization. The benefits of this approach were a highly available, self-healing XIV architecture, lower power consumption and a much smaller storage footprint. One of the unexpected benefits was that when VCUHS substituted IBM's XIV for two older storage systems, the temperature in the data center went from 740 F to 680 F - just from storage, which theoretically does not have the same power consumption/heat production overall as servers. Note that VCU Health System was originally skeptical about XIV (as a new storage architecture), but agreed to IBM's request that they perform a proof of concept. The order given to the storage engineers was to break the XIV system, but they found that they could not. Overall, IBM's XIV actually exceeded the Health System's most extreme expectations. Another strategy was to use disk-based backup instead of tape (which has assumed only a tertiary archiving role at VCUHS). A combination of IBM's Tivoli Storage Management (TSM) and Virtual Tape Library (VTL) delivered an "incremental forever" solution that decreased backup times and improved reliability. More importantly, the solution greatly improved the time to recover files. For example, an administrator was able to recover a lost file in 30 seconds rather than the minimum 3 to 4 hours that it took previously.From a server virtualization perspective, VCUHS is implementing VMware ESX for green computing benefits, automatic load balancing, and failover. For example, an application can be seamlessly moved around to preserve availability and responsiveness to the user without the user even being aware of what had happened. From a hardware standardization perspective, blades are being deployed for their unified management system and high fault tolerance. The next step -- which the VCU Health System views as the final frontier -- is implementing a virtual desktop infrastructure to replace traditional, static PC desktops. Roaming profiles will allow clinicians to sign onto desktops or workstations whenever and wherever they are, simplifying ease of use and improving clinical workflow. Overall, the changes that the Virginia Commonwealth University Health System is making with the help of IBM are enabling it to meet the challenges of growth within the environmental constraints that they face. Much progress has been made, though they recognize that they still have more to do. The Virginia Commonwealth University Health System example is a perfect example of a situation where yesterday's IT assumptions and strategies were constraining today's requirements. While much-needed IT had been put in place over the years, it had left the Health System with a legacy burden that caused the environmental walls to close in at a time when demand was spiraling out of control. Unless VCUHS could solve those problems without further burdening IT staff or resources, the mission of "ensuring uninterrupted access to the data" would quickly become untenable. What VCUHS did was what other IT organizations can also do -- virtualize and standardize their IT infrastructure in general and their storage resources in particular. Although every IT organization is unique in its mix of applications, some elements are commonly shared, such as ownership of information and components of its IT infrastructure. Likewise, the challenges of growing while living within environmental constraints are nearly universal. The ability to apply general principles, such as, "virtualize and standardize everything, despite individual differences" is therefore important. That does not mean that everyone will do everything the same way. Now, others might not need the huge wireless system that the VCU Health System requires, but server and storage virtualization are already common and receiving a lot of attention. Selecting new storage systems and disk-based backup to meet specific needs are also doable actions.Notably three well-known IT vendors -- Cisco, IBM, and VMware -- played prominent roles in the data center transformation process. That is not to say that smaller players with unique technologies did not play a key role, but leaning in the direction of vendors with proven products, track records and support for major portions of the transformation was a critical element in process. That is not to say that all choices were plain vanilla. VCU Health System would not normally have considered XIV, but after careful examination found that it was the storage system that best met its particular needs. All in all, the lesson is that although the particulars of what VCU Health System did are unique to it, the general approach of using virtualization and standardization is applicable to all.