LOS ANGELES -- VMWorld -- Dave Trupkin always thought disaster would come in the form of a fire or an earthquake. As senior systems administrator for the Las Vegas Valley Water District in Nevada, he's paid to consider such prospects -- as well as the contingencies.
But "The Disaster," as he refers to it, did strike over Labor Day weekend two years ago in the form of a data center power failure -- less flashy than a natural disaster maybe, but no less dramatic in its impact on the water utilitys operations, he told an audience here today.
System power was supposed to have been restored the next day, but it took almost three days to get servers, systems, databases, and applications humming again. And that was only because Trupkin and his department were able to scare up some very tough-to-find generators, which LVVWD then relied on for eight more weeks till the electrical problem could be identified and repaired. (Overheated, parallel power lines had burned through their insulation.)
The utility was no stranger to VMware products or the upside of virtualizing servers and applications. In fact, use of the vendor's ESX Server to restore servers quickly and relatively painlessly after The Disaster "made us heroes with management," Trupkin laughed. It also cleared the way for broader use of virtualization across LVVWD's information systems and use of a collocation facility 10 miles away, hot-linked to LVVWD's data center with fiber pairs.
The Disaster pointed up the woeful inadequacy of the utility's "Tape and Pray" DR plan, and it sensitized management, IT staff, end users, and customers to the need for redundancy, smarter and faster backup, and electrical powering issues beyond LVVWDs control.