Disaster Recovery Lessons You Might Have Missed

Posted by David Greenfield
November 27, 2012

Plenty has been written about the disaster recovery and business continuity lessons learned from Hurricane Sandy--backing up your data, testing your preparedness plan and the like. Basically, it's the sort of stuff we all know. What's been interesting to me are some of the lessons you might not have considered.

Include the Human Element Eric Krell at Business Finance points out how the human element of a catastrophe can undermine even the best disaster recovery plan. He wrote about how employees' personal emergencies challenged the company's preparedness.

"Some employees needed to help elderly parents who had lost power for four days; others were bailing out flood-damaged homes, finding childcare in response to lengthy school shutdowns, frantically attempting to track down accurate information (for example, when power would be restored, how to file claims, where to get housing assistance), and much more..."

It's not uncommon for companies to be tripped up by their employees' individual struggles in the face of a disaster. A Mercer report indicates that while 62% of companies have business continuity plans in place to address workforce readiness, only 29% of those companies that experienced disasters (or continuity disruptions) in 2011 followed their workforce readiness plans precisely; 20% of these companies report that they did not follow their workforce readiness plans at all.

Beware Fuel Follies

Making sure you have enough fuel to run your data center is an obvious lesson, but simply having the fuel isn't sufficient; fuel won't help if the generators that use it get flooded. For instance, New York University's Langone Medical Center had to evacuate all 215 patients when its backup power generators were flooded during Hurricane Sandy.

Peer1 Hosting had its own fuel problems. Although the company had an auxiliary fuel tank on the 18th floor of its Manhattan location, there was no way to pump fuel from a fuel truck to the tank. "The fuel line coming out of the truck ... is four inches [wide], whereas the fuel pipe from the basement coming from the large tanks was a three-quarter inch pipe," Robert Miggins, senior VP of business development for Peer 1, told Tech Target's Beth Pariseau. "We got the fuel really close to where it needed to be, and all of a sudden we're looking at these different sized delivery pipes--neither we nor the fuel company had the proper coupling to tie into that existing infrastructure."

Instead, Peer 1's staff carried containers of fuel up the stairs. Independent contractors and volunteers joined them, and, in the end, 30 people were schlepping buckets to the 18th floor.

Consider Virtual Appliances

Virtual appliances are a useful tool for business continuity and disaster recovery. They can be downloaded in minutes, deployed in under an hour and deliver the same performance as a physical appliance. Many vendors that make hardware appliances also offer virtual versions. But virtual appliances do have one drawback--they need servers.

In the aftermath of the hurricane, Silver Peak (my employer) had a number of requests for optimization software, but in some cases clients preferred physical appliances, even though they took longer to deliver than downloading virtual software. Why? With data centers under water, some organizations simply lacked the server hardware to run the virtual appliance.

Go to the Cloud

A number of stories talked about how the cloud helped save their business. Chris Donnell, an IT manager with NYC architectural firm Diller Scofidio + Renfro, for example, told PC Magazine's Samara Lynn that he was in the middle of a companywide migration from Exchange 2003 to Gmail when Sandy hit.

"After Sandy, [when] the extent of damage all over the city [was clear], it was easy to make the call to change the location of mail delivery and have people operate from partially migrated mailboxes instead of waiting for power to our servers." Lynn also wrote that Donnell used other cloud services, including Amazon, to set up business applications that employees could access.

Always Remember

This lesson isn't about technology, but it's still important. Consider Eugene Cederbaum's reaction to his experience in the hurricane:

"What I experienced in just eight days without power or water gave me a first-hand appreciation of what too many people suffer for months at a time: I was cold, dirty, sick, and by the fifth or sixth day, terribly weary.... From my own Sandy experience, I must remember to stay aware of the plight of those for whom the absence of basic housing, food and clothing is a daily experience."

Amen, brother.

David Greenfield is a long-time technology analyst. He currently works in product marketing for Silver Peak.

Related Reading

---