Lessons We're Bound to Have to Repeat

At a recent storage conference in Frankfurt, Germany, I found myself confronting numerous inquiries from attendees, not only about storage technology, but also about Hurricane Katrina and its deeper ramifications. Everyone seemed to appreciate the awesome force of nature that was brought to bear on the Mississippi Delta.

A Category 4 hurricane is nothing less that a fractional kiloton nuclear explosion, horrific in its impact and devastating in its longer term effects, as television footage of the devastation clearly shows. But, that wasnt the issue on everyone's mind. Unlike other significant disasters, such as the Northridge earthquake, Hurricanes Andrew, Hugo and Ivan, and of course Sept. 11, the world did not see the familiar outpouring of compassion and selfless support by neighbors of the victimized area, or the quick, decisive intervention of authorities in the hours--or days--after the emergency.

One fellow said he had always admired the way that Americans came together in a crisis and pulled themselves up from the most horrendous calamity. This time, however, he was sickened by both the protracted delay in emergency response and, even more so, by the violent images.

Doubtless as the post-mortem is done on Hurricane Katrina, as happens with all disasters, many of these theories will be validated to whatever extent that they reflect reality, or they will be put to rest. My only contribution to the discussion was and is the following. First, there are protocols (that those of us in Florida know all too well) for coping with hurricane threats. We are very familiar with this kind of disaster, and the drill is fairly well established.

First, hurricanes provide significant advanced warning. The threat of a storm making landfall is known for days before it happens. That's a good thing, since the alert gives us all time--from private citizens, to businesses, to governmental emergency managers--to take some preparatory steps.Voluntary evacuations are typically announced while a storm still has a fairly low probability of hitting anywhere near our location. The objective is to remove people from a disaster-prone area as early as possible. The more people who head for the safety of higher ground early on, the fewer you need to worry about as the storm draws near.

For businesses, the voluntary evacuation order is the time when disaster recovery plans need to be put into some higher level of readiness or activation. This may include performing last-minute backups or increasing mirror split or snapshot frequency, and possibly even putting a hot site or commercial recovery center on alert, if the company uses such a service. It also involves sending some non-essential staff home early so they can begin boarding up their windows, packing essentials, and getting their families out of harm's way.

Over the next few days or hours, as the threat becomes clearer and probabilities of a direct hit become more compelling, the state or local government usually issues a mandatory evacuation order. This is an instruction to everyone to board up and vacate immediately. Given the population densities in cities and towns around the very appealing Gulf Coast region, we are talking about massive population relocations.

Emergency services get down to business after the mandatory order. They begin the difficult process of filling and balancing the load of displaced persons at evacuation shelters. If it hasn't already been done, they implement the logistics of moving hospital patients, the elderly, and others who cannot effectively fend for themselves. The logistics are never perfect, but if everyone cooperates, things usually work out.

For businesses, this is a time to power down, pack up, and transport their backup data and business continuity personnel to their recovery centers. In cases of companies with redundancies built into their IT infrastructure, it is time to shift workload to alternative processing sites."Secondary" Doesn't Mean Next DoorIt is essential when planning your continuity capability to ensure that your backup location is sufficiently distant from your primary facility to avoid being consumed by the same disaster. This lesson has not been learned by many companies in the Gulf Coast region, unfortunately, perhaps because its been a while since a major hurricane hit a key business area or a major city in the region. Hurricane Andrew, for all of its devastation, hit a mostly suburban area of southern Florida, sparing Miami and Fort Lauderdale the direct force of the storm.

It is worth noting, however, that businesses in the Gulf Coast states are not unique in this respect. Despite the horrific experience of Sept. 11, most financial services companies tend to have primary and backup data centers that are in sufficiently close proximity to expose both sites to the same disaster, a "dirty bomb," for example. The Federal Financial Information Examination Council (FFIEC), which had the opportunity to set minimum safe distances between primary and backup facilities following Sept. 11, failed to do so for reasons that have never been fully explained.

Katrina was only a Categry 1 hurricane when it kicked Miami in the teeth. I know this because I rode it out in a Holiday Inn in Coral Gables, courtesy of a cancelled flight on American Airlines and the subsequent closure of Miami International Airport.

We lost power in the hotel at about 8:30 p.m. Telephones were dead (no switchboard power) and cellular connections were dicey, depending on your carrier. Outside, the darkness of the night was accentuated by the howling wind, oppressive heat, and periodic blasts of rain that actually stung like pebbles when they hit your skin. The only other sound was that of car alarms, which seemed to be crying out from the hits they were taking from felled tree limbs and other flying debris.

Fortunately, power was restored at around 5:30 a.m. the next day. The difference in the condition of the world from what it had been at sundown the previous day was amazing. Decorative canopies on boutiques and restaurant storefronts up and down the street were shredded and many un-shuttered windows were broken. The roads were covered in palm fronds and garbage from toppled trash cans. A Miami Herald headline proclaimed, "Our Luck Runs Out."Louisiana and Mississippi were even more unlucky a week later when a re-energized and strengthened Katrina gave the region pummeling that will probably go down in the history books as the most expensive disaster in U.S. history. The loss of life is saddening and aggravating, in part because it shouldn't have happened. Protocols were observed and emergency evacuations were implemented, but compliance, apparently, was sketchy. The good news is that casualties appear to be significantly lower than originally estimated, but that is small consolation to the families of those who were lost.

The business cost of Katrina is still being tallied at present, but many insurers are nervously re-examining bankruptcy laws to explore their options. The upside: If Florida's hurricane experience is any indication, the recovery will likely produce new revenues and jobs for the region and may well create an economic boom for those businesses that return.

Why We Never Fail To FailIn the final analysis, the victims of the storm will be the people who were too complacent or ornery to leave when they had the chance, and the businesses that knew the risks all too well and failed to plan for them. Hopefully, everyone will take a lesson from this horrific event and begin doing something about business continuity. But if experience is any guide, this will not happen.

Whether as a testament to Teddy Roosevelt’s observation that America does not learn from experience, or just the fact that the human mind tries to forget bad experiences as quickly as possible, no disaster event has ever produced a sustained trend toward improved disaster recovery planning. A grim reminder of this came just a few weeks ago on the fourth anniversary of Sept. 11. Within a year, surveys reported that fewer than 50 percent of companies had disaster recovery or business continuity plans, and of those that did, fewer than 50 percent had ever tested them.

Counting on luck to weather the storm is a fool's ploy. Those businesses that survive disasters are invariably the ones that make their own luck. If you do nothing else to prepare yourselves for the possibility of a disaster, at least make plans to protect your people and your data. In the end, they are your most irreplaceable assets.Jon Wiliam Toigo is the founder of Toigo Partners International LLC and a regular contributor to Storage Pipeline. He can be reached at [email protected]