Blowing In The Wind: Page 2 of 5

When it was all over, almost every network in Florida not in a category 5 data center was offline. Only the large data centers that were built for the Internet boom remained operational. In Florida, we have lived through so many hurricanes that we think we know how to prepare for them, but Wilma caught everyone off guard. The first lesson, then, is to prepare for a category 5 storm at all times and hope for the best. The odds are no longer in favor of being spared for the season.

The same lesson was learned in Alabama, Louisiana and Texas with Hurricanes Dennis, Katrina and Rita. The city of New Orleans more or less ignored Katrina until it was too late. In the Big Easy, even superstition that the gods were going to protect the city put paid to good DR plans and preparation. As a result millions of records were destroyed and can never be recovered. Where businesses escaped the winds, the ensuing flood waters came in and drowned out the short-lived fortune of a sudden dodge to the east that Katrina took in the hours before land-fall. Hard disks just don’t do well under water.

Another painful lesson is that you cannot rely on your battery backups and generators to keep your networks and servers up and running during and after a storm. After the wild 2004 hurricane season, almost every business figured it needed access to a generator for the following year. The rush on generators was so intense in the 2004 season that even the Home Depot in downtown San Diego was sold out because people were going to every city in the United States to get a generator shipped to Florida or Alabama.

So as part of last year’s DR plans many businesses made sure they had generators for the 2005 season. But they left out a key ingredient: Generators run on gas. Virtually nobody stored enough gas to get through even five hours of blackout. You would think that gas stations in the "hurricane states" have generators on site to pump out the gas for other generators when the lights go out. Not so. Florida’s thousands of gas stations were just as stuck without power to pump the gas they had. Only the turnpike gas stations and a few private stations that had electricity were able to pump.

As a result, gas was rationed to $20 per car -- if you could find a station. The gas lines on the turnpike were more than 10 miles long in the first days after the storm, and at $3.20 per gallonyou couldn't get enough to matter anyway. Waiting in line for six hours just to get enough gas to power up one server for ten minutes is an insane proposition.

South Florida businesses by the hundreds then sent employees to the north of the state to buy truckloads of gas to run their business generators. One big insurance company which had all its critical servers on line in a data center still found itself offline because the power at the main office was gone and thus there was no way to connect the VPN to the data center. So no one could work, if they could get to work. Disaster recovery plans called for a massive generator to power the local server room, 500 workstations and one side of the WAN (routers, lines and the UPS). However, the firm ran out of diesel after the second day and had to hire a truck to fetch 50 barrels of the fuel from South Carolina.