DR Fans: Few Black Eyes From Blackout

The massive power outage that hit the northeastern portion of North America last Thursday not only sent firemen scurrying out to save people stranded in hot and dark subways and elevators, but also had IT administrators across the region desperately trying to save their data and get operations back up and running.

For many small and medium-sized businesses it was, of course, a lost cause. Companies and organizations with no disaster recovery solutions in place had no choice but to shut down operations after the lights flickered out and their laptops began running out of juice.

While the full financial blow from the blackout has yet to be recorded, it's clear that the involuntary shutdown will carry quite a price tag. According to Chicago-based retail analysis firm ShopperTrek, for instance, retail outlets in the affected area lost at least $30 million in revenues on Thursday alone. And the New York City Council's financial department estimates that the city lost $750 million in revenues during the power outage.

But while some companies were sweating over lost revenue and the lack of AC, others claim generators and foolproof disaster recovery plans kept their business up and running throughout the ordeal.

One such company was Commerzbank. The German bank not only has a generator at its primary New York offices near the old World Trade Center site, but also a disaster recovery site in upstate New York. "We were one of the only banks open in the area," says Rick Arenaro, the bank's VP of information systems.In fact, the bank was preparing to move its disaster recovery site from its location in Rye, N.Y., to the third location about 10 miles farther away when the blackout hit. While the bank hadn't completed the move yet, it had already started replicating data among its EMC Corp. (NYSE: EMC) Symmetrix DMX2000 boxes at all three sites, according to Arenaro. When the power went out, he says, generators at the primary site and at the new disaster recovery site kicked in, while the old disaster recovery site went dark within minutes.

"It didn't cause problems, since we had a three-site replication solution in place," he says. "And once the power was restored, we were able to get the data back in sync within about an hour. It's good to know we could get the data back so quickly."

The fact that all three of Commerzbank's locations were hit with a power outage at the same time last week is no reason to move the bank's disaster recovery site even farther away, Arenaro says. "It hit Canada as well," he points out. "The question is: How far is far enough? You have to balance practicality with what you're trying to accomplish. We were able to resume business that day without a hiccup and were able to move back without a hiccup... I think our configuration works."

Another company that managed to keep business up and running last Thursday was Canadian-based Telus Corp. (NYSE: TU; Toronto: T). Thanks to backup power generators, the company's two data centers in Toronto didn't go down for a minute, according to Shawn Myron, a product manager for Telus Business Solutions. "I don't get worried about blackouts," he says. "We can run on generators for months if we have to. None of our customers experienced any downtime."

Telus, which uses Sun Microsystems Inc. (Nasdaq: SUNW) and Hewlett-Packard Co. (NYSE: HPQ) storage products, is planning to implement a remote disaster recovery solution by the end of this month, using Nortel Networks Corp. (NYSE/Toronto: NT) and Nishan Systems Inc. equipment that will replicate data between Toronto and Calgary.Power Struggle

Not all companies operate their own disaster recovery sites and networks. Some rely on disaster recovery services from companies like SunGard (NYSE: SDS) to get them back up and running in the case of an emergency.

According to SunGard's senior director of marketing, Judith Eckles, 65 of SunGard's 10,000 customers worldwide "declared disasters" during the blackout, while more than 100 more put the company on alert. Most of those companies shipped their systems to one of SunGard's sites, but at least three of those companies were high-availability clients, which had been mirroring their data to SunGard sites.

Once employees managed to get to one of the company's mega-centers, which are fully generator powered, they could set up their workstations there, Eckles says. "We had more than 2,100 people in seats... in Philadelphia, Long Island, New Jersey, Chicago, and Canada. With the exception of September 11, this was the largest disaster that SunGard has ever supported."

But what about companies that can't afford their own disaster recovery site or a disaster recovery service? "Everyone (including yours truly) always gravitates towards the big sexy stories about huge Wall Street data centers and whether or not the markets will be up and running, etc.," says John McKnight, an analyst with the Enterprise Storage Group Inc. "What about the tens of thousands of small and medium-sized businesses that may or may not have ANY data protection strategy?"According to a survey of 200 IT directors at variously sized companies and in various industries, conducted by removable data storage maker Imation Corp. between March and July this year, one third of all businesses are operating without a disaster recovery plan. The survey also reveals that 66 percent of all companies feel vulnerable to certain kinds of disasters.

The results of the survey, which will be released tomorrow, also reveals that 64 percent of all companies that do have a disaster recovery plan in place don't conduct external audits of their systems on a regular basis. "That number just seems high," says Jim Ellis, Imation's director of business strategy for data storage and information management.

Imation's survey indicates that some companies that had a disaster recovery plan in place last week probably had a nasty surprise, he says. "There's definitely a risk. I hope that number will go down."

Eugénie Larson, Senior Editor, Byte and Switch