Laws of Data Recovery

An IT manager who survived his first Florida hurricane season says it took a full year to prepare his law firm for the worst.

Ben Weinberger, director of IT for Fort Lauderdale-based law firm Ruden McClosky, found out about hurricanes the hard way when he started his job during hurricane season in September 2004.

Soon after his arrival, a hurricane knocked out power at the firm's Tampa office, which had been set up along with Fort Lauderdale as one of its data centers. Half of the company's IT operations were down for two days, and the only way to restore data to Fort Lauderdale was to back up to tape over the WAN. The backup took 30 hours.

"A good lawyer can bill 30 hours in a single day, but no IT department can do a 30-hour backup in a day," Weinberger says. "That solution just sucked for lack of a better word. Attorneys scream and shout if we go offline for a minute, so you can imagine what it was like."

Weinberger knew his main task at his new job was to make the 200-attorney firm with 10 offices throughout Florida hurricane-proof. In his previous job at the City of Los Angeles Attorney's office, he dealt with a few small earthquakes but no hurricanes.There wasn't much left in the budget after his predecessor bought Clariion CX400 and Clariion CX500 SANs from Dell. Fortunately, the SANs had total capacity of 5 TBytes and only 2 TBytes were being used, so he had some overhead to play with. His priorities were to centralize and improve backup in Fort Lauderdale, then set up a secondary site well out of hurricane range. The goal was to make sure the company's employees retained access to its most critical applications: Exchange and an Interwoven document management system.

"Dell came back with unworkable and unaffordable solutions that required buying a lot of hardware and software from them," Weinberger says. "It would have cost around $250,000, and that was not in my budget."

He did buy around $10,000 worth of direct attached storage from Dell for disk backup, then looked at software additions. First, he added Symantec's LiveState Recovery that performs bare metal system recovery by creating system images on virtual machines that can be converted to real servers in case of a failure.(See Symantec Unveils LiveState Manager.) Weinberger says LiveState Recovery lets him recover from system failures in less than an hour.

"We still use tape as a comfort level, but we don't need them," he says. "We send them to Miami and put them in a fireproof safe."

Next, he tackled remote replication. Ruden McClosky purchased EMC RepliStor with its Clariion SANs, and used it to replicate between Florida offices. But Weinberger wasn't happy with the interface or performance of RepliStor, so he looked at more sophisticated replication applications: NSI Double-Take and XOsoft WANSync. (See NSI Does Double-Take and XOSoft Readies Replication.)"Everyone says Double-Take is the big player out there but my guys were impressed when they saw how easy WANSync was," he says. "Double-Take seemed more complex with failing back, and what good is failing over if you can't fail back to the primary server?"

They were also impressed with WANSync's assured recovery feature that allows them to test recovery of replicated data without having to take down the servers. (See XOsoft Assures Recovery .)

NSI Double-Take cost less, but XOsoft gave the firm credit for its previous investment in RepliStor. Weinberger went with XOSoft, but had to pick his spots with WANSyncHA (high availability), which allows automatic or one-button replication while traditional WANSync requires manual reconfiguration of clients to failover.

HA costs $5,000 extra per server, which adds $10,000 to the cost of each pair of replicated servers. "That's not something I was willing to consider for each server," Weinberger says, so he went with HA for his Exchange servers and two pairs of Interwoven servers, and used the standard WANSync for all other applications.

He also set up a hot site in Chicago, for two non-technical reasons. "I like Chicago and it gave me a chance to visit," he says. "And if Chicago and Fort Lauderdale both have a hurricane we'll have a bigger problem than where to store Ruden's data.Weinberger says it cost less than $100,000 for the direct attached storage, LiveState, and XOsoft software. But the process took him longer than he would have liked. "In theory, it sounds great," he says. "In practice, we were late in the game because it took us awhile to come up with the solution. By time all was said and done, we were into hurricane season by time we implemented and tested it."

The real test came last September when Hurricane Wilma hit Florida. The Category 3 hurricane knocked out power in Ruden McClosky's Fort Lauderdale data center for three days, and the office was closed for five days. So were the firm's offices in Miami, West Palm Beach, Naples, and Port St. Lucie.

"We saw the storm coming," Weinberger says, and he made the call Sunday night when Wilma was still off the coast of Florida to fail over. It took Weinberger's staff less than an hour to set up the Exchange and the Interwoven servers with one click. "It makes you feel better knowing that you spent $10,000 extra for those server pairs and they worked. We flipped all our servers over to Chicago. The next weekend when Fort Lauderdale came back online, we failed right back to the data center."

The firm had one glitch during Wilma. Because Blackberry Enterprise Server allows use of one license key at a time, the servers must be brought up and down in a specific order. When Weinberger's staff brought up the second server, it knocked out the first and the firm's Blackberries were out of service for four hours.

"We could have prevented that if we had more time to test," Weinberger says.Dave Raffo, Senior Editor, Byte and Switch

Organizations mentioned in this article: