Market Street Mortgage

While hurricanes over the past few years convinced many companies in harm's way to implement disaster recovery plans, not everybody was caught off guard.

Clearwater, Fla.-based Market Street Mortgage, a residential loan company, has spent more than $100,000 over the past three years preparing for disasters. The DR work paid off when Market Street survived three power outages in the last two years without downtime to any of its 1,000 workers spread across 80 branch offices in every state except Alaska.

"Our branch offices are our sales offices, and they produce revenue," CIO Charlie Pelton says. "If you're sitting in Oklahoma or Colorado, our customers aren't too concerned if a hurricane's coming through Florida, so we can't use that as an excuse if we don't have access to our data."

Pelton notes that his company handled about $3.4 billion in mortgage loans last year. "Each hour of downtime costs us about $1.6 million in lost revenue. It adds up pretty quick," he says, with no small amount of understatement.

Market Street's first stab at DR consisted of tape, and it added Dell/EMC Clariion CX500 SANs in its headquarters and a DR site in central Florida in 2002. The firm now has about 4 TBytes of data on SANs, a Dell PowerVault tape library, and direct attached storage for Exchange, SQL Server and its loan origination application."Disaster recovery has always been important to us," Pelton says. "The technology we have in place today enhances that. Five years ago we had traditional tape backup, but with the amount of data we have stored today, it takes 24 to 48 hours to get it restored. And that's if you don't have any problems. If the tape breaks or you have a bad drive, all sorts of things can go wrong.

"We like the real-time aspect of restoring data. You still need tape, but you hope you don't have to use it."

To move the data between sites, Market Street replicates Exchange, SQL, and its loan program with Neverfail's high-availability software for Exchange, SQL, and File Server.

Pelton says he was unhappy with the replication software from one of the major storage software vendors before switching to Neverfail. He doesn't want to name the vendor, which he still uses for other applications.

"We ran into complications," he says. "They had huge installations in the Unix-based world. We ended up being one of their larger Windows-based customers, and it didn't perform as well as we wanted."He also looked at replication packages from EMC, NSI Software (now Double-Take Software), Veritas (now part of Symantec), and XOSoft before settling on Neverfail in 2004. Pelton says he went with Neverfail because it is built specifically for Windows yet runs independently of the Windows Server 2003 operating system.

"It's a straightforward product, while the others tend to become intertwined with the operating system," he says. "Neverfail is pretty much standalone. Before, I had to install cluster replication software and the operating system got installed under it. Microsoft says 'We don't support that, you need to talk to the replication vendor' and the replication vendor says 'We feel it's a Microsoft issue.' Support became very difficult and time consuming.

"With Neverfail, if we have an issue, we can uninstall it and it has no impact on anything else. Then we can install it again."

He also likes that "the user interface is clean. It doesn't take a PhD to fail over."

Pelton says Market Street runs annual DR tests, in which it redirects users to the DR site. More importantly, the firm has had three actual redirects in the last two years when electrical power failed at corporate headquarters."We perform an annual test of our DR site, including business continuity plans," he says. "We get all users involved to make sure they have the proper connectivity. I can failover and my users don't even know."

Within months after Market Street installed Neverfail software, its headquarters got hit by hurricanes Charley and Francis weeks apart. Francis knocked out the building's power for 72 hours, but the network stayed up and running.

"While power was going down, we failed over," Pelton says. "We basically have the ability to fail over as long as somebody in our company has a connection somewhere."

Dave Raffo, News Editor, Byte and Switch