Northeast Health

If the central data center crashed for Northeast Health, a not-for-profit healthcare conglomerate in upstate New York, the applications for its 35 hospitals, nursing homes, and other facilities would run as usual. A tad slower, perhaps, but they'd still be available to maintain the level of patient care.

That wasn't the case two years ago, when Chris Baldwin, VP of corporate MIS, and his team decided to create a disaster recovery plan that was better than trucking tapes to a warehouse. "We had reached critical mass," he recalls. "We're not just a hospital, we're an integrated healthcare delivery system... We had started to use advanced clinical applications... The more we got into those, the more compelling it became to ensure we had a solid business continuity and disaster recovery plan."

One step in the process had already been taken. A few years before, data services had been consolidated at a primary center in Troy, N.Y., and a secondary site set up at a hospital seven miles away in Albany. A bunch of servers and disk arrays from Data General were installed. A private microwave link, chosen in the years before fiber was affordable, was set up to connect the two sites.

Despite the progress, there was a glitch. Each night, technicians backed up about 4 Tbytes of data from the servers onto directly attached tape drives and shipped the tapes offsite. As Northeast added applications with physician and nurse bedside data entry and PACS (picture archiving and communications system) medical imaging, it became clear that tape-lugging wouldn't cut it if one of the data centers crashed.

There had to be a way to recreate the site whole in the event of an outage.A first step was the addition of a Fibre Channel SAN at both sites about two years ago. A 32-port switch from McData Corp. (Nasdaq: MCDTA) in Troy is linked to a Clariion CX700 array from EMC Corp. (NYSE: EMC), which in turn fed 12 servers from Dell Inc. (Nasdaq: DELL). Over in Albany, 6 servers were linked to storage from a Clariion CX500, in turn connected to a 24-port McData switch. VPN services based on a 10-Mbit/s fractional DS3 links were enlisted from Time Warner Telecom Inc. (Nasdaq: TWTC) to connect the data centers to other remote sites.

Dell, the integrator on this job, already had contacts with Meditech Inc., a healthcare software supplier whose products were the backbone of Northeast's applications. Meditech in turn had a partnership with BridgeHead Software Ltd., a storage software company with offices in the U.K. and Massachusetts. The result, which enabled BridgeHead's backup software to work with the Meditech apps across the microwave link, was installed about a year ago.

Each day, the Meditech software, called Integrated Serverless Backup and Integrated Disaster Recovery (ISB and IDR), take a full backup of the Troy site, and using a compression device from FatPipe Networks Inc., transmit it to Albany. There, the data is set up on the standby servers and SAN, ready for use, just in case. Nightly tapes are still made in Albany as well.

"What we have is a functioning standby," Baldwin says. "Having 50 percent of the server capacity in Albany means apps would run in slightly degraded mode, but they would be fully available."

On the downside, Northeast has multiple vendors with integrated wares. That said, they have sufficient knowledge of Dell, Meditech, and BridgeHead to pinpoint the source of problems on their own and work with the vendors successfully. Not everyone can claim the same rapport.While Baldwin isn't giving out figures, the cost of implementing the standby SAN and servers wound up being 70 percent of what it originally cost Northeast to put in the Dell hardware, Data General disk arrays, and tape storage seven years ago. Not a bad bargain.

Mary Jander, Site Editor, Byte and Switch