IT Managers Leverage Technology When Planning For Disaster

Good IT managers are watching their pennies when it comes to disaster recovery and looking for ways to better leverage technology to ensure fail-safe systems.

March 22, 2004

7 Min Read
Network Computing logo

When it comes to disaster-recovery systems, investment firms can't afford to be cheapskates, given what's at stake. However, in a cost-conscious environment, it doesn't mean they have to be spendthrifts, either, and squander their firm's resources. That's why good IT managers are watching their pennies when it comes to disaster recovery and looking for ways to better leverage technology to ensure fail-safe systems.

Wealth-management firms don't have unlimited budgets, when it comes to planning for disasters, says Nicholas Voutsakis, chief technology officer at Glenmede Trust Company in Philadelphia. "The challenge is to be able to manage it on a tight budget."

Voutsakis' disaster-recovery plan, which was beefed up after the events of 9/11, were put to the test when his building lost its power after the July 4th weekend last year.

Employees who arrived at 7:30 a.m. were cast in darkness thanks to a power blackout at the investment-management firm, which has more than $13 billion in assets under management.

Glenmede's systems went into shutdown mode and its staff had to evacuate the 61-story building, grabbing their red disaster-recovery bags as they left. The bags included things like flashlights, maps, phone numbers and copies of the firm's disaster-recovery plan.The employees met at two pre-arranged sites, armed with cell phones from two different carriers in case one of the telecommunications networks crashed. They also had walkie-talkies.

Company executives declared an emergency and Glenmede's systems, which were backed up using SunGard Availability Services, were restored at Glenmede's offices in four cities. The Philadelphia employees made their way to a SunGard hot site in the city, where 30 workstations were being prepped, and Voutsakis arranged to have phone lines shifted to the hot site. That combined with the other offices gave it 80 workstations, about one-third the normal amount. By 10 a.m., clients were being contacted, and within an hour, the firm was up and running and making trades for clients.

Voutsakis says that events like Y2K and 9/11 woke the firm up to the need to build better business-continuity plans. Voutsakis says Glenmede thought about using its existing offices as backup, but "we didn't feel comfortable." Instead, the firm decided to outsource that task and use SunGard's hot site as the backup site and link to the other offices through it.

There's evidence that suggests outsourcing continuity services, rather than building their own internal-backup sites, can help firms with their total cost of ownership and return on investment. A 2003 study by research firm IDC found firms that outsource their business-continuity needs actually lower their revenue loss, improve the utilization of IT staff and reduce their expenditures.

The report by IDC Analyst David Tapper found firms that don't own their internal back-up structure:

  • Had an average revenue loss of only $1.1 million compared to $4.2 million for financial institutions that do.

  • Reduced their business-continuity expenditures by more than 80 percent.

  • Lowered their IT-capital expenditures by more than 20 percent.

  • Were able to improve the use of IT staff and the network to support employees by almost 37 percent.

    Dave Palermo, a vice president at SunGard Availability Services, says that firms looking to do internal backup themselves find it "hard to keep up with extra capital investment." He says firms need deep pockets to build a mirror image of their data center. Voutsakis says another cost-conscious move he made was to rent rack space from SunGard for Glenmede's servers. "We're able to have the flexibility of having our own hardware at the SunGard location, which is five minutes away. We're able to go in there and gain easy access."

    Gary Foster, chief technology officer at Omgeo, a trade-solutions management firm in Boston, says that when it comes to managing disaster-recovery costs, it's a function of distance. "The real cost issue is how much distance you're willing to pay to manage your company." One city and one building, he notes, means "more risk."He says that the farther the backup site is from the primary site, the more expensive it is to transfer the data in real time. Omgeo has a high-availability environment with a blend of hot sites and cold sites, referring to the sites level of importance.

    One way to manage costs is deciding which applications or systems a firm wants to run hot, warm or cold, says Voutsakis. "A trading system is hotter than something like a CRM system. We have over 100 software applications residing on the network. Data changes are being copied back and forth for those systems that are hotter." For others, they're copied at night.

    Foster notes that setting up a second site that replicates the data center and the connectivity and redundancy needed to provide for backup can cost thousands, making it a "real challenge for mid-sized brokers."

    Steve Higgins, director of business continuity solutions at EMC in Hopkinton, Mass., says that the "economics we've had the last few years has made people cost conscious." One of the things he sees when it comes to disaster recovery is a "migration from tape to disk."

    That's because recovery from tape can be an arduous process that takes precious time. People have to package tapes and send them out and then recall them if they're needed. "The ability to recover and restart your business (from tape) has extended well beyond what people hoped for." As well, he says, there's no guarantee the tape will be able to be read. "Tapes fail over time," he says.Despite those concerns, Higgins says, management is "clearly not giving IT an open checkbook. A lot of IT individuals are saying, 'how can I improve where I was and, at the same time, do that with less money?'" By moving straight to disk, he says, firms are lowering their costs.

    The more critical the information or application, the more a firm spends to protect it. So a firm that has a legal obligation to close out trades and report to clients and authorities would run that reporting system on a synchronous-recovery solution, while a corporate-purchasing program that simply buys pencils and papers runs in an asynchronous mode. Firms need to "balance cost versus compliance."

    He says firms can't avoid having more than one site available to them, but there are ways to reduce hardware costs, especially if a firm opts to handle its own secondary site.

    EMC recently purchased VMWare, Inc. a Palo Alto, Calif., software company that specializes in Intel-based virtual-computing software. It allows firms to use Intel-based servers and run multiple operating systems across them. As a result, financial-services firms can run a Microsoft and a UNIX application on the same server. Typically, he says, firms group servers to run applications across one business line. They will now be able to run applications across many business lines, cutting their server ratio in half. "It has huge implications for costs." He adds that it will help reduce spending and simplify operations.

    For firms that want that little extra protection, they might want to mix refurbished equipment into the equation to reduce costs. Gulfcoast Workstation Corp., a Florida-based subsidiary of Relational Funding Corporation, reports that it helped a large integrated financial institution build a ghost site that parallels the firm's main site. In the event of a denial of service attack, blackout or similar disaster, the ghost site picks up the applications in real time. The site runs on refurbished gear, including an IBM AS/400 that cost $36,000 compared to $100,000 brand new. Gulfcoast estimates that it reduced the firm's costs by 10 to 12 percent over a 36-month lease of the equipment.As for Voutsakis, he's looking to further improve efficiencies and invest in his firm's connectivity to SunGard. He plans to scrap Glenmede's T1 connection and move to a DS connection that will provide additional bandwidth and 35 times faster movement of information. "What this will allow us to do is create a more robust list of software applications available on a more timely basis for our end users in the event of a disaster."

Stay informed! Sign up to get expert advice and insight delivered direct to your inbox

You May Also Like

More Insights