SunGard Consolidates Disaster-Recovery Centers

From the outside, SunGard Availability Services' New York metro area disaster-recovery center is unassuming, except for its multiple-football-field size. Just two stories high, the facility's windows make it seem more like a factory or warehouse. But the one-way mirror behind the security desk is the first hint that this is no ordinary building. The security command center resides behind that glass, with enough monitors to view what's going on both inside and out. Once inside, every door seems to require identity access. Even Len Guddemi, SunGard's VP of northeast operations, couldn't get through some. Customers are king here, and if they're in one of the customer-command rooms, nobody else gets in.

Many of those rooms were taken on a recent uneventful October Tuesday, a sure sign that SunGard and its megacenter have become more than a disaster-recovery facility. The vendor isn't giving up its disaster-recovery business, but it has a new focus as well: information availability.

Jim Simmons, SunGard Availability Services CEO, considers information availability much broader than just disaster recovery. "Information availability is a business process, a way of thinking, as compared to disaster recovery, which is a reaction," he says. These days, Simmons meets with C-level executives who want their companies to remain up and running, disaster or not. "It's about living healthy, versus rushing to the emergency room after a heart attack," Simmons says.

Nick Voutsakis, chief technology officer at Glenmede Trust, an investment-management firm and SunGard customer for nine years, recalls how SunGard changed its strategy from cold sites, where equipment remains offline until it's needed, to hot sites, where a broad mix of SunGard's infrastructure is live with mission-critical applications that multiple companies can access. "Availability isn't just in case of a natural disaster, but in case of disasters that happen throughout the year, like a server going down," says Voutsakis, who hopes to leverage a SunGard hot site for dual data-center activity for trading systems, performance management, and trust accounting. "Day to day, multiple things could go down," Voutsakis says. "Another benefit [of information availability] is not having to shell out money for a disaster that never happens."

Customer demand, as well as brute reality, is behind the move from disaster recovery to information availability, David Tapper, an IDC analyst says. Even with wildfires and blackouts, disaster recovery isn't exactly a hot market, he says. "On-demand computing is the next big market, and disaster recovery should be a subset of that," Tapper says. SunGard owns about $800 million of the $2.5 billion disaster-recovery market, with IBM taking in about $450 million, according to IDC.While information-recovery awareness is high among companies, Tapper says IT executives don't lay out much money for the necessary infrastructure such as servers, storage, and software. The largest companies spend around 5% of their total IT budgets on information recovery, but most businesses only spend half a percent, he says.

One loyal SunGard customer is poised to renegotiate the terms of its disaster-recovery service. Schneider National Inc., a logistics and trucking company, is adding servers, storage, and software to its dual data center. For more than five years, Schneider has used shared space at one of SunGard's 35 megacenters for multiple disaster-recovery tests each year. "But we've never found it necessary to declare a disaster," says Paul Miller, VP of technology services at Schneider. The trucking company already distinguishes disaster recovery from business continuity, which it considers a business process that echoes throughout the organization. A business-impact analysis that Schneider conducted last year, including the history of class 4 and 5 tornados in the Green Bay, Wis., area, provided justification for a balanced load between dual data centers. "Our SunGard relationship will continue as we implement the second center," Miller says. "Our contract runs beyond next year, but it's uncertain what our ongoing SunGard relationship will be like."

SunGard already provides multiple levels of service to customers. Most use SunGard sites for recovery in times of trouble, be it fires in Southern California, a hurricane pummeling the East Coast, or a blackout knocking out much of the country. At the New York-area megacenter, dual 12,000-gallon diesel generators left a smell in the air but sat quietly. During August's blackout, Guddemi says, they kept the center running for 96 hours.

Many customers get a secure room in one of SunGard's centers for an application or two, and a handful keep their whole back ends at a center. SunGard gives everyone a big fat pipe, all the infrastructure they can handle, and nonstop technical support from SunGard engineers at every center. "A customer's own data center costs a lot more, they have a hard time delivering two, three, or four hours of recovery time, and that leads to more lost revenue," Guddemi says.

SunGard is consolidating five disaster-recovery centers into one at its New York metro area megacenter, which already has more than 3,000 square feet in support of Unix and Windows infrastructures and another 23,000 square feet ready on the second floor for mainframe infrastructure soon to move in.A brand-new information-availability center is slated to open soon. It's within driving distance from the megacenter, which went live in 2001 and has plenty of room for growth. But the new center, with 172,000 square feet, will focus primarily on information availability. SunGard experts will operate and manage customer equipment for the cost of two full-time people, performing tasks such as database and storage administration. One service SunGard provides along with Dell lets small and midsize business customers access a pre-configured server in times of trouble for $300 per year.

In case of a major disaster that affects many customers, SunGard relies on its crisis-management center near company headquarters outside Philadelphia. A voice-recognition system tracks customer calls as they come in. It operates on a first-come, first-served basis, but nobody is turned away: SunGard can bring all 35 of its megacenters to work as one if needed. Whether it's a disaster or everyday nonstop availability, SunGard offers customers the option to keep information always on for less money and pain than they could do on their own.

"The biggest cost to customers is the network pipe," Guddemi says. "Instead, they can tie into our network from anyplace in the country."

A former Comdisco Inc. customer was so happy with SunGard after the two disaster-recovery companies merged that it extended its contract for three more years. "Two and a half years ago, we evaluated our recovery needs and decided on four hours of tolerance," says Mitchell Hodus, VP of technology operations at Evergreen Investments. "But we were lucky to recover information in 30 hours from the tapes we used at the time." Evergreen shifted gears dramatically and began mirroring its infrastructure in a SunGard facility. SunGard provides the generators, telephony, and extra space for desktops that Hodus doesn't want to have to deal with in case of a disaster. SunGard Data Systems Inc., SunGard Availability's parent company and a provider of managed services to the financial-services industry, isn't immune to the economic doldrums. Though revenue in the third quarter ended Sept. 30 grew 18% to $92 million, and net income was up by about 15%, the vendor admitted it suffered from flat IT spending and only grew as a result of acquisitions.

SunGard Data Systems claims 20,000 customers worldwide, many from its Comdisco acquisition, and counts 47 out of the world's 50 largest financial-services companies as customers. The company realizes it must ride with customers that want to diminish investments in pure backup facilities and implement processes around business continuity. SunGard Data Systems processes 3 million trades daily on its systems and maintains 75 data centers, 50 mobile data centers, and more than 15,000 end-user recovery positions in Europe and North America.Some former Comdisco customers are crying foul when renewing service contracts. They say SunGard is forcing new services on them to ratchet up prices. An industry expert explains the disconnect between SunGard and some customers. "Some old Comdisco deals go back to the dark ages," says Carl Greiner, VP at Meta Group. "Some of its contracts are older than heck and have nothing to do with today's prices or levels of service." Windows, a large part of most infrastructures, is absent from those old contracts, many of which are eight years old. Even if a customer only wants mainframe support, that platform's cost has typically grown 20% each year, Greiner says. Every customer has different needs, he says, but few devote the whole infrastructure to the information-availability process, and they shouldn't. Companies that need it the most will mirror their data centers, and these companies could still use SunGard for remote offices, secondary servers, and less-than-critical apps. "Some people aren't allowing for today's environment," Greiner says. "Contracts just vary on what customers really need."

SunGard CEO Simmons claims he's unaware of any brute-force negotiating tactics. "Some customers have been with us for a couple of decades," he says, "and in three years some will migrate to three different platforms."

One customer started working with SunGard two years ago, choosing it from among four players for its national reach, stability, and ease in cutting a deal. Alan Leib, president of Stats Inc., a 20-year-old provider of sports statistics to major broadcasters and individual sports enthusiasts, started with SunGard and its disaster-recovery service, but got on board with information availability when the National Football League season began over the Labor Day weekend.

Leib was prepared for the increased interest in fantasy football, but he never dreamed he'd face double the demand and need double the bandwidth. "We held up the first weekend, but it pushed us beyond our comfort zone," Leib says. "We made the first call to SunGard Tuesday and were up and running by Friday of that week." Since then, SunGard has been Stats' Web-hosting firm.

Leib credits the mutual knowledge that Stats and SunGard share with each other as the reason for such timely implementation. "SunGard engineers do the work, but we manage it all remotely," he says.Whatever the service SunGard provides, the company appears to take its responsibility to customers very seriously, as illustrated by an exchange in its New York megacenter during our recent visit. VP Guddemi pointed to a customer-command room and said it was one customer's entire infrastructure. When asked about the man working in the room, Guddemi indicated that he was from the customer company. A few seconds later, catching himself, he said, "But you can't talk to him." It's that kind of dedication to customer privacy and security that SunGard hopes will keep its customers happy--disaster or not.