Bank Dumps Quantum Gear

Bank of America's investment banking group has ditched a services contract with its long-time tape library supplier Quantum Corp. (NYSE: DSS) for some of its data center locations -- not because the price wasn't right, but because the service was severely subpar, according to Thomas Mixer, a manager with the banking group's data storage and recovery team.

And the bank is also powering down some of its older Quantum equipment and bringing in tapes and libraries from Storage Technology Corp. (StorageTek) (NYSE: STK) to replace them.

We were running into a lot of grief with Quantum,” Mixer says.

He says that when Bank of America had a Quantum ATL P1000 library go down in its disaster recovery site in Columbia, S.C., it took a week of back-and-forth to get Quantum out there to fix the library. "And I wasn’t trying to negotiate with them on price. I just said, ‘I need someone out there to fix that library as soon as possible,’ and it still took a week for them to get there.”

Byte and Switch gave Quantum the chance to respond to Bank of America's allegations before the original version of this story was published. The company sent us this statement: "In June 2001, Bank of America contracted Quantum to design and implement a SAN solution that included the sale of tape automation libraries. Although not a strategic supplier to them at that time, Quantum was selected because of its expertise in Fibre Channel connectivity. At this time, Quantum maintains a service and support relationship with the company for the tape libraries."After the article appeared, Michael Zeglin, director of public relations for Quantum's Storage Solutions Group, contacted us with some additional comments. Zeglin says that the service call in question was for a library no longer under contract with Quantum. He also claims that the delay was on the part of Bank of America's staff in giving Quantum the approval to perform the work. "For any libraries not under service contract, a purchase order is requested from the customer before service is provided," he wrote in an email to Byte and Switch. "This is industry standard."

Mixer, however, claims that this wasn't the only time Quantum dragged its feet before getting a job done. The tape vendor’s sluggish, uncooperative attitude is more the rule than the exception, he says. Mixer says he has had to hound Quantum to fix two other P1000 libraries as well: one in Atlanta and one in St. Louis. In both cases, he says, it took the company days to respond. [Zeglin, responding after the original article was published, claims that Quantum has no recent record of any service call from Bank of America in St. Louis.]

“If I have a piece of equipment that’s still under contract with them, I’ll talk to them,” Mixer says. “But after that, if I don’t have to call them again, I never will.”

And that’s not the end of the bad news for Quantum, which has seen the momentum for its DLT tape products dwindle in the face of competitors’ LTO and AIT tape formats, as well as an intensifying onslaught of ever newer and cheaper disk products (see Quantum Tape Drives Speak Up and Quantum Tapes Up Roadmap). The investment banking side of Bank of America -- which makes up only about 2 percent of the bank's overall headcount but brings in roughly 40 percent of its revenues -- appears to be following the lead of the larger commercial banking division in its tape purchasing decisions. “The bank’s pretty much an 800-pound gorilla when it comes to vendor decisions,” Mixer says. [Responding after the original version of this article was posted, Mary Waller, media relations manager at Bank of America, says that Mixer "does not make vendor relationship decisions."]

But let’s start at the beginning. Bank of America's investment banking group is in the midst of a huge renovation of its mainly direct-attached storage (DAS) environments. At the data center where Mixer is located in Charlotte, N.C., there are about 2,000 DAS servers and about 150 Network Appliance Inc. (Nasdaq: NTAP) filers. And that’s just one of seven data centers run by the investment bank in the U.S. The company has about 7,500 servers in total."We started doing shared storage in the backup space, and we started running into backup-window problems," Mixer says. "We back up around 50 Tbytes of data a week in daily incrementals and weekend backups, and our storage is growing between 5 and 6 percent a month... You can’t push that much data through the system in so few hours, so we needed to upgrade our technology."

The investment bank was using a number of the Quantum P1000 tape libraries and three large P3000 libraries, but that wasn’t enough to get the job done. "We just couldn’t push enough data through them fast enough, especially on the weekend."

In addition, Mixer says, there was the cost of using older DLT 40/80 tape drive technology. “We were spending about a quarter of a million dollars a month just on DLT tapes,” he says, pointing out that the warranty on much of the Quantum technology had expired, forcing the bank to pay annual maintenance fees. “So we were trying to reduce the cost of the backup, and reduce the huge consumption of tapes.”

By switching to StorageTek L700 libraries instead, Mixer says, the investment bank hopes to save more than $1 million a year on media costs.

In his email to Byte and Switch, Zeglin insists that the performance problems the bank was experiencing were due to the fact it was using older models of the Quantum products. "[This is] not an issue of service or lack of proper product performance," he says. "These products are performing at their level of capacity, which was state-of-the-art when purchased two to three years ago."And though it was too late to catch Bank of America's eye, this week Quantum launched the "MAKO" PX720 tape library, which is designed to provide higher densities than the vendor's previous libraries. The PX720 can house up to 20 DLT or LTO drives and up to 732 slots in a single frame (see Quantum Launches High-End Library).

In any case, the investment bank has already shut off most of its Quantum equipment, except for one P1000 library, Mixer says. “We need to have some sort of DLT system online for legacy restores. We have to do that for compliance reasons until those tapes expire. We’re going to need to have one of these running for many years to come.”

And while Quantum's role at Bank of America appears to have fallen into Rick Moranis’s kid-shrinking machine, StorageTek seems to have been hit by the reverse blast of the apparatus, with potential contracts of gigantic proportions.

The investment side of the bank already has two StorageTek L700 libraries, each costing upward of $1 million, up and running in Chicago and Charlotte, Mixer says. [Bank of America's Waller says the "upward of $1 million" figure is "inaccurate," but she would not say how much the company is paying for the StorageTek libraries.] Another one, in New York, is not yet in production, he says. [Waller says the bank hasn't actually purchased the library for New York yet.]

StorageTek is not only providing the investment institution with libraries and tapes, but has also been pushing its information lifecycle management message hard (see StorageTek Looks to Bag Buzzword). It has been helping Bank of America identify large sets of data that don’t need to be backed up, to help lighten the data load being pushed through the company’s backup window.“For instance, large numbers of our users have large personal files stored on our network,” Mixer says. “We don’t need to back them up. We’re already storing those files permanently somewhere else for compliance reasons.”

Mixer's group is using software from Zantaz Inc. to archive and monitor all email correspondence in order to comply with securities regulations. As such, the investment bank has set its Veritas Software Corp. (Nasdaq: VRTS) NetBackup software to skip individual users' personal mail files completely for daily and weekly backups. The company will still back up all the files once a month, though, Mixer says.

As Bank of America moves forward with the cleanup of its storage environment, it also plans to purchase StorageTek disk array systems, according to Mixer. [Waller contends that Bank of America has "made no decisions to purchase StorageTek disk array systems."] In addition, the company is looking at buying hierarchical storage management software providing space allocation and chargeback options, and is currently looking at different vendors.

“Everything’s supposed to be running fine by next summer. We’re being pretty aggressive... We’re trying to finish this off as quickly as possible.”

— Eugénie Larson, Senior Editor, Byte and Switch