BrokerageAmerica's Blackout Begets Backup

The fire knocked out BrokerageAmerica's electricity and voice and data lines, including T1s, DS-3s and a 100-Mbps Ethernet LAN. The firm didn't lose any data--all its data had been backed up to storage provider ManagedStorage. But BrokerageAmerica had been in the midst of moving its Unix-based trading system and Nasdaq servers from the old site to Internap, and this threw a wrench into the process of getting its remote sites connected to the new data center location.

The core of BrokerageAmerica's redundancy and backup strategy was relocating the data center to a 9-by-24-foot cage at Internap, as well as setting up redundant lines between Internap and the firm's Boston, San Francisco and Red Bank offices. After the fire on Saturday, the company set up a makeshift trading floor for its New York traders in one of Internap's conference rooms. The traders were up and running Monday afternoon around 3:30, just before the market closed. BrokerageAmerica, which supports the broker-dealer side of the financial industry, operated there for a few days, then set up a longer-term temporary trading floor at the data center of RoyalBlue, its ASP.

Almost Like HomeUsing the interim facilities, traders conducted their transactions just as they did back home. They logged onto the firm's RoyalBlue Fidessa front-end application servers through a RoyalBlue thin client on their desktops. Once they set up a stock purchase or trade, the application routed the order through a dedicated DS-3 to RoyalBlue's order-management back-end servers at RoyalBlue's site, which also resides at Internap. RoyalBlue handled the actual order and sent it on to Nasdaq's servers over another dedicated DS-3. BrokerageAmerica has now moved out of the interim facilities, but it continues to operate its trading system out of Internap. The firm's Windows 2000-based e-mail, back-office and other financial services application servers also run in the Internap cage.

BrokerageAmerica's IT manager, Simon Janason, remotely manages and maintains BrokerageAmerica's servers and networking devices at the Internap site from the firm's headquarters in Manhattan. "I connect to the collo and our other sites as if I'm sitting there at the desktop, for example," Janason says. He uses mainly SolarWinds Software's SolarWinds tool for network monitoring and the freebie Virtual Network Computing tool for remotely managing servers and desktops.

The firm's old data center, meanwhile, was an IT pro's worst nightmare: worn-out wiring and archaic, shared T1 multiplexers sitting in the bowels of a building that had been plagued by outages. When a telco technician bumped into one of the muxes in 2003, he inadvertently knocked out the building's T1s, including BrokerageAmerica's WAN. "It was indicative of the problem with that building," Martinez recalls. "There was no backup power.

"Although BrokerageAmerica immediately began moving its critical network devices and servers to the collocation facility, the move was slow, hampered by budget constraints and limited IT staff (see "The Hard Sell,").

Expansion PlansThe next step for BrokerageAmerica will be to enable the remote offices in Boston, San Francisco and Red Bank to run independently in the event of a future disaster. If the firm's network or the Internap site goes down, the remote sites can access trading and back-office apps from a backup data center. These sites house mostly sales operations, though Red Bank is adding some trading and accounting operations as well. The branch sites rely on the Internap data center for their critical applications--Red Bank and Boston connect to it over point-to-point T1s and a VPN connection; the small San Francisco office uses a VPN connection.

The Red Bank backup site requires Brokerage-America to buy mirrored servers, Cisco switches and routers, desktop machines and point-to-point circuits, says Janason. BrokerageAmerica estimates that such a site would cost $2 million.The outages also delivered a wake-up call about the firm's vulnerable PBX voice system. The Lucent PBX failed in both the fire and the T1 outage, leaving the company without any voice service. The IT team is now looking into voice over IP. Martinez favors a more portable voice solution that doesn't depend on a particular site or hardwired line in the event of a disaster. "If we had VoIP, we could've had voice back up and running elsewhere" immediately after the fire, Martinez says.

Anthony Martinez: 41, senior vice president and head of BrokerageAmerica's trading operations

Martinez spent 3 1/2 years overseeing all IT operations for BrokerageAmerica (he has since left the firm).

Most bizarre IT inquiry: After the fire, we had no electricity or data lines, and the CEO asked why his e-mail didn't work. This was before everything was moved off-site.

Biggest annoyance with collocation: We usually depend on remote access. I can't just have someone walk down the hall to check something out.How life imitates IT: My mother's computer always has something wrong with it, and she can never get e-mail. Just like users at work.

Best geek joke: A trader complains to IT that he has a system problem. Says the IT technician to the trader: "The problem is between the keyboard and the seat."

If you were stranded on an island, would you rather be stuck with Bill Gates, Larry Ellison or Steve Jobs? Steve Jobs. At least he has some exposure to hardware and can build things, rather than just software.

Favorite stock: Motorola--it's having tough times, but always ends up coming back.

Biggest industry flop: Winstar Communications. After Lucent cratered and called in all financing, that set back the entire industry.Wheels: Yukon XL on weekends. It's big for a lot of cargo and stuff for a boat. During the week, an old Mercedes diesel with 183,000 miles on it.

After hours: Boating, fishing, skiing

Favorite team: New York Yankees

Dream job: My same job, but in a better working environment--on my 34-foot Silverton boat, with remote access.

Fave dig: At Comdex, when Bill Gates said if General Motors had kept up with technology like the computer industry has, we would all be driving $25 cars that get 1,000 miles to the gallon.Fave comeback: Part of GM's response to Gates: If GM had developed technology like Microsoft, your car would crash for no reason twice a day; the airbag would ask, "Are you sure?" before deploying; and every time a new car is introduced, drivers would have to learn to drive all over again.

It started nearly two years ago, when a telco technician brushed a faulty T1 multiplexer, jarred loose the mux card and took down BrokerageAmerica's WAN. The incident cost BrokerageAmerica more than $100,000 in lost trades over the two-day outage. The firm's executives then signed off on a long-debated plan to move BrokerageAmerica's data center to Internap.

"A disaster changes your mind-set. Unfortunately, that's what sold management" on the collocated data center, says Anthony Martinez, senior vice president and head of BrokerageAmerica's trading operations (he has since left the firm).

The collocation deal with Internap helped launch the first phase of a disaster-recovery strategy, and the electrical fire expedited the company's changeover to the new data center. The next logical step is development of a full-blown disaster-recovery site--the Red Bank, N.J., location will be retrofitted for that purpose.

With the firm's notorious disaster streak, it should be easy to sell management on IT projects like this, right? Not so, Martinez says. It's tough to prove ROI to management, especially for disaster recovery, he says. You can't quantify the bad press that comes with a financial services firm's outage: "The negative PR we got when we were down for six hours was a big deal," he says.The firm's CEO and president routinely grill Martinez and his IT team on how to save money on proposed projects--by purchasing used or lower-end equipment, for example. Martinez and his staff then get into the nitty-gritty of why a high-end Cisco router is a better fit than a low-end one. "We explain how much it will cost if we get this equipment versus if we don't and we go down," says Simon Janason, IT manager for BrokerageAmerica.

Martinez describes it this way: "We need to process 30,000 trades within milliseconds before the close of the market, and our high-volume operations require the right equipment."

But as in other organizations, IT money remains tight. BrokerageAmerica's IT budget is about $4 million, and the move to Internap cost around $50,000--not including the $10,000-per-month service, plus high-speed service fees. The firm needs about $2 million to build out its mirrored data center. The IT team recently got the green light for the project from BrokerageAmerica execs, according to Janason, and plans are now under way.

The Disaster after the Disaster

A couple of days after BrokerageAmerica moved its brokers to Internap's conference room as a temporary trading operation, the firm faced another relocation effort--where to put more than 50 traders until its damaged headquarters was renovated.BrokerageAmerica's first mistake was not getting a backup site under way, but plans for the site are finally in the works (see "The Hard Sell,"). "We should have had another standalone site ready to go on its own, instead of having to put this all together again," says Anthony Martinez, senior vice president and head of the company's trading operations (who has since left the firm).

The only option was to build another temporary setup for the traders. So BrokerageAmerica rented space from RoyalBlue, its ASP, and set up shop there. The good news was that the firm was able to ride off the two DS-3s it uses to connect to RoyalBlue's back-end Fidessa trading system. "We plugged in a Cisco router and they patched us into their network and then back to our New Jersey site, so we could trade out of that, too," says Simon Janason, IT manager for BrokerageAmerica.

The bad news was that the relocation effort cost the firm more than anticipated. The moves to the Internap makeshift conference room and to RoyalBlue's site cost about $400,000. "We're still adding it up," says Martinez, who expects that the firm will likely tap out its $1 million insurance policy in the wake of the disaster.

Martinez's advice: Don't wait to set up a disaster-recovery site. If disaster strikes, it can be costly if you don't have a backup site of your own.