"It is getting harder to tell where a physical connection goes these days, all in the name of avoiding single points of failure," Manning says. He mentions an outage that happened last year when there was a fire in one of the Baltimore highway tunnels underneath the harbor. Turns out two of Freddie Mac's ISPs had data lines that passed through that tunnel as well, and the company lost some connectivity on those connections. Now Freddie Mac asks to see the routing maps from any ISPs it considers working with, though service providers have been more hesitant to supply this kind of information post-9/11.
As part of this implementation, the company uses F5 Networks's Big-IP load balancers so users don't even realize there are redundant ISP connections. "That was something we did early on, and it has withstood all kinds of problems," Potosky says.
Second, Freddie Mac has not one, not two, but three firewalls to separate its network traffic into various layers of protection. It has different security zones depending on the application, the user and the context of the user. It has a standard DMZ, and behind that is a zone where applications are authorized to play. Behind that is another zone, where the data lives. As you might imagine, maintaining the various firewall rule sets isn't easy, and several staffers are devoted to these operations. Any changes to the rule sets take at least a week to review, test and implement.
"We have deliberately created rings within rings--that has saved our butts a few times," Manning says. "About 18 months ago, we had a hardware problem with a particular network segment in the data centers; one of the core routers went into a loop that was deep within the data center. If we didn't have all these firewalls, we could have brought down the majority of our infrastructure. As it was, the problem only stopped traffic on a small portion on that segment, and we didn't have to bring down any applications."
Another time, employees were doing weekend maintenance on one building when a second building was struck by lightning. "We were able to limit the damage and route around those two buildings," Manning says. "Otherwise, we would have had to bring applications down."
