Mortgage Company Cashes in on Web Services

If you've bought a house, chances are Freddie Mac has touched your loan. A leader in the secondary mortgage market, Freddie Mac--the Federal Home Loan Mortgage Corp.--owns one-sixth of all American mortgages. Its computers process tens of billions of dollars every day and exchange critical information with a host of banks, brokers and investors.

So when the company decided to migrate critical applications from mainframes to Web services, the executive team knew the transition would be a high-wire act without a net. A single misstep could choke off the flow of transactions, compromise security and land Freddie Mac in serious trouble. But the payoff--streamlined operations that would save billions of dollars--was worth the risk.

Freddie Mac, based in McLean, Va., nailed its balancing act through a combination of great planning, distributed architecture and solid investment in the right standards.

Mac DaddyFreddie Mac manages a securities portfolio of $1.4 trillion and last year purchased $760 billion in mortgages. That's millions of dollars in loans purchased every second of every day.

This volume of information processing requires a small army to monitor and manage: Of the company's 4,500 employees, about one-third work in IT-related functions. It also requires a robust infrastructure. This year Freddie Mac will spend more than $350 million on new IT applications. Its 1,700 servers are mostly Unix-based, running IBM AIX, Linux and Sun Solaris, though there are some Microsoft Windows and even 78 Novell NetWare servers running in its data center. It also has a 1,200-port SAN--with equipment from EMC, Hewlett-Packard and IBM--that holds 350 TB of data. And not surprisingly, desktops and laptops outnumber the staff, with more than 7,800 PCs scattered around the McLean campus.

Freddie Mac's business puts it in the midst of the complicated web that is the mortgage banking industry. Here's how it works: Mortgage brokers troll the waters looking for the best deals and the most appropriate loan packages for consumers. Lending institutions--mostly banks and savings and loans (originators, in industry parlance)--put the loans together. Finally, title agents, escrow agents and other actors process the loans and produce the foot-high piles of closing documents.

For the consumer, this is the end of the loan process, but not for the mortgage companies. At this point the secondary loan market takes over. Think of this as the stock market, but instead of trading shares of corporations, the investors trade shares of groups of mortgages.Freddie Mac buys loans from its 2,500 originators, then sells those loans to investors, such as other banks and large pension funds. The originating banks use this revenue to plow more money into the mortgage market, keeping rates down and liquidity up. Freddie Mac's job, along with the other secondary mortgage players, is to keep the funds moving so more people can buy homes at the most attractive rates. It is joined by Fannie Mae (Federal National Mortgage Association), which began its life selling mortgages insured by the U.S. government and is now Freddie Mac's main competitor.

Greenbacks, Yes; Green Screens, No

Until the late 1990s, Freddie Mac processed, serviced and packaged loans using several hundred green-screen mainframe applications that ran in a CICS/Cobol batch environment. Then it started to get the Web services religion.

The company's first foray into distributed online processing came in 1999, when it created LoanProspector. com--an automated loan-underwriting app. Lenders key in information on potential mortgages to ensure they meet Freddie Mac's guidelines for risk and the borrower's ability to pay. Before LoanProspector went online, "you had to go to our legacy system to close the mortgage and deliver the loan to us," says Kim Petty, vice president of Freddie Mac's sourcing and servicing business information services. Now more than 99 percent of Freddie Mac's single-family loans are serviced over Web applications.

LoanProspector isn't the only Web application Freddie Mac uses; more than half of its legacy apps have been converted to XML-based Web services apps that run interactively, in real time. Where it used to take days to approve a particular loan, now it takes seconds, and loans can be processed typically within 24 hours rather than a week.

This makes for more nimble financial markets, in which Freddie Mac's banking customers get their money quickly--in some cases more quickly than they can accommodate. "With the older system, when we purchased a loan, we would turn the money back around to the lender in seven to 10 days," says Edward Albrigo, vice president of the enterprise program office for Freddie Mac's mortgage operations and technology division. "Now it can take as little as 24 hours, and in some cases the back office systems of our lenders weren't prepared to get that fast a response." Imagine going to your bank and having it say it can't deposit your money until tomorrow. Fortunately, these back-office systems have been updated.

The mainframes haven't completely gone away, of course. Freddie Mac still uses them for a variety of applications and is even considering using new ones to leverage its big IBM iron (see "Web Services Won't Kill Mainframes," below).

The loan origination side of the house isn't the only place seeing some Web action. The packaging of Freddie Mac's securities is also heavily Web-dependent now too. "Before the Web, we would call our dealers ... when we had debt that we wanted to issue," says Bill DeLeo, Freddie Mac VP and business information officer, investment and capital markets division. "Now we can be much more efficient in our auctions, and we have saved from three to five basis points in selling our securities." (A basis point is one-hundredth of a percent.) That works out to nearly $1 billion over the past several years.Making The Transition

How did Freddie Mac get from green-screen batch jobs to modern interactive Web apps? In a word, planning. "Having an overall architectural plan and agreeing on the overall high-level architecture at the start were key points," says Steve Manning, Freddie Mac's VP of computer and network operations, business information services division. "And also taking one step at a time and trying not to make too many changes at once." Not that the executive staff had much choice. The project began in the middle 1990s and won't be completed until the end of 2007. What's important isn't how fast they made the change, but the careful and deliberate sequence of events that has led up to their Web services embrace.

The first step was moving to a private all-IP network. The company knew any solution to moving off the mainframe was going to involve Internet protocols, so admins created a transition plan. "We had an opportunity to get our feet wet with Internet technologies without getting exposed," says Bryant Potosky, enterprise architect at Freddie Mac. "In the early 1990s, we created an extranet, based on a private IP network that was very controlled from a security perspective. That was a key stepping stone. We could monitor and mitigate many of the security issues before we had to open up to the public Internet."

This sounds simple, but in reality was probably the most difficult step. Part of the problem is the sheer number of entities involved in processing and selling loans, and that each of these entities had to connect to Freddie Mac's private IP network.

"There are so many moving parts and players," says David Barkley, the company's director for industry standards and practices. "The lenders don't have total control and have to interact with many others, such as title agents and real estate agents." And when you factor in 3,000 county clerk offices, each with different documentation requirements, and hundreds of banks and savings and loans, the cast of characters becomes enormous.

The problem was that Freddie Mac's batch systems--like many other batch systems at other enterprises--weren't good enough. "These systems will tell you about what happened yesterday, rather than saying what was happening now and what will happen tomorrow," Potosky says. "It wasn't meeting our needs or our customers' needs." Freddie Mac was also spending a great deal of time training customers on proprietary systems.

The private extranet provided a proving ground for debugging its interactive applications, and also was important in modeling user behavior. In fact, Freddie Mac learned that the interactive world was an entirely new environment, and more important, learned what was different about servicing interactive users and how to build its systems around these needs. "We found that the behavior of the interactive users cannot be predicted," Potosky says. "As we gave the customers these new capabilities, they found a lot of different ways of using it than we had anticipated. That caused some interesting system behavior, including performance issues."

Freddie Mac had to develop new diagnostic routines to trace these problems, and also understand the various interactions of its apps as users logged in online. "Since then, we put in the ability to turn debugging and tracing capabilities on and off in all our applications," Potosky says. "The problem is that we ask for 400 different data elements for some of our more complex transactions. So one user session can kick off a hundred different interactions. This isn't like ordering a basketball from an online retailer."The efforts paid off. The early proponents that favored big iron saw that the security concerns could be managed over IP networks. And as more case studies came in showing what other industries were doing with IP technologies, such as manufacturing and technology vendors, confidence in putting together an Internet solution grew within the company. "We saw that we weren't quite bleeding edge," Potosky says. "There was viability there." Up until three years ago, more than 85 percent of its IP traffic went over that private network, but now the majority of traffic is routed over the public Internet. As you might imagine, given the volume and nature of its transactions, Freddie Mac takes security and redundancy seriously (more on this later).

Along with deploying a private IP network, Freddie Mac also was creating the application architecture. Initially it opted for Java-based applications and avoided Microsoft servers because of security concerns. Now that it has built a solid infrastructure for these applications and it understands the issues in hardening Windows servers, the company is looking at the .Net environment and IIS. This is typical of the arc many companies that have opted into Web services have taken.

Standard Operating Procedures

Freddie Mac also recognized that Internet standards would go a long way to helping it integrate with the innumerable entities that took part in processing loans. To that end the company took an early and leading role toward adopting standards and proselytizing these standards across the mortgage industry, including the WS-* Web service standards.

Dozens of Freddie Mac's staffers are involved in various capacities with the Mortgage Bankers Association's group called the Mortgage Industry Standards Maintenance Organization (MISMO). The group began in 2000 with plenty of support from Freddie Mac and Fannie Mae, and the idea was simple yet powerful: Set data interoperability standards to enable more efficient loan processing and closing, and facilitate transactions between lenders and the secondary loan markets where these loans are resold. At the heart of MISMO's standards efforts are XML documents and schemas that enable paperless transactions as loans are moved from the originating bank to Freddie Mac and then into a collection of securities. "Our goal was to create a legally binding document that we can use everywhere, and create a paperless transaction from end to end," Barkley says. Although there's still a lot of paper involved in the transaction, MISMO has laid a solid foundation with these XML standards.The MISMO standards efforts were helped by a general standards climate in the mortgage industry that dates back to X.25 EDI networks and X.12 standards bodies. The idea was to bring them into the 21st century with Web transactions, using the structured "smart documents" that XML could describe and that could be used by just about any part of the Freddie Mac mortgage ecosystem.

But the more the company relied on XML and Web standards, the clearer the need became for a wholesale upgrade of its legacy systems. This fed into its next step, the actual conversion effort. Freddie Mac had hundreds of different mainframe applications covering numerous business processes, some with code that dated back to 1986. Most companies would have started a small pilot program and dipped a hesitant toe in the Web waters. Not Freddie Mac. It tackled its biggest customer-facing apps--the ones involving loan approvals for their member banks for single-family residences--first. Why? "The single-family side was the most antiquated side of all our applications, and it was the side that was the furthest behind the financial services industry in adopting new business techniques and methods," Albrigo says.

This conversion went along two different tracks-- developing Web apps that any size bank could use with minimal computing requirements, such as an Internet connection and a browser, and larger-scale systems designed to interact with its biggest customers. The key was that both tracks used the same Internet standards, protocols and methods. "These are the business-to-business standards that guide how we interact with each other and how we use Internet protocols and standards to specify the information that moves across applications," Albrigo says.

Freddie Mac used various bootstrap methods to evangelize the mortgage industry and get the word out about the new applications, and developed various training materials to quickly deploy the new applications. In a year, according to Albrigo, more than 1,200 customers were brought onto the new system; and the majority of them came on board in the last few months of 2005.

Having the browser-based infrastructure helped the lenders quickly learn how to use the system. Freddie Mac's move to Web services was also well-timed because it coincided with Web initiatives at its major banking customers. And these customers all wanted to migrate to more flexible systems that could give them faster turnarounds on their mortgage investments.The company also distributed its development operations beyond its headquarters and has begun to employ developers in other cities to force itself toward a more distributed architecture. This lets it tap into other labor pools around the country and hire the best developers possible.

Finally, Freddie Mac moved from building its own applications to buying the best-of-breed packages from third parties and integrating off-the-shelf components. This change took the longest to inculcate into the corporate culture, but now is accepted practice to the point where they are widening the kinds of applications supported and considered. The company did this to help cut costs and shorten the time to develop applications.

Redundancy And Security To The Max

Freddie Mac's mainframe architecture is a conventional one with two data centers located about 10 miles apart in suburban Virginia. Once the Web came into play, the company had to restructure this environment and still provide the same level of reliability it got from its IBM big iron. The company takes redundancy--and security--to the extreme, given its circumstances. "We don't need 24/7 reliability all the time, but when we need it, we need it big time," DeLeo says.

First, Freddie Mac uses three Internet service providers--and is considering adding a fourth later this year. Each ISP goes through a different central office and has as little duplicate path as possible for the D.C. area."It is getting harder to tell where a physical connection goes these days, all in the name of avoiding single points of failure," Manning says. He mentions an outage that happened last year when there was a fire in one of the Baltimore highway tunnels underneath the harbor. Turns out two of Freddie Mac's ISPs had data lines that passed through that tunnel as well, and the company lost some connectivity on those connections. Now Freddie Mac asks to see the routing maps from any ISPs it considers working with, though service providers have been more hesitant to supply this kind of information post-9/11.

As part of this implementation, the company uses F5 Networks's Big-IP load balancers so users don't even realize there are redundant ISP connections. "That was something we did early on, and it has withstood all kinds of problems," Potosky says.

Second, Freddie Mac has not one, not two, but three firewalls to separate its network traffic into various layers of protection. It has different security zones depending on the application, the user and the context of the user. It has a standard DMZ, and behind that is a zone where applications are authorized to play. Behind that is another zone, where the data lives. As you might imagine, maintaining the various firewall rule sets isn't easy, and several staffers are devoted to these operations. Any changes to the rule sets take at least a week to review, test and implement.

"We have deliberately created rings within rings--that has saved our butts a few times," Manning says. "About 18 months ago, we had a hardware problem with a particular network segment in the data centers; one of the core routers went into a loop that was deep within the data center. If we didn't have all these firewalls, we could have brought down the majority of our infrastructure. As it was, the problem only stopped traffic on a small portion on that segment, and we didn't have to bring down any applications."

Another time, employees were doing weekend maintenance on one building when a second building was struck by lightning. "We were able to limit the damage and route around those two buildings," Manning says. "Otherwise, we would have had to bring applications down."These multiple firewalls are all part of the company's strategy to have a solid security infrastructure in place. "We spend enormous time and energy on security prevention--that has grown more so than our bandwidth capabilities," Potosky says. "Now we have three different crews of people dealing with security: one at the broader information security level, one for the Web infrastructure level that also handles the application servers and integration servers, and then one crew working at the pure network level. There is a lot we can track and monitor, and this granularity is a key part of our architecture."

Third, the company virtualizes as much of its computing infrastructure as possible, including servers, operating systems, storage and software. Employees use VoIP phones so as they move from one job to another, they can keep the same phone number without the IT department having to track the change. This is nothing new--after all, IBM has been running virtual machines on its mainframes for decades. But Freddie Mac is attempting this throughout its network, such as running multiple instances of applications to share the load.

The next frontier is to virtualize the entire network infrastructure, having a personal VLAN created on the fly for particular users and particular applications. "That is a lot more difficult," Manning says. "VLANs have issues that cross wide-area networks. That is a fixed infrastructure, and we want to be able to do it on the fly as much as possible." To get ready for this situation, Freddie Mac has upgraded all its Cisco routers to the latest versions and is beginning a pilot network access control project to monitor endpoint security.

Finally, Freddie Mac doesn't take anything for granted and has been particularly fussy when it comes to handling insecure applications, such as file transfer protocol. The company uses a proprietary FTP server that doesn't grant access to the underlying file system, requires authentication for sending or receiving, and has point-to-point firewall controls, under a VPN. Freddie Mac uses both FTP-SSL products as well as those compatible with FTP over SSL connections.

David Strom was the founding editor in chief of Network Computing and is now a freelance writer living in St. Louis. He is the author of two computer books and is a frequent writer, blogger, speaker and podcaster. He can be reached at [email protected]; you'll find his blog at strominator.com

Petty, Vice President of sourcing and servicing in the business information services group at Freddie Mac, leads the planning and implementation of systems changes for the organization's single-family, multifamily and affordable lines of business. Since joining Freddie Mac in 1986, Petty has held a variety of positions involving strategy integration, credit risk oversight, automated underwriting operations and corporate re-engineering.

Most bizarre IT inquiry: "Can I use a hair dryer on my wet laptop?"

Shortest IT project ever worked on: Three weeks

Most unusual family-related computing request: Playing Jib Jab's "This Land" parody during the 2004 presidential campaign for my young son ... over ... and over ... and over again.How many PCs my immediate family owns: Five (three laptops, two desktops)

Where you can find me most Saturday mornings: At the gym

What my co-workers don't know about me: Spent four years in the U.S. Air Force programming computers that displayed data in an underground command post

Subject that makes me rant: Current favorite--price of gas

Actor who would play me in a film: Kate CapshawWheels: 2007 Lexus ES350

CD in my car stereo system right now: CD? I plug in my iPod.

Must-see TV: 24 (very edgy)

Sports team: Pittsburgh Steelers (no question!)

Web browser of choice: Internet ExplorerComfort food: Macaroni and cheese

Last thing I purchased online: Exercise DVDs (see comfort food)

Web Services Won't Kill Mainframes

Before Freddie Mac started down the Web services path, it had hundreds of mainframe applications. Many are still running and won't be replaced even when more Web applications are brought online. That's because Web apps are just one way for Freddie Mac to do business with its customers.

"The Web is nothing more than an electronic channel and another way to enter trades into our system," says Bill DeLeo, vice president and business information officer for Freddie Mac's investment and capital markets division. "We are using the Web to deliver current products to new markets, to expand the number of dealers that we work with. We are using the Web to enhance our current system, not to replace mainframes."The mainframes--two IBM Cluster 1350s--are one of two Geographically Dispersed Parallel Sysplex configurations in North America, and have more than 770 MIPS of processing power between them. They process more than 5,000 batch jobs each day. These two mainframes aren't going the way of the dodo anytime soon--given Freddie Mac's transaction volumes, it still makes sense to use them. "We absolutely need the mainframe for some high-volume transactions and some of the newer applications we're looking at," says Kim Petty, VP of sourcing and servicing in Freddie Mac's business information services group.

Part of the reason is the complexity of Freddie Mac's interlocking applications. "We have more than 350 different applications," Petty says. "Understanding how all those applications fit together to deliver capabilities to our customers wasn't easy, and each of them plays a critical role in how we do business." Petty needed to figure out the various linkages and "touchpoints" before she could begin to replace these mainframe programs with Web counterparts.

"In some cases, our legacy apps do things we had no clue about," Petty says. "For example, we are now building a Web-based mortgage servicing engine, and we are finding out the mainframe one is doing things to handle mortgage defaults that we didn't expect. Now we have to make sure that this piece is taken care of someplace else."