MMS: The Muscle Behind the Life Time Fitness Machine

History In the Making

Just a few years ago Life Time's IT department barely existed, and MMS was a twinkle in CEO Bahram Akradi's eye. The group was made up of contract labor and a CIO in the making, Brent Zempel, who came up through the health and fitness industry and lacked technology experience. One thing he did have in spades, though, was management expertise, and he used it to get his contractors pointed in a common direction. He also worked a deal with Renaissance Worldwide, the agency providing programming staff, to waive the finder's fee should he hire its employees. And Zempel did exactly that: He assembled his core development team from this contract labor, saving a bundle on recruiting.

Zempel also enticed Wesley Bertch, who taught Microsoft technology classes that Zempel attended, to evaluate the Life Time infrastructure, which wasn't nearly as buff as most of the health club's members. Bertch was hooked by the challenge and joined the Life Time team as director of software systems.

A New Plan

A number of factors drove Life Time to build MMS. The old system was down so often that many members bypassed it, resulting in nonpayment of dues. Data integrity was so low that lapsed accounts showed up as paid. And every time there was a major hiccup in the code, the system creator--a high-priced consultant--had to hop on a plane.But it's a sad fact of IT life that ideas rarely translate easily into working technology. Life Time pursued some dead-end vendors and architectures.

Bertch was key in deciding to use Java and J2EE as the core development environment. Everything had been developed using Visual Basic and other Microsoft products, but it was decided that the Microsoft environment could not support Life Time's goals for MMS--spiraling licensing costs and Microsoft's failure to deliver new software architectures on time (can you say .Net?) spurred the move. It was a bold decision--more than $1 million had been spent on contract labor, development and infrastructure to support a Microsoft solution. But Bertch felt the company had no choice. He says he firmly believes that none of MMS' success would have been possible had they stayed with Microsoft.

The company considered off-the-shelf member-management systems, but none could meet Life Time's requirements. The only viable alternative was Java, even though it would require heavy development and customization. This choice spelled the end for many contractors working on the Microsoft platforms. A wholesale body-and-skill swap took place, and soon the seats were filled with Java developers. Zempel credits the strong Renaissance relationship as the reason Life Time was able to make such a strategic shift. (Renaissance has been acquired by Boston-based Aquent.)

MMS 1.0 had two main architecture components: a client configuration based on Windows 98 running Internet Explorer 5.0, and a Web, application and database server infrastructure. The clients were local to each club; the server architecture resided at corporate. The Web servers, running Apache 1.3, served client requests, while the Tomcat 3.1 code handled servlets being fed to the application server. The application server ran BEA Systems' WebLogic Server 5.1, which controlled all EJBs (Enterprise JavaBeans). FreeBSD was the core OS on which the Web and application servers ran. The file server's performance characteristics and the cost of FreeBSD made this setup a no-brainer for Life Time. The database was Microsoft SQL Server 7.0, and directory services were handled by Sun ONE Directory Server (formerly Netscape LDAP and iPlanet Directory Server) running on Windows NT 4.0.This architecture went live in August 2000 and supported the conversion of all Life Time clubs outside Minnesota. That's because a major goal of MMS was to let members use any Life Time facility. And since most members are enrolled in Minnesota facilities, these clubs were most likely to use this cross-club perk. Therefore, they were scheduled last and a controlled, learn-as-you-go approach was taken using out-of-state clubs as a test.

This proved to be a smart move when problems with MMS 1.0 surfaced. Scalability and performance concerns made Life Time rethink the base architecture. The WebLogic server wasn't stable, and BEA was not very helpful because it did not support FreeBSD. Gary Lien, Life Time's senior systems analyst, said the company was "forced" to move WebLogic to NT 4.0 as a workaround, with the goal to get the server onto a Solaris platform as soon as possible.

Web server performance was also a major concern. The Tomcat Java servlet software would freeze occasionally. The solution: Move it off the Apache Web server and onto its own Solaris x86 platform. Two servers were added to help distribute the expected load when Minnesota was brought online.

Another core performance issue involved the database. All reporting functions were performed on the live production database; this led to intermittent slowdowns. So Life Time installed a second SQL server as a reporting database and set up replication from the main production database. The last major glitch centered around serving images to the club registration clients. To help curb fraud at check-in, digital images of all members are stored in a repository. Serving those images from corporate across the WAN brought unacceptable latency, so the company installed image proxy servers, based on FreeBSD 4.2 and Squid, at each club. The proxies are seeded from the central image store regularly, and the club client machines are configured to access the proxies for all image calls. Display speeds increased dramatically.

Moment of TruthThese problems were identified by November 2000, and plans for MMS 2.0 were born. Once 2.0 was fully baked, Life Time could bring its primary Minnesota membership online. The conversion took place in June 2001, and the IT staff's hard knocks and tinkering paid off.

"We were expecting a tsunami," Bertch says, but "got a ripple." Data conversion from the old platform took 12 hours, and because Life Time had cleaned up inaccurate records, data integrity was high.

However, not everything went smoothly during the Minnesota go-live. For example, a misconfigured SAN let multiple applications write to the same data store, thus corrupting that area. But the technology problems paled in comparison with the personnel issues. Just before MMS 1.0 went live, the lead contractors in charge of development and testing left abruptly--taking some strategic intellectual property. It took a trip to the developer's house to retrieve much of the code, but Lien says they were able to "hack through it and get it to compile."

Life Time didn't stop with MMS 2.0. The company is now using version 4.0, a release aimed at decreasing some of the complexity related to the previous architecture. Specifically, the two Tomcat servlet systems created administration and maintenance headaches, so the team elected to collapse that functionality into the WebLogic application servers and kill the Tomcats. Because many of the MMS software modules reside on the same physical platform, running the servlet and EJB systems within the same box should allow for easier troubleshooting and optimization while cutting the complexity of code rollout.

But the Life Time application development folks can't rest on their laurels. A number of enhancements are in the works for MMS 5.0 and beyond. A Web services architecture will play a big role in future development efforts; in fact, Lien is considering BEA as the platform of choice for a Web service integration server. This decision will involve building JCA (J2EE Connector Architecture) adapters into key MMS software modules, focusing first on Life Time employee data systems.Lien views Web services as the key transport mechanism for all future Life Time application development. About the Web service interface Life Time created using SOAP with scheduling system ASP Xtime, he says, "The first Web service implementation takes some time, but it's pretty simple once you go through it."

The appeal of Web services is the ability to move specific data elements among internal business systems and to external partners. Application-to-application integration for every business system creates a hard-to-support environment, says Lien. But a hub-and-spoke architecture, where the JCA adapters sit in the middle playing traffic cop and each application has a single interface to communicate to the central JCA integration server, will simplify code development and maintenance while decreasing time to production for new features and applications.

Other key application environment changes Life Time is considering include a move to a non-Microsoft client OS, possibly Linux, at the clubs. This would help drive down licensing costs.

Browser neutrality is another goal, though Life Time realizes this will be hard to achieve. Finding a browser that supports Windows, Linux and Apple Mac OS at the client is nearly impossible, especially as some non-MMS applications support only certain browsers.

Another focal point is software module administration and code maintenance. Lien says MMS is very segmented, letting software modules be reused or shared among varied business applications. The adoption of the JCA and, in turn, Web services will decrease the number of software integration points, thus easing maintenance.Despite successes, using FreeBSD as a production OS has been a mixed bag. Life Time aims to gain better hardware control and support by deploying Sun SPARC systems running Solaris. FreeBSD will still play a role on department file servers, but the MMS environment and core business applications will run on Solaris wherever possible.

A final--and potentially far-reaching--matter for Life Time is resale of MMS. Just about any industry requiring member-management control could benefit from the maturity and modularity of the MMS code. Although major issues still exist around packaging, support, documentation and professional services, Life Time is eyeing MMS as a potential revenue generator and has launched this effort via an organization dubbed AveriSoft.

Life Time's IT operations group supports a decidedly heterogeneous environment. Products from Microsoft, Apple, Cisco, Sun and Netscape all play critical roles in the new MMS rollout. But technology alone doesn't guarantee success--IT people were required for the initial rollout and are needed for continued support of every club in the Life Time chain. Having an IT staffer at every facility is not fiscally feasible, so operations had to devise ways to remotely manage each site. Standardized client and server configurations help keep all clubs uniform, allowing for cookie-cutter installations. WAN requirements are identical at every facility, so each receives a 256-Kbps frame relay circuit to link back to corporate business resources, including the Internet. For redundancy, each club has a dial-up ISDN service.

As in many other enterprises, Cisco dominates Life Time's network infrastructure, supplying routers, PIX firewalls, Catalyst switches and the VoIP system. But Life Time is veering a bit off the beaten path by using open-source Squid Web proxy cache software running on FreeBSD instead of Cisco's content-delivery solutions.

Jud McKee, director of network operations, and Eric Edgar, systems engineer for Life Time Fitness, says that to maintain control over a growing infrastructure, strong server administration and build processes were critical. Because Solaris and FreeBSD dominate the server population, open-source software seemed a perfect fit. The solution was Cfengine, which helped automate the server administration and rebuild of the Sun server environment to the point that "we can rebuild a server in less than one hour," Edgar says.As MMS gained acceptance within Life Time, the need for a testing and quality-assurance infrastructure led to the creation of Testworld, an environment that emulates the production MMS system.

People Who Need People

Life Time's IT department now has a staff of 48 people, comprising four main parts: operations, software, project management and administration. The well-defined roles of the technology teams is impressive, especially when you consider that none of it existed a few years ago. We've seen many start-ups forego structure when in high-growth mode, leading to flat organizations with no clear reporting responsibilities and weak role definition. That model does work--for a while. Then things start to unravel. But Life Time created an IT organization that provides logical career paths for its employees and should be able to scale as the company pursues its 50 percent per year growth goal.

In the near future, however, scaling won't be an issue: A hiring freeze has been placed on the IT organization as the company prepares for continued facility growth and an anticipated IPO. Increasing efficiency has become the main focus. Streamlining project-request processes has helped better track line-of-business initiatives that require IT's help (for more on the decision-making process, see "Where the Gloves Come Off").

The Sky's the Limit Despite the hiring freeze, Life Time IT has a full project list that keeps growing to keep pace with the expansion of the company. One concern: The selection of a Siebel CRM system will strain the current infrastructure and raise MMS integration issues. Installing the CRM application outside the corporate offices will require a complete assessment of the club server and network infrastructure.

To help future application rollouts to the clubs, Life Time is rebuilding its WAN topology, looking to do frame-link consolidation at strategic geographic termination points, thus pulling them back to corporate via larger bandwidth links. Life Time says this change should cut costs and allow for easier provisioning of new club WAN services. That's critical because the company plans to add six clubs per year.

McKee is evaluating QoS software to help prioritize critical business traffic coming from the club facilities. The products he is considering are software-based and would operate in conjunction with the Squid proxies at each club. The QoS and Siebel CRM projects represent the bulk of funded initiatives for the network operations group. But as is often the case, projects lacking formal funding are just as critical to the success of the IT organization and company as a whole. Strategic initiatives, such as antivirus software to the desktop, evaluation of dDOS (distributed denial of service) prevention technology, and creation of custom applications for the accounting and payroll departments, are creating serious challenges for an organization with a hiring freeze and fixed budget. Stay tuned for updates on how Life Time IT is keeping its infrastructure in top shape.

James Hutchinson is Network Computing's director of editorial content. Send your comments on this article to him at [email protected].

• Extensible Markup Language (XML): Similar to HTML, XML is a text-format language that is the basis for structured documents on the Web. See www.w3.org/XML/1999/XML-in-10-points.• Simple Object Access Protocol (SOAP): Developed by Microsoft, SOAP uses XML and HTTP to let apps running on different OSs communicate. SOAP has been presented to the IETF. A somewhat dubious distinction is that SOAP calls can easily pass through firewalls, which normally accept HTTP traffic over Port 80. See the spec at www.w3.org/TR/SOAP/.

• Universal Description, Discovery and Integration (UDDI):Envisioned as the mother of all yellow pages, UDDI is an XML-based registry where businesses can list themselves on the Internet by name, location and product or service offered. The UDDI specification incorporates W3C and IETF standards, such as XML, HTTP, SOAP and DNS. See www.uddi.org/specification.html.

• Web Services Description Language (WSDL):WSDL is an XML-based language that describes the services an enterprise offers. Works in tandem with UDDI to help businesses reach customers. See the spec at www.w3.org/TR/wsdl.What would you do if you inherited an infrastructure built from a hodgepodge of gear that nobody ever tracked or documented? If you're as fortunate as Jud McKee, Life Time's director of network operations, you'd get to rip it out and start fresh. McKee and his IT team adopted FreeBSD as the server platform, dismantled many of the dozen or so Microsoft PDCs (Primary Domain Controllers) that were wrestling for Windows network control, started a campaign to make Life Time execs take security and authentication seriously, and looked to create some best practices for server administration.

Picking FreeBSD in a world dominated by Windows NT and Solaris was risky for a company that was just starting to get its sea legs in the IT world, but the move paid off for Life Time. Software and licensing costs, which in a typical Microsoft installation can quickly become a millstone around a fledgling enterprise's neck, are nonexistent. But as with any technology choice, you need the proper skill set to manage it. That task fell to Eric Edgar, systems engineer for Life Time Fitness, and his team. "FreeBSD has the best file server performance numbers of any operating system," Edgar says, so the choice made sense for the real-time response requirements of the new MMS. And by incorporating Samba file and print services for Microsoft client compatibility, FreeBSD helped meet all of Life Time's immediate server requirements for its clubs.

Authentication via LDAP directory services lets club employees access MMS based on location and user privilege. The concern was that high employee turnover would make tracking and maintaining an accurate user database unwieldy. Instead, certain stations, such as the main desk area, were set up without user authentication and have limited MMS functions; those stations focus on member check-in and registration.• Apache

• BEA Systems WebLogic Server

• Cfengine

• FreeBSD

• Java• J2EE

• J2EE Connector Architecture

• Samba

• Sun ONE Directory server

• Squid cache• Tomcat

Some of the lessons life time learned are universal:

• Get your operations and development groups talking regularly. It'll save you time and money.

• Invest in a solid test environment. Before rolling technology solutions into production, you must put them through their paces.

• Train the users. The only way to prevent a tsunami of support calls is to prepare the tech consumers.• Crawl before you run. When launching an enterprisewide solution, start small. Pick a group willing to iron out the kinks with you.

• Get everyone from end users to the CEO to buy into IT initiatives by putting technology in terms they understand.

• Don't take on every project; it's OK to say "no" as long as the affected lines of business understand why. If possible, present workarounds and offer a time line for re-evaluating the need.