On Location: Chicago Tribune: Server Consolidation

Unfortunately, the cutover to the new server environment hit a snag: A small piece of software written for the transition contained a coding error that caused the Tribune's editorial applications to experience intermittent processing failures. As a result, the Tribune was forced to delay delivery to about 40 percent of its 680,000 readers and cut 24 pages from a Monday edition, costing the newspaper nearly $1 million in advertising revenue. Don't ever let anybody tell you that data-center renovation isn't risky.

More than nine months later, the Tribune's IT department is still sold on server consolidation, despite the high-profile outage. "One five-hour delay doesn't change the fact that this project has brought us significant cost savings and improved our system reliability for the long run," says Darko Dejanovic, vice president and CTO of both the Chicago Tribune and Tribune Co., which also owns the Los Angeles Times, Long Island's Newsday and about a dozen other major metropolitan newspapers. "There's no question that server consolidation was the right move for us."

Following the outage, the Tribune's IT staff re-examined its processes and made a few changes. CCI, the Tribune's editorial applications vendor, now maintains a more regular on-site presence in the data center. Problem-escalation policies have been changed to help end users raise concerns more quickly. Application-testing processes have been modified to incorporate more thorough testing by business users. And to the IT staff's credit, the consolidation project not only stayed on track, it finished a few weeks ahead of schedule and about 10 percent under budget. After editorial applications were stabilized, the Tribune proceeded to migrate applications for operations--that is, the physical production and printing of the newspaper--and circulation to the new Sun server environment. Both made the transition without a hitch, and no special migration code was needed. By the end of August, all the applications scheduled to run on the new servers were operational.But though the Chicago Tribune's consolidation project is largely complete, that of its parent, Tribune Co., is just getting started. For one thing, the company has consolidated all its mainframe applications at its Long Island, N.Y., data center, decommissioning its one remaining IBM System/390 in Chicago. Mainframe processing, mostly for billing applications that don't run on Solaris, is now centralized on Long Island, and Chicago staffers access them through remote 3270 terminal connections.

"As we gradually took [other] applications off the mainframe, we realized that we were incurring very high costs in maintaining underutilized mainframes at two different locations," Dejanovic says. "By moving from two locations to one, we've achieved several million dollars in cost savings."

The consolidation effort is beginning to reach into other Tribune Co. properties as well. Dejanovic says that one of the company's other newspapers is planning a move to a similar, clustered Sun server configuration in the near future, though he declines to say which one. Several of the Tribune Co. newspapers have begun to explore ways to use one another as backup sites, though the lack of WAN-distance dark fiber limits the newspapers' ability to do clustering, as the Chicago Tribune did.

"There are some gigabit-speed services available in the WAN, but they are very expensive and there are not many choices out there," says Milind Dere, one of the project managers for the server-consolidation effort.

And while it is consolidating hardware, Tribune Co. is also looking to consolidate software, Dejanovic says. Currently, each newspaper maintains its own applications for classified advertising and billing, which means that the parent company must support about 10 different billing packages and the same number of classified advertising programs."Most newspapers feel that they have their own unique processes for handling those functions, but we've found that most of the business processes can be standardized," Dejanovic says. So far, the Tribune newspapers have standardized about 95 percent of their classified advertising processes and about 90 percent of their advertising sales processes. Over the next three years, Tribune Co. will replace the disparate billing and advertising applications across the company with a single package that will be used by all the business units.

"The different [newspapers] will not necessarily share the same data, but they will have the same processes and the same GUI for accessing them," Dejanovic says. "Over time, that will let some of the call centers handle calls for each other--East Coast centers can handle some of the early-morning calls, and West Coast centers can handle the late-day and evening calls."

As the company's newspapers prepare to migrate to new server architectures and common applications, Dejanovic hopes to do the migrations in groups. This process is sometimes called "regionalization" and is usually done geographically. But Tribune Co. will instead group the newspapers based on their readiness to migrate.

"If a business unit is not going to be ready for a new environment for three years, it gets expensive to wait," Dejanovic says. "It's better to migrate the four or five that are ready, regardless of where they are."

Back at the Chicago Tribune, there are some new projects in the works as well. One is the implementation of Solaris 10, which will let individual apps run on partial CPUs, freeing up processor power and making more efficient use of the Sun server cluster. Solaris 10 will make server consolidation an even more attractive option for Sun users, Dere predicts. "That capability will bring us closer to realizing the full potential of the Sun environment, both in terms of server capacity and clustering," he says. Sun is shipping Solaris 10, but the Chicago Tribune will not do the migration until more of its specialized newspaper-industry applications are available on the new OS.What advice does the Tribune Co. IT staff have for other IT departments considering data-center renovation?

"Don't hesitate," Dejanovic says. "We were convinced it would be cheaper to consolidate, and we realized those savings. And don't underestimate dealing with one vendor. We have saved a lot by negotiating good maintenance contracts and lowering our overall cost of support."

Scott Tafelski, director of publishing technology for Tribune Co., suggests that server consolidation will become a popular trend among large enterprises. "Tomorrow's data center will look a lot like yesterday's data centers, in that a lot of the technology will be centralized," he says. "From a cost perspective and a management perspective, it's a direction that makes a lot of sense."

Tim Wilson is Network Computing's editor, business technology. His background includes four years as an IT industry analyst and more than 14 years as a journalist specializing in networking technology. Write to him at [email protected].

In August 2004, our sixth "On Location" documentary-style case study took us to the Windy City to profile the Chicago Tribune, the seventh-largest daily newspaper in the United States (see The Chicago Tribune, server consolidation ). We reported on the company's massive server-consolidation project, which combined mainframe migration, Sun server upgrades and disaster-recovery projects with a blazing high-speed fiber connection that enabled the Tribune to maintain an active-active configuration between data centers two miles apart. In this update, we checked in with the IT folks and learned that they're pleased with the project's outcome and are, in fact, preaching consolidation across all the Tribune's properties while moving forward on standardization initiatives and an upgrade to Solaris 10.Visit our "On Location" home page for more on this and previous stories.

'On Location' Packages:

• American Airlines Center, Wi-Fi network

• The Chicago Tribune, server consolidation

• University of Florida, P2P blocking• U.S. Navy, knowledge management

• McCarran International Airport, Las Vegas, common-use terminal equipment

• Children's Hospital Boston, ERP

• Life Time Fitness, Web services