Case Study: Building Network Stability

Company: Rudolph and Sletten

Project Leader: Sam Lamonica, Executive In Charge of Information Services

Technology In Focus: Open-source network and host monitoring software

Problem: An unstable network based on old, misbehaving equipment was disrupting access to business-critical resources for workers at the company's multimillion dollar construction sites.

Solution: Rudolph and Sletten deployed GroundWork's open-source Monitor system.

Bottom Line: A more stable network infrastructure has enhanced productivity by eliminating downtime, while giving the company the flexibility to deploy new technologies, including wireless access points for its permanent buildings and construction sites.

If any company can say it "built" Silicon Valley, it's Rudolph and Sletten. The Foster City, CA-based construction company has planned, developed, and built office and manufacturing facilities for just about every major Silicon Valley manufacturer, from industry pioneers to latter-day market leaders.

Rudolph and Sletten's early customers include Fairchild Semiconductor and Memorex, two of the Valley's first successful start-ups. More recently, it has developed corporate campuses for Veritas Software, Apple Computer, HP, and Sun Microsystems.

At any given time, the 800-employee company has about 50 large-scale construction projects in the works. Rudolph and Sletten is, for example, rebuilding the LAMC Sunset Hospital in Los Angeles, the new Molecular Foundry at the Lawrence Berkeley National Laboratory, and the University of California Engineering Building Unit 3B in San Diego.

Because of their complex nature--we're talking huge, multimillion dollar sites, with subcontractors working on hundreds of construction-related jobs spread across several acres--the company's projects often take years to complete and involve dozens of employees. That makes the operations trailer at each Rudolph and Sletten construction site a sort of de facto regional office, but one with a twist.

Each site's information services requirements are similar to those of any remote office: They all need high-speed connectivity to the company's headquarters in order to access business-critical applications, including construction planning and collaboration programs and e-mail. The twist is that they lack the stability of a permanent location. Running a mission-critical network out of a trailer isn't for the faint of heart, especially when connected via temporary broadband links, surrounded by heavy construction traffic, and managed by non-technical users.

In the past, that unpredictability, coupled with a hodgepodge of aging switches, routers, and firewalls, often spelled trouble for Rudolph and Sletten's network. Down nearly 20 percent of the time, it played havoc with productivity, says Sam Lamonica, the Rudolph and Sletten executive who manages the company's information services.

"When we have connectivity problems at a job site, we literally lose the lifeline to our infrastructure," says Lamonica. A network outage leaves onsite project planners, supervisors, and engineers without access to Prolog Construction Manager, the business-critical application the company uses to manage virtually every aspect of its construction projects.

To gain control of their balky network environment, Lamonica and his staff of 14 turned to an open-source alternative to the major systems management solutions in the market. Since deploying GroundWork's GroundWork Monitor, Rudolph and Sletten's distributed network uptime has jumped to 99.8 percent, according to Lamonica. That means monthly downtime has gone down from nearly a week to about 12 minutes.

Moreover, the new monitoring system has helped increase the company's leverage with suppliers, which can no longer point the finger when their products or services fail. It has also enhanced Rudolph and Sletten's ability to pinpoint problems before failures occur, as well as allowed the company to deploy extensive wireless capabilities at its job sites--all at a fraction of what a similar system from HP or BMC Software would cost.

COBBLED TOGETHER

When Lamonica joined Rudolph and Sletten in the fall of 2003, the company's network infrastructure was in a state of disarray--a hodgepodge of home-grown, aging hardware cobbled together by four or five regimes, as he describes it. "When I got on board and asked questions such as 'How do you know when the network fails?' I was told 'When the phone begins ringing off the hook,' or 'We go into the computer room and the lights aren't blinking,'" says Lamonica. "It was all pretty rudimentary."

The most glaring problem was a home-grown VPN solution that regularly shut down communications on the DSL and frame relay circuits the company uses to link remote sites. "People would be going about their business, then we'd get calls from job sites that they couldn't connect," recalls Lamonica. "One of us would troubleshoot and find the box that was problematic, but ultimately we'd just have to reboot it."

Then there was the company's antiquated dial-in capability. "When I got here," he says, "we had banks of 28.8Kbit/sec modems. Dial-up was extremely painful to employees on the road, those working from home, and small job sites."

The result wasn't pretty. "We had daily outages lasting for hours, and productivity and connectivity were a real problem, with network uptime at about 80 percent," says Lamonica. "It all made for a rather unstable environment."

As noted, such network outages left onsite construction personnel without access to Prolog, disrupting their ability to track the work of contractors and subcontractors, develop project bids, and manage job costs. According to Lamonica, these network ills contributed to the delay of a number of critical infrastructure projects.

LAYING THE GROUNDWORK

Rudolph and Sletten realized that to keep its reputation as a leading builder, it had to walk the walk from an infrastructure point of view. "Our emphasis is and always will be doing what we do best, which is building buildings," says Lamonica. "And what we want to spend most of our time, energy, and resources on is making sure we do the best possible job we can in building high-tech buildings."

So, as Lamonica puts it, it was clearly time to "drain the swamp"--that is, replace all the company's aging infrastructure devices. He also says he wanted to dump the company's unreliable Sendmail POP3-based e-mail system for Microsoft's Exchange/Outlook tandem, which would provide the company with collaboration capabilities such as calendaring and contact management functions lacking in Sendmail.

"We couldn't do that, however, because we didn't have a stable infrastructure," he says. And the only way to get that was to get a handle on what was working and what wasn't. What the company needed was a network monitoring solution.

GroundWork's open-source Monitor got the call. Several factors were at play here, not the least of which was cost, Lamonica admits.

Because Rudolph and Sletten included Monitor as part of the rebuilding of its infrastructure (more on that shortly), Lamonica says it's difficult to pin an exact number on what the company paid for it. That said, GroundWork's vice president of marketing, Will Winkelstein, estimates Rudolph and Sletten's GroundWork deployment to be in the $50,000 range.

According to Winkelstein, that's typical for a GroundWork installation at a medium-sized enterprise. That figure, which he says is less than 10 percent of what a comparable HP OpenView or BMC Patrol system would cost, includes not only the monitoring software itself, but the planning, integration, and deployment of the system, he says.

Lamonica, who has worked with OpenView and Patrol in the past, says they weren't a fit for Rudolph and Sletten. "In my experience with the bigger tools, they take a long time to deploy, are expensive to deploy, and require a significant amount of resources to maintain," he explains.

OTHER FACTORS

Cost wasn't the only factor, of course. For one thing, Lamonica had used Monitor at a previous employer, so he knew it would be a good fit for a medium-sized business like Rudolph and Sletten.

The fact that GroundWork's staff handled the deployment was another plus. "I didn't have to invest a lot of my resources in building and implementing the tool," he says. "They came in and did it within 60 days, from first discussion to the time we began receiving consistent and regular alarms on our system."

The deployment began with GroundWork engineers sitting down and talking with Rudolph and Sletten technicians about the company's environment and the types of monitoring and surveillance it wanted, says Lamonica. "Then they put together requirements specifications, reviewed it with us, rolled it out, and handed it over to us. It was very straightforward."

The only caveat? "Make sure you know Red Hat Linux 8," the open-source OS that Monitor runs on, says Joe Tan, the senior systems administrator with day-to-day responsibilities for managing Rudolph and Sletten's network.

GroundWork's Monitor is based on Nagios, an open-source application that provides monitoring functionality comparable to that of the large commercial products. Monitor is a composite application that enhances and extends Nagios by incorporating several other open-source software modules, a variety of optional proprietary software elements, as well as GroundWork software, documentation, and configuration guidelines and parameter settings.

Its list of capabilities include a variety of reporting tools, e-mail and pager notification functions, and client plug-ins for multiple platforms, including Windows and Linux. GroundWork distributes most of the Monitor software under the General Public License (GPL), so customers are free to copy and change it.

With Nagios, GroundWork can monitor any device that supports TCP/IP. It uses the standard Internet Control Message Protocol (ICMP) echo request-and-response mechanism to detect a device's status. In addition, it can monitor the services running on a network device via a variety of other methods, says Thomas Stocking, GroundWork's CTO.

For instance, GroundWork can evaluate the response of any port for occurrences such as error messages and "OK" strings to determine the service's status. "If a service doesn't manifest itself on a TCP/IP port, the information can be gathered through agent technology such as SNMP," he says.

GroundWork also includes limited proactive management capabilities. For instance, it ships with an open-source plug-in called NRPE_NT that runs on a Windows server. When a critical service on a production machine shuts down unexpectedly, the plug-in can automatically restart it.

OFF TO WORK

With GroundWork in place, Lamonica and his staff were able to begin rebuilding the company's infrastructure. First to go was the home-grown VPN solution, replaced with one based on a Cisco Systems PIX firewall.

The company rolled out an Exchange/Outlook system next, developing a distributed e-mail environment with Exchange servers at headquarters, as well as at the permanent regional offices. The bank of aging modems was also replaced with new 56Kbit/sec boxes, significantly speeding dial-in for road warriors and telecommuters.

In addition, the company swapped out its old switches and routers for a variety of Cisco products, including Catalyst 4507R switches, PIX 515E/501 firewalls, VPN 3000 concentrators, and 2600, 3470, and 1700 series routers. The new gear has proven extremely stable and provides much greater throughput and security, says Lamonica.

THE PAYOFF

It's all paid off handsomely for the company, according to Lamonica. "We now have far better productivity, with fewer failures, and we can continue to bring in the latest, greatest technology."

Most importantly, the stable networking environment has allowed Rudolph and Sletten to deploy wireless access points from Airespace at each of its regional offices, including construction sites and its Foster City headquarters.

"Wireless is huge," says Lamonica. "It's great for workers who spend a lot of time at job sites. They can fire up a laptop without worrying about rooting around for a Category 5 cable."

Going wireless simplified life for the IT department as well. "Instead of having to pop in more hubs and manage connectivity for more users, if the trailer is wirelessly enabled, we need only one hub at each site, with the wireless access points using one port," says Lamonica.

Wireless technology also eases training. "We have a mobile hands-on training environment consisting of a number of laptops configured for wireless that we can ship to regional offices, and our trainers don't have to figure out cabling or attaching to the network in order to set up a classroom and conduct training," he explains.

The stable network has given Rudolph and Sletten leverage with vendors. "We can now go to DSL or frame relay providers, for instance, and provide hard-core evidence about how poor their performance is," says Lamonica.

And it all goes back to the ability to monitor Rudolph and Sletten's infrastructure. "If we hadn't been able to get a grip on our unstable environment and find out where we needed to plug gaps, we wouldn't have been able to justify this," he says.

Is your enterprise making innovative use of a networking technology or service that you'd like us to write about? Contact Jim Carr, an Aptos, CA-based freelance business and technology writer, at [email protected].