The Worst VoIP Gotchas

Anyone thinking a switch to a voice-over-IP phone system will be smooth and easy should remember Ruth Harenchar's ruby-red nail polish. At the Hobart West Group, where Harenchar is CIO, the company's VoIP project required tough decisions, like whether to spend money training existing IT staff or hire expensive consultants. It meant learning to live without certain common telecom features in order to get the savings the company wanted. And it involved helping employees through the culture shock of replacing the familiar phones they'd been using for years. That's where the red nail polish comes in.

Businesses love VoIP's promised benefits, but too many underestimate just how arduous implementations can be. Hobart West, a legal-services and commercial-staffing firm, bought Cisco IP phones with gray "hold" buttons; administrative staffers, used to red hold buttons, were constantly hitting the wrong key and dropping calls. Harenchar's low-tech answer, implemented in 60 offices nationwide: Paint red nail polish on handset hold buttons.

If only the rest of the challenges had such simple solutions. VoIP has hit the mainstream, but when it comes to installing the technology on a large scale, there's hard work ahead. "There's a reason it's called the bleeding edge," Harenchar says. "It's painful."

For example, although caller ID is a staple at many companies, Hobart West's VoIP system won't tell employees who's calling. A special kind of telecom circuit, called primary-rate interface ISDN, is necessary for caller ID on VoIP, as is a module for the call-management system. Together, they were too expensive.

And Harenchar discovered that simple mistakes can cause big problems. A recently hired network engineer clicked on the wrong item in the call-manager software and accidentally routed all calls through Hobart West's Houston office, overloading the VoIP network. "He made a decision he thought was simple and brought down the whole system," Harenchar says. "There are so many little places in the call-management system that have to be programmed exactly, perfectly right for even the easiest things to work the way they did." So why are companies putting themselves through all this aggravation? Foremost, they're finding that VoIP, which chops up voice conversations into data packets and routes them over a data network or the Internet, can dramatically reduce the cost of phone service. Most companies pay a flat monthly rate for an Internet connection, so costs don't increase with call volume. And when voice calls are just another application on the network, there are opportunities for integrating voice into business processes. Hobart West anticipates a day when customers' files automatically pop up in employees' Web browsers when calls come in. "It's worth it in the long run," says Harenchar, whose company finished its VoIP deployment in July. "It opens up the ability to do customer-workflow integration in ways it's never been done before."

That's a familiar sentiment among businesses betting on VoIP, which include Ford Motor Co. and Bank of America Corp. Companies upgrading their phone systems face the choice of sinking money into yesterday's technology--traditional PBX products--or investing in VoIP, the platform for the future. And they're picking the future: VoIP line shipments increased 52% last year, while PBX-based line shipments were down 29%, according to Infonetics Research. Spending on VoIP will jump from $1.2 billion last year to more than $23 billion in 2009, Infonetics predicts. Research firm Frost & Sullivan forecasts the number of hosted IP-telephony lines will increase from 292,000 last year to 9.7 million by 2010. This means thousands more businesses are about to come face to face with VoIP "gotchas."

Meeting Demand

Equipment vendors and service providers are scrambling to accommodate the growing interest in VoIP. Network Instruments LLC last week released a new version of its network-management software that lets administrators see VoIP traffic statistics and more than 20 metrics that track call quality. Comcast Corp. and Cox Communications Inc., two cable-TV companies, recently started offering VoIP services. Microsoft two weeks ago bought its second VoIP company, media-streams.com ag, a Swiss applications maker. And just last week, a group of VoIP champions, including EarthLink, Google, Pulver.com, Skype Technologies, and Sonus Networks, banded together to promote the technology through the Internet Voice Campaign.

Companies often underestimate how big a change they're in for when installing a new phone system and don't plan thoroughly enough. "They're assuming that, 'OK, it's a different protocol, but it's not very different from the type of system we have today,'" says Lisa Pierce, a VP at Forrester Research. Even basic steps like taking inventory and network testing have their pitfalls. Inventory requires sorting out if existing telephony gear has any value, whether the company needs more or different switches, and what the strategy should be for software additions and upgrades. But there tends to be a lot of confusion around what can and can't stay, warns Tom Thomas, a senior solutions architect at consulting firm Dimension Data. Also, companies need to assess their networks for available bandwidth and quality of service, taking into account the metrics that come preinstalled on networking equipment and on network-management tools like those offered by Network Instruments, Network General, and others, to make sure the networks can handle the additional load of delay-sensitive voice traffic. And not all network-testing tools work for VoIP tasks, some administrators find. The checklist of things to consider is long. VoIP signals often are compressed so they take up less bandwidth, but too much compression reduces voice quality. A network administrator needs to consider whether to go with G711 compression or G729, which applies higher compression and consumes less bandwidth but offers poorer voice quality. Vendors' descriptions of technologies and features can vary widely, too, creating confusion among customers.

Snap, Crackle, Pop

One gotcha is to assume that any new problem is the result of the VoIP network. In some cases, the fault might lie in the connection between the VoIP network and the public switched-telephone network. Testing before fixing is key to discovering the root of a problem. Advocate Health Care, a group of hospitals and clinics throughout Illinois, started switching to VoIP in late 2002 to cut the phone bill and the cost of employee support. The company has reached a result it likes: Now, when an employee changes offices, the support staff doesn't have to go into wiring closets and mess with phone connections. Employees take their phones with them, because a phone's identity is managed on a server, not in a wiring closet.

But Advocate Health Care had to work out some glitches first. When Gary Horn, enterprise architecture and network security director, and his team began testing the system, clicking, popping, and echoing made calls unintelligible, and the echo was so bad callers couldn't understand what the other person was saying. "It would be like the worst type of cellular call you've ever heard," he says.

Horn assumed the problem happened on all the company's phone conversations, but he discovered that only calls that crossed over to the public phone network had quality problems; intraoffice calls were fine. After a few days of testing and tweaking scenarios, his team found they needed to install echo cancelers where the public phone network met the VoIP network. After installing the new hardware, Advocate spent a few days testing the network with network-management tools during the day and reconfiguring and testing the echo cancelers' timing elements in the evenings.

The quality of network-testing tools, Horn found, can make a big difference: Few are designed solely for IP telephony. Testing is important not only to head off problems, but also to teach the networking and telecom teams more about VoIP. "We learned a lot from that first site," he says. "There's a definite learning curve."

And there's more than just technology to learn. When something as familiar as the phone changes, expect negative reactions. "It was a 'You've changed my phone' culture shock," Horn says. "It's almost sort of a religious thing." The shock wore off with time, and the people eventually got used to the new phones.

Another challenge is devising a deployment plan that makes the most sense for your company. When insurance company Allstate Corp. began deploying VoIP two years ago, telecom configurations varied widely among its thousands of offices, as did the age of the equipment. Brandi Landreth, senior manager for network and voice solutions, decided the wisest choice was to attack the challenge piece by piece, focusing on offices that had the oldest equipment, such as a tech-support center in Northern Ireland. "Instead of looking at what we have to replace for the entire enterprise, we do a few offices each year," she says.

Who's In Charge?

For some companies, the biggest gotcha is figuring out who should be in charge of the project. VoIP crosses the boundaries of traditional telecom and networking, so including technical staffers from both sides on the deployment team can be a smart move. But don't assume people in one discipline know what the others are doing. Allstate didn't create a blended team when it started its first test site. As a result, it was like two separate projects run by two separate groups. On one key question--would the phones be powered by electricity from a wall socket or by power-over-Ethernet technology?--the teams didn't communicate, so they didn't have the correct parts ordered, setting back the entire effort. Allstate now uses a VoIP deployment team with staffers from both departments and is cross-training those people, teaching telecom people more about networking and helping the network team learn more about telecom and the routing of voice calls. Landreth thinks certain tasks such as advanced call routing are best handled by the telecom staff and network-specific tasks should go to the network team. But she's still working out the right formula. "Talk to me in a year, and I may have a different answer," she says.

Because the technology is new, and its use is growing so fast, people with talent and experience doing VoIP implementations are hard to find--and come at a premium. Companies have two options: train or hire. Both can be more expensive than expected. "Customers often assume that there's more staff out there that knows the infrastructure than is true," Forrester's Pierce says. And while new employees may understand VoIP, they don't know much about a specific company's operations.

In-house training always sounds like the logical option, but it's both costly and slow. This isn't a couple-days-in-the-training-room kind of technology. "There's a very steep learning curve," Hobart West's Harenchar says. Based on experience and projections, she estimates it can take $15,000 of classes over nine months, plus 12 to 15 months of on-the-job experience, to fully train a network administrator to manage an enterprise VoIP system. On the other hand, outsourcing or hiring can be even more expensive. Hiring someone with VoIP expertise would cost around $40,000 above current pay scales, Harenchar says. She's doing a mix of both training and hiring.

Business-technology managers considering VoIP need to consider a host of fundamental questions that can turn into gotchas if not carefully thought through. And not all of them can be decided by the IT team alone. Will you sacrifice sound quality or features to save money? Does your network have enough bandwidth to handle voice calls and provide high-quality sound? Can your network give voice traffic priority over E-mail and Web browsing, which aren't as affected by network delays? Is it designed to transmit VoIP packets efficiently, with low latency, or does it need to be restructured? How will you provide backup power to keep the phone system operating in case of a power failure? Do you need extra outside lines as part of that backup plan?

Businesses can't assume that their current phone systems will serve as a model for a VoIP system. "It's very difficult to do a one-for-one match," Advocate Health Care's Horn says. "To say, 'If we have a certain number of phones, we need this much bandwidth,' it just doesn't work that way." Horn allocates roughly 30% of a 1.5-Mbps T1-connection in small clinics to VoIP and about a full T1 at Advanced's hospitals and larger offices. After the systems are in place, Advanced uses tools such as metrics programs built into routers and network-monitoring tools to make sure bandwidth is allocated properly. Details Are Important

Sweating the small stuff is important, as seemingly minute technological problems can cause lots of trouble when they're tied to something as indispensible as the phone. "Things as simple as headsets--you think existing ones are going to work, but they're older or not compatible," Allstate's Landreth says. Despite the fact that many of the company's headsets were less than 3 years old and had the right kind of jacks, they didn't work with the company's IP phones at the test site. The headsets were too sensitive and picked up background noise. Trial and error was the solution. "We bought a bunch of headsets and said, 'OK, does this one work and are we hearing the same echoes and feedback?'" she says.

When Advocate Health Care began rolling out VoIP, the network team sometimes felt as though it knew VoIP technology better than the vendors, since hands-on experience with the technology was new to both of them. To a degree, this challenge still exists. "The outside companies are still learning the technology," Horn says.

The right vendor, the proper network tools, a well-trained IT staff, an efficient network architecture--all are key requirements for deploying VoIP while minimizing the disruption to a company's workforce and business operations. In the end, careful planning and a thoughtful implementation program are the way to navigate the VoIP minefield and come out unscathed. That, and a little ruby-red nail polish.

Illustration by Hal Mayforth

Continue to the sidebar:

On The Front Lines: A VoIP Survival Story