Network Downtime: The Costly Fallout

Human error when making changes to network cores chipped away at companies' revenue by disrupting everything from employee productivity to supply chain processes during 2013, according to a recent survey of IT professionals.

Avaya polled 210 IT pros at companies in the U.S., Canada and the U.K. with at least 250 employees, and 82% of the respondents said they'd experienced at least some network downtime as the result of human error made while configuring changes to the network core.

Of those who had experienced core errors, 80% reported having lost revenue as a result of such outages, to the tune of an average of $140,000 per incident. Hardest hit were respondents from the financial services industry, who reported having lost an average of $540,358 for each incident.

Those losses were triggered by a variety of disruptions, as 94% of respondents reported suffering some kind of consequence of downtime resulting from human-induced errors during core changes. Nearly half (49%) said employee productivity was hampered, 40% said IT projects were interrupted or delayed, and 30% reported supply chain disruptions.

"Any outage or disruption in the core is sure to affect the rest of the network," Brad Casemore, research director at IDC, said via email. "The ramifications from downtime in the core are felt everywhere."

Given the wide-ranging impact resulting from core errors, it's not surprising that one in five respondents to Avaya's survey said their companies had fired IT staff as a direct result of such outages.

Casemore said that whether a company fires a network professional for such an error depends on its corporate culture, the employee's record, the enormity of the error, and whether there were clearly prescribed policies and procedures to prevent such an error.

"It's certainly possible that errors of the sort covered by this survey could be considered sufficiently egregious for heads to roll, figuratively speaking," Casemore added. "I've known of networking professionals who've been fired for less, and of those who've survived errors that might have been considered firing offenses elsewhere."

[Read how best practices can go a long way toward preventing unplanned downtime caused in "How To Avoid Network Outages: Go Back To Basics."]

The survey findings underscore a growing need to eliminate human error by automating the management of network cores. In fact, Bob Laliberte, senior analyst at Enterprise Strategy Group, said via email that network management ranked second behind network security on the list of network investment priorities in a recent IT spending survey published by ESG.

"Not having complete visibility into the environment or properly understanding the impact of a change before making it can lead to unintended consequences," Laliberte said. As a result, "the ability to effectively manage large and complex network environments is a priority for most organizations."

Casemore said many companies are investigating SDN, network virtualization and other approaches as they seek to automate as many elements of their IT infrastructures as possible. "Automation is a touchstone, from provisioning through to orchestration," he said.