Gartner: Potential For Network Failures Looms Large

Gartner analysts are sounding the alarm about the potential for network outages due to defective network and security products containing faulty Atom C2000 chips. The flawed component issue that affected a wide swath of products from Cisco and other networking vendors surfaced earlier this year, but many organizations still don't know about it, they said in a new report.

Cisco and other vendors began warning of defective products in February, advising of the risk of complete equipment failure due to a buggy clock signal component. While the component wasn't named in the notices, it's widely believed to a documented issue in the Intel Atom C2000 chip line.

Vendor response to the problem has been inconsistent, according to Gartner, which interviewed more than 20 networking and security vendors. "We believe that, as of early May 2017, many organizations that are at risk are not aware of this industry-wide issue," Gartner VP and Distinguished Analyst Mark Fabbi and Research VP Greg Young wrote in a report.

Indeed, the risk is widespread: Gartner estimates 150,000 to 500,000 units deployed around the world are impacted, including branch office routers, wireless LAN controllers, branch office security and switching appliances, data center leaf-and-spine switches, and core routers.

In a blog post, Fabbi called the situation a ticking time bomb. Gartner advises organizations to take immediate action.

"Products that fail cannot be rebooted or recovered and replacement is the only remedy. Most impacted vendors are offering replacement programs for products under service contracts to avoid future failures, however we believe some vendors are not owning up to a potential problem within their offerings," he wrote.

"The big issue is that few vendors have proactively notified impacted customers, so many enterprises are unaware of this issue that can have a significant impact on network uptime (which is hard to maintain as-is). With the potential for network failures impacting business critical processes, it’s time to take notice of this industry-wide problem."

fuse.jpg

In their report published May 15, Fabbi and Young provide specific recommendations on steps organizations should take to address the issue. The recommendations include reviewing network and security inventory to determine exposure to the defective component, prioritizing replacement of impacted products based on business risk, and reassessing replacement strategies to ensure alignment with business requirements.

Appliances in remote offices will likely represent the greatest likelihood of downtime and cost the most to replace, according to Gartner, which estimates the cost of replacing branch office devices as ranging from $500 to $1,500 per device. The costs include travel time, parts, and potential overtime, none of which will be paid for by the vendor.

"For a business with 500 remote locations, this amounts to $250,000 to $750,000 of unbudgeted, out-of-pocket expense," the analysts wrote.

Some organizations are using the forced disruption as a reason to consider new technologies such as SD-WAN, they noted.

The report includes a list of vendors and whether their products are impacted by the faulty component, but the analysts said it shouldn't be considered definitive. "We have reason to believe that additional networking and security vendors may be impacted by this issue," they said.

In addition to Cisco, impacted vendors include: Dell; Extreme Networks; Fortinet; HPE; Juniper Networks, and VeloCloud. Some vendors such as Citrix, Extreme and VeloCloud have proactively notified affected customers and replaced defective units, Gartner said.

Most vendors have said that only products covered under active warranty or support contracts will be covered, according to the report. "We do not believe any vendor is treating this issue as a product recall, which would have allowed for replacement of products out of warranty," Fabbi and Young wrote.

Many of the details about the outage risk from the faulty products, such as how failure rates increase over time, have not been widely reported, according to Gartner. The lack of details likely stem from supplier nondisclosure agreements, the analysts wrote.