What’s Causing Cloud Outages? A Network Managers’ Guide

As enterprises rely more and more on cloud services to meet their network infrastructure, compute, data storage, and security needs, cloud computing outages have a significant impact on operations.

Many believe (or hope?) that moving services to the cloud would eliminate some issues. After all, you would assume cloud providers make use of the latest technologies, have staff with expertise in these technologies, and build in lots of redundancy.

Unfortunately, what we find is that cloud outages have a lot in common with their data center outage counterparts. Many occur due to human error, power outages, malicious acts, Mother Nature, or plain bad luck. 

cloud-1-2J0NPBA.jpg

What’s causing cloud outages?

There are several common culprits causing cloud outages. Over the last few years, we have seen examples of each. All have had a significant impact on the enterprises using the services. Here are some of the top problems that keep reoccurring.

Configuration mistakes

We’re in the age of graphical user interfaces (GUIs) and automation. Yet, many critical IT chores like deploying a new server, provisioning storage for an application, or setting up new router tables are done manually via command line interfaces (CLIs). As one would expect, that can lead to configuration mistakes.

That is often the case with cloud outages. One such mistake caused a six-hour outage of Facebook, Instagram, Messenger, Whatsapp, and OculusVR due to a routing protocol configuration issue. As we wrote at that time: “The outage was the result of a misconfiguration of Facebook’s server computers, preventing external computers and mobile devices from connecting to the Domain Name System (DNS) and finding Facebook, Instagram, and Whatsapp.”

Essentially, BGP routers were unrecognized, preventing traffic destined for Facebook networks from being routed properly. Resolution of the problem was more challenging than normal because not only was communication between routers interrupted, but so too, were DNS traffic and all applications.

The problem here was that everything ran over the same network. As a result, IT staff could not remotely correct the problem because they could not access the impacted systems. And making matters worse, IT staff were locked out of facilities because their access control system also ran over the same network.

Unexpected or unknown system behavior

Obviously, an incorrect configuration change can cause outages. But in one recent case, a correct change still led to a major outage. The reason, unbeknownst at that time to the IT staff, was that the same command operates differently on routers from different vendors. That was the case in an extensive Microsoft outage.

In that event, a network engineer was performing relatively common tasks to add routers and capacity to the company’s Wide-Area Network. The work involved modifying the IP address for the new routers and integrating them into the IGP (Interior Gateway Protocol, a protocol used for connecting all the routers within Microsoft’s WAN) and BGP (Border Gateway Protocol, a protocol used for distributing Internet routing information into Microsoft’s WAN) routing domains.

As we wrote at that time: “The change added a command to purge the IGP database – however, Microsoft noted that the command operated differently for different router manufacturers. Routers from two manufacturers limit execution to the local router, while those from a third manufacturer execute the change across all IGP joined routers, ordering them all to recompute their IGP topology databases.” Due to the scale of the network, that work took everything offline for about two and a half hours while the routing tables were recalculated and updated.

Power issues

One of the main factors when selecting sites for cloud data centers is the availability of abundant and low-cost electricity. Why? Data centers of any type, whether in the enterprise or for a cloud provider, consume 10 to 50 times more electricity per square foot than a typical commercial building, according to the U.S. Department of Energy. As a result, the major cloud providers have clustered their data centers in regions like the Pacific Northwest (known for its low-cost hydroelectric power), Arizona, Virginia, and other comparable places.

Even with such an abundance of power, power-related outages account for 43% of all data center outages, according to the Uptime Institute. Naturally, cloud data centers, like their enterprise data center counterparts, have backup power capabilities in the event of an outage to the electrical network. Unfortunately, that may not be enough. One of the longest cloud service disruptions, a 12-hour outage of Microsoft's Virginia data center, was due to a problem with a provider's redundant power system.

East Coast companies served by that data center were unable to access any of their Microsoft services. As we noted in a roundup article about cloud outages, the source of the problem was that the facility’s redundant power system created unexpected electrical transients. Air handling units designed to cool the center detected the oscillation and shut themselves down to prevent damage. Once the source of the problem was identified, the units had to be manually reset to restore services at the facility.

Physical damage

In the days and years after the breakup of AT&T, when carriers were expanding their networks, there were numerous stories of major outages due to backhoes cutting cables. Those types of outages have greatly diminished in recent years due to a greater awareness of the issue, better mapping of underground cables, and more.

Lately, it has been Mother Nature doing the damage. Last year, a volcanic eruption took out the only connection between Tonga and the outside world. The blast cut the submarine cable linking the island with Fiji.

The story brought attention to the issue, noting that 95 percent of intercontinental global data traffic travels over undersea cables that run across the ocean floor. And worse…many of the most concentrated terminations of such cables are in areas subject to earthquakes, volcanic eruptions, and flooding.

That latter point was an issue in 2012 when post-tropical cyclone Sandy’s landfall points and associated tidal surges along the New York and New Jersey coasts aligned with the termination points of 25 submarine cable systems. The storm cut 11 of the 12 high-capacity cables that connected the US and Europe.

Accidents and malicious actions

As noted, the fragility of the network of undersea cables is of great concern. Beyond the acts of nature mentioned above, the cables and especially concentrated termination points are ripe for terrorist or nation-led attacks.

But a more common problem is accidental cable cuts at sea. Ships, particularly fishing vessels, will anchor at sea during severe weather. In some cases, the ships are displaced by strong winds or currents. That drags their anchors across the ocean floor resulting in damage to a cable.

Secondary or unintended impacts

Most cloud service outages directly impact access to an application or suite of applications or services. For example, a Microsoft center outage might mean enterprises cannot access their Outlook, Sharepoint, and Teams apps. Or a Facebook outage also cuts off access to Instagram, Messenger, and Whatsapp.

But things are getting more complicated as many cloud services are now dependent on other services. That was the case when an Amazon outage inhibited and interfered with the invocation of its AWS Lambda function. As we wrote at the time, that was a major problem because many AWS services and enterprises are making use of AWS Lambda's serverless capabilities. The problems with Lambda cascaded, taking more than 100 AWS services offline.

cloud-2-2G83Y0T.jpg

How to protect against cloud outages

There are several ways for cloud service providers to minimize the chances of an outage and for enterprises to minimize their impact.

Cloud providers are taking a number of steps. Many are trying to improve outage detection. Most are clarifying methods, developing best practices, and implementing standard operating procedures (SOPs) for things like router configuration changes or adding equipment to scale their services.

More advanced providers are continuously auditing those SOPs. They want to be sure they are being carried out and that the procedures are still correct, given the dynamic nature of cloud environments.

Additionally, the larger cloud providers make use of redundant everything. They use multiple circuits and cables to carry traffic between centers and for users to reach their centers. They have hot backups standing by to take over and run applications and services if there is an outage. Additionally, they make use of different power supplies, including traditional line-delivered power, on-site generation, on-site uninterruptible power systems, etc.

From an enterprise perspective, though, network managers’ defenses against cloud outages remain limited.

Enterprise network managers’ first step is to see what their cloud service providers are doing in these areas to make their services resilient. Other tactics to take include:

cloud-3-2K4P331.jpg

Cloud outages key headlines