The headlines drew us in for days, with stories of how millions of Texans found themselves without light, heat, or power. And even as the state recovers, as life returns to normal and power has been restored, the economic impact is just beginning to be felt; the Texas Standard quotes an analyst as saying the total damage to the state will easily measure in the billions of dollars.
What happened in Texas is not an isolated incident, to be sure. The International Disaster Database (see chart) shows a three-fold increase in natural disasters around the world between 1970 and now.
DR101, a basic disaster recovery course, teaches you to prepare for the worst at any time…because you never know when it’ll hit. It could be a winter storm. Or a hurricane. Or a tornado. Or the direct actions of people, such as a “cyberransom” attack…even some guy named BIFF (Backhoe Induced Fiber Failure).
Lots of companies still run their own data centers, but too many of them aren’t as prepared to recover from a disaster as they should be or need to be. Business continuity experts routinely speak in terms of an RPO (Recovery Point Objective) or RTO (Recovery Time Objective) when things go wrong: companies want to know how quickly they can be back up and running and how much of their data do they need to be able to work effectively. These firms are leaving too much up to chance, overlooking the elements they need to ensure uptime.
Avoiding client downtime is our job. We are data center operators, and it is in our charter to be well prepared for these types of events. We routinely test the systems, processes, and procedures at our data centers around the world, across any number of strict operational metrics. For reference, here are some (but not all) of the things we consider…points you may want to consider if you’re running your own data center:
- Make sure the exterior of your data center is clean to avoid any debris coming loose.
- Check your oil and fuel levels; how many days of fuel do you have on hand, and how many hours of uptime can it be used to provide, if needed?
- Check your emergency water supply for HVAC purposes.
- Check to make sure your UPS systems are fully functional.
- Make sure you have enough food, water, even cots on hand if your staff is required to stay on-site for several days.
- Test your emergency preparedness plan regularly.
- Ensure you have an emergency phone list in place, making sure that all of the contacts and numbers are current.
Meanwhile, many other companies have decided to move to a hybrid IT approach, where they store some of their data in the cloud, with more sensitive data being stored offsite, within a third-party colocation data center. We agree with this approach and think it is the right one. This enables companies to move away from the functions required to maintain operations at their own data center. However, it does not mean they can simply ignore all of the operations that go into accessing and protecting their corporate-critical data, applications, and workloads. Here are questions that a company should ask of a potential colo firm about how they prepare for any kind of disaster:
- Do you have an emergency plan in place to proactively prepare for incidents?
- How often do you test your emergency plans to ensure they're fully functional?
- Do you have adequate stocks of fuel in place to run your centers if needed? How many days, and what plans are in place to secure additional fuel?
- Do you have interconnectivity between your data centers?
- Do you recommend that I locate my information across multiple data centers so that if a disaster occurs, I can access it readily?
- If I have a problem, how can I quickly contact you?
If you haven’t thought through all these questions, and if you haven’t gotten the answers you need, start now by taking a proactive approach in how you will deal with these issues. “Lessons learned" and your ability to get the attention of the folks in your C-suite (not to mention their purse strings) may have a shorter shelf life than you think. And if the Texas storms have taught us nothing else, again, it's that disasters have a way of striking, regardless of whether we're ready for them or not.
Bill Leedecke is EVP, Operations at Evoque Data Center Solutions.