Thirty years ago, I started my career as a network engineer with the U.S. Air Force. Many things have changed since then. For starters, back then, if we could deliver a T1’s worth of bandwidth to your office, that was considered a lightning-fast network connection. By comparison, the Google Fiber connection that many people have delivered to their homes nowadays is roughly 650 times faster. We spent a lot of time focused on ensuring the continuity of network operations because infrastructure back then was undeniably less reliable. From frame-relay DLCI misconfigurations to broadcast storms to spanning tree bugs, the networks of yesteryear were a lot harder to manage.
In order to ensure that the network was stable and secure, one of the tactics that many organizations employed was to limit change as much as possible.
The recent catastrophic disruption at Southwest airlines highlights the fact that the ‘if it ain’t broke, don’t fix it’ mindset is still prevalent within many organizations. Now that the deficiencies in their systems have been laid bare, it is safe to assume this mindset will be changing quickly -- and not just at Southwest.
Did you know that 87% of organizations have experienced an attempted exploit of an already-known, existing vulnerability? That statistic comes from Check Point Software’s 2022 Security Report. In my expert opinion, close to 100 percent of organizations that have been connected to the internet for more than a hot minute have been attacked using a well-known and yet existing, i.e., present in their environment vulnerability. The 13 percent gap is likely made up of organizations that have been attacked but don’t know that they have the vulnerability even though it’s been reported and is known within the industry. Also within that number are companies that don’t realize that they’ve been attacked.
According to the U.S. Department of Justice, only one in seven cyber crimes are reported, which means that over 85% of cyber crimes are left hidden within an organization. So, why is that? If the cyber world is really that dangerous, why aren't organizations more focused on continuously improving their network security? Quite frankly, it's because there aren't enough network engineers with cybersecurity expertise available to make the number of changes that modern organizations require.
I speak to network operations leaders every week, and they all say the same thing - they're completely snowed under with work, and there's no end in sight. Although it may be hard to believe, many companies still rely on manual efforts or homegrown scripts that are error-prone, incomplete, and resource intensive to automate their network and security device changes. Even the easy stuff that keeps the network secure, like backups, upgrades, and configuration grooming is going undone.
Many teams simply haven't had the time to form and implement a network automation strategy, and that's the only way out of the current situation. The number of changes that standard business operations alone require outpaces most organizations' ability to staff. That's without even considering that modern network engineers are, in effect, warfighters now responsible for performing the updates required to strengthen their perimeters and defend their organizations from malicious actors and nation-states.
And while there are many unsung heroes on network operations teams that are desperately trying to keep everything up to date, it's just not humanly possible to do it manually -- or even using homegrown scripts -- in this age of complexity and constant cyber attacks.
Improving network operations with automation
Organizations faced with the undeniable fact that their network teams are overworked and understaffed must look to network automation as a solution. It’s imperative that a company’s network automation strategy deals with the following items within the first phase of deployment:
- Reliable backups with the ability to quickly recover from an outage, regardless of who is on call that day
- Rapid discovery and remediation of risks from devices with OS levels containing known CVEs
- Continuous detection and grooming of configurations that fail compliance checks for CIS or other relevant standards for their location and industry
- A trustworthy source of truth for network configurations and risk states
Once these items are reliably automated, you can move on to the next phase. Key focus areas for continued automation should include:
- End-to-end automation of ITSM workflows involving netops tasks
- Continuous deployment of new packet signatures for network security devices
- Automated routing changes when needed for stability, cost management, or security
- Network-level Privileged Access Management to secure and record manual changes
A renewed emphasis on these routine but critical tasks and an automation strategy that ensures consistent execution are things every network team can do to strengthen the security of their organization.
Josh Stephens is the Chief Technology Officer at BackBox.