That's how many approvals were required to make a single change on a production firewall at the global transportation company where I was a technical architect. Fifteen separate approvals, all captured on paper and ultimately dropped into a file cabinet somewhere, likely never to be seen again. The process was long, painful and, worst of all, drew the time to deploy a new application out much longer than the business would have liked.
Most IT shops have similar polices that, as a whole, have spawned what CIOs and business leadership call the "IT bottleneck," a phenomenon that causes undesirable delays in getting an application "to market." Whether that market is customers, partners or employees doesn't really matter. The consensus is that IT is standing in the way of getting there.
That consensus may be right.
Now, the complexity of today's data centers is such that without processes, applications would never make it "to production." A vast number of network and application components require rule and policy configuration changes to support any new system. And many of the data center components on which these rule and policy changes must be made are not just critical but super-, mega-, ultra-critical. The kind of critical that means a wrong configuration could bring the entire business to a screeching halt.
Such risk is, understandably, approached with caution and due process. The problem is that processes tend to grow organically along with IT. As more systems -- and thus operators and administrators -- are added, procedures expand to include them. No one wants to be the one to cut Joe out of the loop, even if the loop has grown into a wildly inefficient tangle. The realignment efforts that happen from time to time within IT tend to only add to the "cc" list.
This has always been a problem, but it's about to get worse because the value of cloud, DevOps and even software-defined networking is largely predicated on the premise that automation will improve IT responsiveness and thus offer a faster time to market. Whether you're talking automating scaling processes, enabling continuous deployment or alleviating the tedious tasks of switch configuration, the most-hyped technologies of the past five years have all had at their core one simple premise: The system can do it faster than you can.
That is, of course, true. Automation technology can configure components faster than people can. But configuration usually happens after the approval process, which is rarely included in or supported by those shiny IT automation technologies. Go ahead, document the time it takes to consider, approve and implement a typical change request. I guarantee that the consideration and approval parts of that process comprise the bulk of the time required to move an application into production.
If IT is going to break its bottleneck problem, we must focus on process improvement. That means hacking away at inefficiency. Here are three things to consider.
First, ask: Is it a needed authorization, or just CYA or FYI? Many of the approvals required to move an application into production are not approvals per se, but notifications. Some involve groups that may have changed charters over the years and have no reason to be in the approval process today. Others loop in individuals responsible for configuring component X even though X has long since been replaced by some other system or service. Some are required to route applications through systems that, while still in place, serve no purpose for that application.
Second, realize that an audit can pay benefits beyond addressing inefficient or broken processes. For example, a top benefit of automation is the opportunity for IT to align processes with business goals. Take the opportunity to consider everything from organizational structure to responsibilities to the systems that are providing critical application services. Ultimately this examination and reconsideration can result in streamlined processes that not only speed the time to market but actually reduce the risks of misconfiguration or conflicting policies.
Finally, a process audit does not need to be painful. Everyone scatters when you say the word "audit," so call it something else: "We need to assess the processes involved with pushing applications into production so we can take better advantage of automation and orchestration." Or maybe: "By carefully considering the systems and people involved in change processes, we can eliminate redundancy and enable continuous deployment efforts."
Even simply aligning the timing of change reviews to match IT deployment windows can dramatically improve the business' perception of IT.
Bottom line, process audits are critical to efficiency. The more rapidly the business is growing, the more frequently you need to make sure the processes that direct change through the system do not become -- or remain -- the source of the dreaded IT bottleneck.