• 07/09/2013
    3:03 PM
  • Rating: 
    0 votes
    Vote up!
    Vote down!

To Break The IT Bottleneck, Stop Covering Your You-Know-What

Slicing through layers of bureaucracy is a must for automation. It's time to see who you can cut out of the loop.
Fifteen signatures.

That's how many approvals were required to make a single change on a production firewall at the global transportation company where I was a technical architect. Fifteen separate approvals, all captured on paper and ultimately dropped into a file cabinet somewhere, likely never to be seen again. The process was long, painful and, worst of all, drew the time to deploy a new application out much longer than the business would have liked.

Most IT shops have similar polices that, as a whole, have spawned what CIOs and business leadership call the "IT bottleneck," a phenomenon that causes undesirable delays in getting an application "to market." Whether that market is customers, partners or employees doesn't really matter. The consensus is that IT is standing in the way of getting there.

That consensus may be right.

Now, the complexity of today's data centers is such that without processes, applications would never make it "to production." A vast number of network and application components require rule and policy configuration changes to support any new system. And many of the data center components on which these rule and policy changes must be made are not just critical but super-, mega-, ultra-critical. The kind of critical that means a wrong configuration could bring the entire business to a screeching halt.

Such risk is, understandably, approached with caution and due process. The problem is that processes tend to grow organically along with IT. As more systems -- and thus operators and administrators -- are added, procedures expand to include them. No one wants to be the one to cut Joe out of the loop, even if the loop has grown into a wildly inefficient tangle. The realignment efforts that happen from time to time within IT tend to only add to the "cc" list.

This has always been a problem, but it's about to get worse because the value of cloud, DevOps and even software-defined networking is largely predicated on the premise that automation will improve IT responsiveness and thus offer a faster time to market. Whether you're talking automating scaling processes, enabling continuous deployment or alleviating the tedious tasks of switch configuration, the most-hyped technologies of the past five years have all had at their core one simple premise: The system can do it faster than you can.

That is, of course, true. Automation technology can configure components faster than people can. But configuration usually happens after the approval process, which is rarely included in or supported by those shiny IT automation technologies. Go ahead, document the time it takes to consider, approve and implement a typical change request. I guarantee that the consideration and approval parts of that process comprise the bulk of the time required to move an application into production.

If IT is going to break its bottleneck problem, we must focus on process improvement. That means hacking away at inefficiency. Here are three things to consider.

First, ask: Is it a needed authorization, or just CYA or FYI? Many of the approvals required to move an application into production are not approvals per se, but notifications. Some involve groups that may have changed charters over the years and have no reason to be in the approval process today. Others loop in individuals responsible for configuring component X even though X has long since been replaced by some other system or service. Some are required to route applications through systems that, while still in place, serve no purpose for that application.

Second, realize that an audit can pay benefits beyond addressing inefficient or broken processes. For example, a top benefit of automation is the opportunity for IT to align processes with business goals. Take the opportunity to consider everything from organizational structure to responsibilities to the systems that are providing critical application services. Ultimately this examination and reconsideration can result in streamlined processes that not only speed the time to market but actually reduce the risks of misconfiguration or conflicting policies.

Finally, a process audit does not need to be painful. Everyone scatters when you say the word "audit," so call it something else: "We need to assess the processes involved with pushing applications into production so we can take better advantage of automation and orchestration." Or maybe: "By carefully considering the systems and people involved in change processes, we can eliminate redundancy and enable continuous deployment efforts."

Even simply aligning the timing of change reviews to match IT deployment windows can dramatically improve the business' perception of IT.

Bottom line, process audits are critical to efficiency. The more rapidly the business is growing, the more frequently you need to make sure the processes that direct change through the system do not become -- or remain -- the source of the dreaded IT bottleneck.


re: To Break The IT Bottleneck, Stop Covering Your You-Know-What

CYA has been a long-cherished strategy for some people. Does anyone have advice to share on how to manage the culture change as your shop becomes more and more automated?

Laurianne McLaughlin

re: To Break The IT Bottleneck, Stop Covering Your You-Know-What

Laurianne, can you clarify your question? Do you mean that, as more systems become automated and people are "reskilled" (laid off perhaps?), how do you keep the right people plugged in?
At my company (large insurance org), the rule of thumb is that it takes 1 month to get a non-emergency change thru all of the change control committees. 1 month. There is this notion that somehow quality is enhanced as more and more people -- who do not have the slightest idea of what the change is about -- review the change for approval. I don't see the benefit.

re: To Break The IT Bottleneck, Stop Covering Your You-Know-What

I wasn't thinking of layoffs so much as IT process becoming more streamlined thanks to factors like virtualization and public cloud services. While you might have needed weeks and 3 signatures to spin up a new test environment in the past, you may need only hours and one signature now. That helps IT get to yes faster for people on the business side. A month for non-emergency change? Yikes. Must create friction when the business side is in a rush.

re: To Break The IT Bottleneck, Stop Covering Your You-Know-What

It's brutal. IT imposes this 1 month process onto itself. Then IT wonders why things take so long to develop and make production. DUH.

re: To Break The IT Bottleneck, Stop Covering Your You-Know-What

Govt apparently isn't alone in strangling IT in red tape

re: To Break The IT Bottleneck, Stop Covering Your You-Know-What

What this article does not take into account is that the CYA is usually done because top level management does have the balls to say that IT is correct in what it's doing. A culture shift cannot happen from IT, but must happen from the top down.

re: To Break The IT Bottleneck, Stop Covering Your You-Know-What

CYA has a different meaning when considering application deployments/launches vs. changes to infrastructure. It's not hard to replace physical signatures on a piece of paper with clicks in a central approval app, or collaboration tool, so that deployments can be green-lit faster. But no one wants to be the one to say, after an incident for application A, "yes, I was making a change in the infrastructure. Didn't you know that?" After enough instances where the right hand does not know what the left hand is doing, and the business complains, you will end up with lots of CYA notifications for upcoming changes so that no one is surprised. Automation may not help that, unless it helps predict the "knock-on effects".