A Black-Box Future For IT Admins?

We IT admins don’t really care much for black-box technology. It’s not that we have root arrogance or a compulsive need to directly observe the minutiae of every system component; we’re actually happy to accept a minimum numerical aperture for our systems, below which we let the system take care of itself. For example, I care about the utilization of my Nexus’ CPUs, but don’t care about ASIC pipelining efficiency. There’s peace in having limited monitoring granularity.

But what happens in the not too distant future when vendors convince IT management that the key to reduced IT staffing is systems that manage themselves? What’s the role of network and IT admins in a world of black- box systems?

The cloud effect

Three years ago, I wrote about the sea change of IT in an inevitable transition to cloud. However, as a recent SolarWinds study on hybrid IT confirmed, it’s turned out to be not a rising tide, but a storm causing real turbulence in IT. Usually it’s us, or at least the cool kids on the application and network engineering teams, that push businesses to adopt new technology. The “experiments” with Azure and AWS are paying off for businesses, however, and today, senior IT managers push cloud, SaaS, public internet WAN and a host of other technologies on engineers. 

Executives are talking to one another and vendors at tradeshows about ways to accelerate the demise of the data center, and even the somewhat logic-insulting goal of “zero infrastructure.” (I’m waiting for someone to offer EaaS, Ethernet as a Service.) This shift puts significant strain on already overwhelmed IT, who must increasingly monitor and manage not only everything on-site -- networks, applications, virtualization, storage, and security -- but simultaneously adopt new technologies, tools, and processes to ensure cloud, SaaS, SDN, containers, and DevOps all work seamlessly together. It’s a tough time to be a traditional admin.

IT admins on notice

Even if a business discovers ROI calculus that keeps significant application infrastructure in-house, hyperconvergence vendors are rolling out products that offer the same extremely homogenous, monitoring-opaque platform of cloud. In all cases -- managed services, managed offsite infrastructure (cloud) or managed boxes on-premises -- these systems, by design, are not intended to be monitored in detail.  They are designed to be managed by SLA. For a given cost of service, they’re expected to deliver a specific metric of performance without intervention by IT.

Consider your cloud resources. You’d probably love to get the same metrics for your internal VPC networks that you’ve relied on for years from your on-premises network. Netflow between machine instances would be really helpful when watching application traffic mix for custom applications. But the point of cloud is that the provider manages that for you, and it becomes a black-box network.

With hyperconvergence, you’re not watching VM hosts chassis by chassis, and you’re not monitoring drive arrays or even top-of-rack or end-of-row switches. It’s intentionally horizontally scaled with identical black boxes delivering application services. Instead of watching the internal stack for resource contention or component failure, you’re monitoring quality of service. And that means those doing the monitoring don’t need specific understating of IOPS, routing, memory read/write performance or most of the myriad technologies they troubleshoot. And in the minds of management, that makes them less expensive.

The future is half empty

If you’re a pessimist, it’s easy to imagine an IT future mirroring most other service industries, like auto mechanics. Not that many years ago, we relied on skilled mechanics who could troubleshoot any problem down to a root cause, methodically disassembling and testing parts to find a solution. But today, shops demand greater efficiency and repeatability while not relying on scarce senior mechanics. Their solution is to replace entire assemblies, even at higher parts cost to the customer, because it’s faster and requires less staff, expertise, bays and tools.

In this IT future, admins work under the direction of the systems they manage, which use machine learning and large assembly triage in the form of containers and elastic provisioning to recommend the best course of action for a given circumstance, engaging human administrators only when they get stuck. Even today’s complex, nuanced and highly-tweakable on-premises networks are ripe for massive simplification and standardization when (almost) everything moves offsite or converges into a few racks. Campus networks become so simple their maintenance can be directed from elsewhere.

The future is half full

If you’re an optimist, however, and the numbers work out the right way, there’s a future for IT admins that’s really interesting. It comes down to the chicken or the egg paradox of IT: Is a shortage of talent driving automation to fill the gap, or is increasing automation discouraging STEM careers? If skills shortage is driving automation, then the future of IT is likely significantly fewer admins doing way more interesting work. Instead of driving to the office at 10 p.m. on a Saturday for a maintenance window, adding the same config ACL for the 500th time or vMotion-ing that troublesome mailbox server yet again, we’ll become instructors to our systems. We’ll think in programs and policies, make great decisions based on assessed risk, and evolve systems capacity, not make knee-jerk upgrades.

Will it be difficult letting go of detailed metrics that reassure us we’re doing everything in our power to ensure great experiences for our users? Yes; it already is with cloud. But will we still have dominion over our systems and provide innovative services that drive our businesses success? You bet.  Once we cared about the motherboard, and now memory sticks fail and the hypervisor soldiers on, if with a blinking bezel and a limp. I’m ok with having no idea where an application is running, as long as it’s available without intervention. There are other things to worry about.