Network Engineers Becoming System Administrators

These are exciting times in IT, with one technology after another being reinvented and operational lines blurred as a result. There are wild predictions of SDN being so disruptive to every nook and cranny in the enterprise that practically even the custodial staff will be able to run the simplified network. At the same time, we have subtle but notable changes that are stretching the responsibilities of the typical network engineer into that of a server farmer.

No two IT support groups are set up exactly the same, and some degree of functional overlap has always been common. For example, as a network engineer/architect, I also used to manage a fairly large VPN appliance environment, including compiling clients and managing user databases.  In other organizations, this might fall on security analysts or Windows administrators who also have other primary duties.

But now we’re seeing a lot of technical cheese being moved in a way that frequently puts network support folks squarely in the roles of system administrators. It has happened gradually over the last few years, and in many ways is like the frog in a pot analogy. Without realizing it, many of us who build the network are devoting a greater percentage of our work day to also caring and feeding for a variety of systems and servers used on the network.

This multitasking is well known to VoIP engineers who for years have maintained not only switches and routers, but also soft PBXs and voicemail servers. On the network side, the trend towards sprawling collections of specialized support systems for enterprise environments means that network engineers have less time for routers and switches as they also tend to new suites of physical and virtual appliances that don’t really fit anywhere else for support. I'll focus on enterprise WLAN operations to illustrate my  point.

Unless you have a cloud-managed WLAN environment, you likely have a server-based wireless network management system. If it’s a hardware appliance, it still needs to be regularly patched and updated. If it’s virtualized, then you probably spun up an installable OVA file, and built the box yourself. The same goes with your RADIUS servers, client onboarding system, and analytics engines. There’s the usual network monitoring systems, too. Another recent trend is for WLAN controllers to run in the VM environment instead of on rack-mounted physical appliances. This means that now engineers have to keep up both the controller functionality and its VM host instance. The bigger your network is, the more time you spend keeping up the boxes that make up ever-expanding solutions (and systems that monitor other systems), rather than on the actual network itself. 

As long as things stay healthy, the added duties of server guy/gal are all in a days’ work. But when trouble hits and the hours spent with tech support turn into days and weeks trying to get a sick services server back on its feet, you may find yourself behind on actual network-related work. It’s just the world we live in now.

I remember well a recent incident where one of my group’s major NMS systems went down at the worst possible time. Our project list was fat with work needing to be done, and we’re staffed efficiently to fix problems and to then get on to the next thing while also juggling projects. A month later, our NMS issue was no closer to being solved despite at least two solid weeks  poured into it while a database expert on the other side of the world did her best to figure out what happened. Other work suffered, and finally we pulled the plug on the failed box and started over to cut our time losses and get back to the business of being network engineers and architects.

I’ve talked with four major network vendors over the last couple of months, all who declared that their once-vibrant network hardware core businesses are now just backdrops to their new software focuses, right before they got into all of the new services that they’d be offering. The elephant in the room? Folks like me will be administering those server-hosted services, because they are network-centric and don’t fit anywhere else in the organization

Nothing about this paradigm shift is necessarily bad, but the effect can be insidious in how it impacts staffing and meeting deadlines. If your network engineers seem busier than they used to be, they probably are.