Networking Pros Will Need These Skills

Recently, I’ve come across a number of blogs about networking skills. I have some thoughts on the subject, which may be a good bit different than what you’ve been reading. I’m curious whether people agree with me or I’m just being contrary -- so leave a polite comment either way!

Programming/scripting for networks

Let’s start with programming and scripting. Cisco was pushing hard for learning coding, then eased up a bit. Several people I highly respect are enthusiastic about programming, or at least scripting. For that matter, I like coding. Getting someone to pay me to do it is the problem!

Concerning tools like Python, Puppet, Chef, Ansible, etc. — knock yourself out. Having some idea what they can do and how to use them might be helpful. I agree with Cisco’s toned-down version of “all networkers must learn programming,” namely, “must be able to credibly talk to programmers” — although I might quibble about which programmers, i.e. not just tools developers. (How many firms will be building their own tools, instead of DevOps teams? More on this below.)

What concerns me about lots of people scripting is the amount of bad coding that might happen. I’ve programmed in 15-20 different languages, developed one big GUI-based program, and had my share of humbling bugs in all of them. PERL is especially handy if you like obscure bugs. Regular expressions, ditto. Another thing I’ve noticed over the years is the amount of bad or sloppy and under-documented code out there, including my own. I’ve worked at it and write very carefully indented and self-documenting C or PERL code — just because I know I’m going to have to fix it at some point.

Think about it: Someone succeeds in automating some aspect of your network. They leave. Who maintains it? Or your attempt at scripting breaks in some obscure way. Worst case: After breaking routing in numerous routers due to mildly different CLI syntax. From experience, the testing one does is seldom as much as the testing that should have been done, and bugs do hit production.

In summary: One’s odds of avoiding a CLM (career-limiting move) might be better with supported code from a vendor, be it Cisco, Apstra, or whomever.

For reporting, OK, that’s less dangerous. An API is useful while a product is being developed, in case their canned reports don’t do what’s needed. Having said that, in the last year or two I’ve used Python to probe at some APIs. My conclusion: badly under-documented. Telling me syntax and a table schema does not give me context. For instance, one network management product gives me performance data. I couldn’t readily determine what time period the data rolled up, in part because I had something wrong with my query and no good example of accessing that particular API element.

Is scripted automation and reporting really all there is? It’s useful, and bigger shops might pay you to do it. Smaller ones, I doubt it.

Yes, we probably need to be able to script to get more from network management and automation tools. Network management is a relatively small market, and just doesn’t seem to generate the R&D to automatically pull data together into a network model and provide correlated results, let alone root cause. Just getting good graphs out of present tools can be a bit of a hassle.

In fact, that’s one of my gripes. Without naming vendors (and there are several), some seem to think I should use their API to make up for their immature product’s lack of a good set of canned reports. That just doesn’t work for me. If I sit through a sales pitch and find out the product is rather incomplete, I “enable product dampening” — as in you lose points with me for years.

While we’re at it, too many APIs, inconsistent across products from a single vendor — that redefines inefficient use of my coding time. Put differently, APIs are an enabler. Too many APIs, a disabler. Inconsistent semantics, a disabler. Sure, I can write code that does different things based on device model. Waste of my time having to do that?

Catching bigger fish

Maybe we should be widening our skills perspective.

To catch bigger fish, you might use a bigger net.

It is useful to understand how ACI, VMware NSX, and containers do networking. Try to figure out or learn the best practices around hierarchy of deployment, manageable addressing and routing, manageable and secure networking, etc.

Example: ACI can control NSX security. I’m not sure I like the approach Cisco recommends. For one mostly NSX deployment (80-90 percent virtualized), I plan to use ACI as a fabric automation and management tool, and use NSX management natively. That way we won’t be adding troubleshooting complexity due to the control plane interaction between the two. Learning one tool well: good. Learning the two or three competing tools in an area: better. Learning how to make them interoperate: small market for the skill, high risk/complexity?

Another example might be more storage related. I’ve been wondering about where hyper-converged systems are appropriate, and where not. As you scale them up, you add network traffic and latency. From a Google search, I see that “scaling hyperconverged systems” is a thing — but no clear answers. Detecting that applications are slow due to storage IOPS is something we’ve run into a few times lately, as part of proving it’s not the network.

The implication here: perhaps some server/virtualization/storage skills are relevant for in-house datacenter networking teams.

The meta skills point here is: Don’t just learn one way of doing things; learn to be able to discuss the alternatives, their pros/cons, and where they fit or don’t fit.

Next page: DevOps and cloud

I’d like to suggest that another place where network people can really have impact is DevOps and cloud.

Here’s my logic:

DevOps and networking

Sure, there’s DevOps for networking. Except we rarely have Dev/QA and test environments. If you have an actual networking lab, and not just VIRL, raise your hand. Yeah, I thought not.

Consider this: DevOps is supposed to be collaborative. (Although code before documentation and collaboration seems to be the approach in some places.)

Is there a network person on a local DevOps team? Security person? Operations person?

There’s a useful liaison role for someone who can talk to all three of those concerns. Maybe only periodically coordinating with any one DevOps team.

The need (job role) I think I see here: tying network architecture to application and cloud architecture, defining standards, and keeping it all manageable, while not slowing development down. If nobody does it, well, take your present “accrued technical debt,” throw in DevOps teams making it up (or figuring it out) as they go, and you’ve got chaos x 10! (Feel free to interpret that as 10 factorial, if you want.)

There’s also an up-front role, when a given DevOps project is kicking off. I’ve seen one large project need to radically change direction, at substantial cost (I suspect $1M+). The DevOps team apparently did not realize their “micro-services” (actually SOA) code was chatty, and compounded that with perhaps 5-10 milliseconds of latency in their hybrid cloud approach. CoLo providers may cross-connect you with a high-speed link to a cloud vendor, but the cloud servers are highly likely to not be physically located in that same CoLo.

I’ve seen lack of planning around application or service addressing, resulting in awkward routing (8 random /32 host routes needed to selectively override a /24, with corresponding firewall exceptions), or awkwardness in summarizing security zone addresses. That’s something a network/security person might be able to help with up front or as the project evolves. Assuming you don’t like accruing technical debt.

How soon?

The time for new skills may be sooner than you think. Some small firms are moving fast, some not. Large firms generally do move much slower. Anyway, I’d like to share a hint at what future networks might look like.

I’ve been dealing with some small “hollowed out” (my term) networks lately. They have user sites WAN- or VPN-connected to a rack or half-rack in one or two CoLo sites. Their computing is already in the cloud or multiple clouds. Office 365 as SaaS. Salesforce. Time sheets and HR as a Service. Other business functions (accounting, etc.) either in the cloud or in a converged chassis in the CoLo.

The next step for such firms may be virtual routers running in the cloud as WAN/VPN hubs, probably via IWAN or SD-WAN. Or physical routers, where more performance is needed.

The networking person for this likely maintains it, orchestrates buying the hardware, circuits, CoLo and cloud footprints, and coordinates implementation services by consultants.

This blog is getting long, so I’ll expand on this last topic in a subsequent blog.

This article first appeared on the NetCraftsmen blog.