Troubleshooting High CPU Usage On Cisco Nexus 7000
Something a bit different came up in troubleshooting recently, something I’d like to share briefly in case it bites you.
Symptom: High CPU on core Nexus 7K in datacenter, leading to UDLD and VPC failure and some short-lived STP oddness. Nothing really showed up as hitting against the COPP (Control Plane Policing) policy. The site was running Strict level COPP, albeit from a prior NX-OS version. The newer Strict COPP template policy hadn’t been applied, but isn’t much different. Intermittent CPU spikes visible in “show process CPU history” command, which appear to have been normal “platform” process idle time processing.
More data: Syslog showed some interesting messages. It showed a lot of Glean syslog messages like the following (carefully edited to hide identity):
%ADJMGR-3-IP_GLEAN_EXCESS_TRAFFIC: adjmgr  Excess traffic (10048 packets / 14709540 octets) were dropped in 00:04:48 seconds for ungleaned prefix .../32 on interface ... in vrf default
Cause: After tracking down the destination host (/32 in the syslog message), looking at the sources from NetFlow data, Googling, etc., someone realized that the destination was a NetFlow collector that had been recently decommissioned.
Resolution: Taking out/updating the NetFlow export commands on about 30 routers stopped the above syslog messages, and appears to have lightened the CPU load somewhat. Looked at in a different way, the Glean handling may indeed have been doing its job, protecting the CPU.
Explanation: When a packet has to be forwarded to a destination, the last hop router is responsible for ARP lookup of the destination MAC address. When hardware forwarding doesn’t find an adjacency for a known ARP entry, a Glean occurs so the supervisor can send an ARP to get the missing info. The supervisor creates a negative ARP entry until an ARP reply is received or a timeout period has passed. That saves pointless further ARP queries when one is already in progress, protecting the supervisor from a high load.
Note that Gleans occur in response to a normal packet being received with normal destination on a local subnet and requiring ARP, thereby triggering CPU activity. COPP only rate limits traffic going to the router control plane. The ARP table misses may not be handled by COPP, although apparently the Nexus 7K has hardware Glean rate limited.
Gratuitous Advice: When decommissioning a NetFlow collector, particularly one with lots of source routers feeding it, remove the relevant NetFlow (or IPFIX) export commands first. That way, you avoid what might be described as a “self-inflicted DDoS attack.”
I do have to note that the transient VPC/STP issue root cause is still unclear. It was a one-time transient condition. Some spike in NetFlow traffic might possibly have bumped up the Gleans to where they were a problem. Since the transient, we haven’t seen anything peg the CPU nor cause other problems. So there might well have been another cause for the VPC problem, or another cause compounded by the Glean situation.
This article originally appeared on the NetCraftsmen blog.
Recommended For You
There isn’t a standard way of performing an application baselining or profiling. Here is a how-to video with suggestions on how to work through the process.
Hybrid and edge data centers are expanding the role of the traditional data center. This makes DCIM more important today. As with any management software, organizations need to know when it makes sense to keep it on-premise versus going with cloud-based DCIM.
The Interop 2019 speaker discusses ways that enterprises explore DevOps, the skills gap, and the rise of security as code.
Composable infrastructure provides a cloud-like experience for provisioning resources. Understand how it works and how it differs from Infrastructure as Code.
Big changes are happening with data center management as emphasis shifts from core to edge operations. The core is no less important, but the move to the edge opens new challenges as the environment becomes more complex.
Video overview on how to use a portable WAN emulator to validate bandwidth requirements to a backup server.