Networking

07:00 AM
Ethan Banks
Ethan Banks
Education
Connect Directly
Twitter
LinkedIn
Google+
RSS
E-Mail
50%
50%

Network Device Management, Part 1: SNMP & Logging

Learn best-practices for monitoring your network devices with SNMP and for tracking events the devices generate.

Over my career as a network engineer and architect contributing to several different organizations, I've inherited several different networks. My pattern has been to start a job, get acclimated, and then crawl the network by hand so that I learn how it's actually been configured. My process is to start in the core and work my way out, documenting important layer two and layer three connections as I go.

Along the way, I inventory device configurations, analyzing how well they've been secured and the overall consistency of configuration from device to device. And then I usually have a bit of a cry. For whatever reason, it seems that many organizations struggle with enforcing a standard configuration across the board. Sometimes the issue is the lack of an overall standard. In other cases, there is no audit process to enforce standards compliance.

There are many tools on the market to help with device standards and compliance. The larger the organization, the more likely you've deployed one or more of them. But what if you're not ready to go with a tool like that, which is probably expensive and difficult to implement? What if you don't even have a standard for network device configuration as yet? How do you get started? While the most appropriate standard for every organization will vary, there are several features that I like to bring to a network that I'm responsible for. Let's start with SNMP and logging.

SNMP
Most network devices are managed via Simple Network Management Protocol (SNMP) these days. Network management systems at all price points rely on SNMP to poll statistics, using the gathered information to create alarms, generate reports, graph trends, and display statuses. Here are my recommendations for configuring SNMP.

1. Use SNMPv3. While SNMPv2c is easier to configure, the data is sent across the network unencrypted, including the community strings (password) required to gain access to the device's SNMP engine. The nature of the data shared in SNMP is potentially quite interesting to someone sniffing that data off the wire, something certain kinds of malware can do. SNMPv3 has the ability to encrypt the data going between the network device and the SNMP manager.

2. Consider making views. SNMP views are limited parts of the SNMP database that authorized users can access. Think of this like limiting access to the folders of a server by assigning specific permissions to a user or group. While it is more work to maintain, there's merit in giving specific users access only to the specific SNMP object identifiers that they require.

3. Limit the hosts allowed to query SNMP. All too often, I've seen devices configured so that anyone, anywhere who knows the community strings can access the SNMP engine of a device. The best practice is to limit what devices are allowed to access the SNMP engine. SNMP has a lot of valuable and potentially sensitive data about your network to offer someone with malicious intent; that data should be carefully protected.

Logging
When it comes to logging, I see the same problems over and over again: insufficient local logging, inconsistent remote logging, and logging to servers that no longer exist. Network device logging is absolutely critical both when troubleshooting and performing forensic analysis of network outages or security break-ins. The time to discover you haven't got the logs you need is not when you actually need them.

1. Make sure the local logging buffer is used effectively. When troubleshooting an issue, the first place network engineers will look is on the local device itself. Different network devices handle their logging in different ways, all depending on their underlying operating system. For instance, the logging process on a Juniper Junos-based device is distinct from that of a Cisco device running IOS. However the device in question performs local logging, the key is to make sure that there's enough local buffer to be useful for initial troubleshooting. For example, on Cisco IOS, the default buffer is only 4K. I usually bump this up to 128K, which usually provides several days worth of logs.

2. Ensure the network device is logging to a remote server, or even servers if you have a redundant logging design. Also, make sure that all devices are logging to the same server(s). Many times, I've run into network devices that don't log off-box at all. If the box itself is inaccessible, you want to be able to access whatever logs it had to offer before it fell off the network. In addition, you want to be sure that logs are being sent to a log server that's actually still there. It's not uncommon for log server infrastructure to be replaced by the server team; the network team isn't in the loop and doesn't realize there's an issue until it's missing log data that was shipped into the darkness.

3. Log the important stuff. Juniper firewalls don't log traffic flows by default. Cisco routers don't log SSH connection attempts by default. Not all platforms log routing adjacency changes consistently. There's no one size fits all here, but the big idea is realizing that enabling logging as a feature does not enable the logging of all possible events. The network engineer needs to spend the time to ensure the different types of important events are logged.

Next steps
With a good handle on monitoring your network device and handling the events that your network device is generating, a good next step to think about is securing remote access to the device. I've found that network device authentication, authorization, and accounting (AAA) are often under-addressed. I frequently see a single, generic local account with administrative privileges shared across all devices in the network.

This approach has the benefit of convenience, but isn't particularly secure. Those who know the account have the potential to gain access to other devices in the organization. In addition, user specifics are obfuscated behind the generic account ID, making it easier for someone with malicious intent to hide. In the next post, I'll discuss a practical strategy for implementing AAA.

Ethan Banks, CCIE #20655, is a hands-on networking practitioner who has designed, built and maintained networks for higher education, state government, financial institutions, and technology corporations. Ethan is also a host of the Packet Pushers Podcast. The technical ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
aditshar1
50%
50%
aditshar1,
User Rank: Apprentice
6/30/2014 | 3:18:17 AM
Re: Event Logging
It is not easy to play on configuration for SNMP, some MIBs only support IPv4, using the legacy ipAddress type, which cannot hold an IPv6 address.
jgherbert
50%
50%
jgherbert,
User Rank: Ninja
6/28/2014 | 10:51:45 PM
Event Logging
"The network engineer needs to spend the time to ensure the different types of important events are logged."

You spoke a mouthful there, Ethan. I'd extend that to say spend the time to ensure that you are logging important events (e.g. routing protocol neighbor changes, interface status changes, etc) but also are properly handling or filtering things you don't care about.

You also have to be careful with the multiple off-box trap destinations to make sure that you don't end up logging to 10 different servers. This adds an unnecessary load to the device and also starts begging the question of whether there's a better way to do this. In the old days, things like a Trap Exploder would assist nicely with this.

Incidentally, when you talk about configuration consistency, snmp and syslog destinations are in the top 3 "randomly assigned" elements in most configurations I see. And usually if I ask "Ok, so what servers should I be logging to" it takes a long time to find a satisfactory answer, if one can be found...
AbeG
50%
50%
AbeG,
User Rank: Black Belt
6/28/2014 | 6:47:36 PM
Re: SNMP is Key
"For whatever reason, it seems that many organizations struggle with enforcing a standard configuration across the board."

Whenever I learn the history of a given company's IT department, I often find that they have experimented with conflicting ideas from various "experts" which include different IT outsourcing companies, IT managers, Network Techs, etc.

It seems that many IT departments rarely provide the necessary incentives to keep IT staff around long enough to see their initial vision through completion.
aditshar1
50%
50%
aditshar1,
User Rank: Apprentice
6/17/2014 | 5:19:26 AM
SNMP is Key
SNMP has evolved as important protocol/ network manager in order to run network, after constructing a network we need some one to monitor daily events on our devices which gets shaped up with help of SNMP. But with same CPU utilization increases rapidly when you allow SNMP traps.
Hot Topics
16
Open Source Vs. Open Enough
Bob Laliberte, ESG senior analyst,  7/18/2014
5
Guide: The Open Compute Project and Your Data Center
James M. Connolly, Editor in Chief, The Enterprise Cloud Site,  7/21/2014
5
Do We Need 25 GbE & 50 GbE?
Jim O'Reilly, Consultant,  7/18/2014
White Papers
Register for Network Computing Newsletters
Cartoon
Current Issue
2014 Private Cloud Survey
2014 Private Cloud Survey
Respondents are on a roll: 53% brought their private clouds from concept to production in less than one year, and 60% ­extend their clouds across multiple datacenters. But expertise is scarce, with 51% saying acquiring skilled employees is a roadblock.
Video
Slideshows
Twitter Feed