To help, vendors have put some RMON capabilities on every port of a switch. Now, most switches have RMON Statistics and History groups on every port; some have Alarm and Event groups, which notify the network management station only when statistics have reached a threshold. This is more scalable than constant polling of all the ports of a switch. The Statistics group runs counters of basic utilization and error statistics on every port, and the History group lets you store samples of this data that can be queried for trends. This is great in theory--especially as commodity RMON ASICs make it possible without serious additional costs--but in reality, it's fraught with peril.

The first problem is simplifying the identification of the ports and mapping them to the actual physical connections. To do this, RMON relies on an older MIB called MIB2 (RFC 1213), which was first designed for routers. It includes an entry in what is called an ifIndex table for every interface on the device. The problem is that the numbers in the table are sequential, starting with 1 or 0. These numbers don't map well to the physical port numbers on the switch, making it difficult to interpret the RMON data. A modular switch complicates matters further: Adding a card may change the number sequence. Each interface in the ifIndex can include an interface description, letting vendors put data, such as port speeds and media descriptions, in this field to help mitigate the problem.

Concord Communications' Nethealth software creates a database entry for every port from which it collects data. A user-definable field makes it easy to modify the port description to a more meaningful value, such as a true port number. Entering and updating this information does not scale well, so concentrate on the key connections, like those on a backbone. But if you can keep the port mappings up to date, you can publish a data report on the Web.

Switches pose another problem. Backbone connections are usually full-duplex, providing two separate physical point-to-point connections. RMON was designed for eavesdropping on shared networks, so it can't differentiate between the two directions of the full-duplex connection; RMON will underestimate the available bandwidth by at least half. But even this information can be of some value if utilization is not high and traffic patterns are asymmetric.

You can circumvent this by returning to the older MIB2 standard, which looks at transmit and receive bytes separately. Another approach uses a probe with a full-duplex pass-through feature. We looked at Shomiti Systems' 8-port Voyager Probe, which uses a port for each device on either end of the full-duplex connection, enabling traffic levels to be seen from each direction separately. Monitoring four full-duplex connections at once makes RMON-probe capabilities on multiple connections more cost-effective.

RMON2: The Sequel
RMON only shows statistics based on the Ethernet layer. For example, while RMON will display traffic patterns based on utilization and errors per MAC address, or per MAC-address pairs, traffic that arrives on the network from the other side of a router port will be identified only by the router port's MAC address. Vendors created their own extensions that offered a better view into the network layer and above--helpful, but defeating the purpose of standards. The IETF answered with RMON2 (RFC 2021), a standardized way to characterize traffic based on IP address and applications based on TCP and UDP (User Datagram Protocol) ports.

In some ways, RMON2 is almost too good. It can track and generate tremendous amounts of data in no time. It identifies conversations based on IP address pairs and every application based on the TCP or UDP port of every IP-address pair, and counts the bytes associated with each. We installed the Shomiti probe on Syracuse University's 10,000-node network between a backbone router and the Internet. The placement meant the probe had to add thousands of entries per hour to its RMON2 tables. Fortunately, the ASIC-based Shomiti probe came with 128 MB of memory.

But even if a probe can keep up, what do you do with all the data it collects? RMON2 made it possible for us to pair the Shomiti probe with one of the best RMON2 management applications we've seen: 3Com Corp.'s Traffix 3.0.4.

The Traffix server component collects data from multiple probes and stores it. The data is retrieved at will via the powerful, Java-based GUI client; you can choose a probe and specify the time and date range you want with a few clicks.

The Traffix client summarizes IP networks and individual nodes in an expandable strip on the left. The main window forms a graphical matrix of each node or network with lines between them representing each connection, color-coded by protocol. Although it's possible to zoom in and out of the matrix, the amount of data from our Internet connection filled it up as one solid mass. Clicking on an IP address on the left filtered out most traffic to reveal only connections to and from the selected node. A window on the right summarizes all protocols for the selected data and the top 10 connections. Right clicking on the matrix offers another useful list of filters and graphs.

Given the high CPU and memory requirements of RMON2, few vendors can support statistics that reflect the network layer and above on switches. If you really need these stats, invest in a probe. RMON2 probes can be expensive, so limit your monitoring to strategic backbone ports, such as switch-to-switch, switch-to-router or server connections.

Even with a probe, monitoring switch connections can be tricky since you must tap into the traffic. To do this, plug the probe and both switch connections into a shared-media hub, but not with full-duplex connections. The Shomiti Voyager probe uses a built-in transparent tap to overcome this. Another approach is port monitoring or mirroring, which sends a copy of all traffic originating on one port or a group of ports out of another port to which you would attach the probe. One caveat: A full-duplex port could oversubscribe the mirrored port, which can transmit only in one direction. The IETF is addressing many of these issues with new standards.

Cisco Systems and 3Com use a technique called Roving RMON. With a probe installed in a slot in a chassis-based switch, network management software easily switches RMON collection from one port to another as needed. But you can only monitor one port or group of ports at a time.

3Com leverages its NICs via its Dynamic Performance Network Performance Manager. A driver running on a system with a 3Com NIC collects RMON and RMON2 data. A server queries the NIC and aggregates the data. While the data collection process is proprietary, any RMON-/RMON2-compliant network management application can query the server for compiled RMON data. The approach distributes the resource-intensive burden of RMON2 among all clients, and helps real-time troubleshooting by

remotely initiating a packet capture at the NIC. Choose good community strings for the clients; these could be dangerous in the wrong hands. And though Cisco, 3Com and others offer some compelling solutions, it's best to avoid proprietary components whenever possible.

Peter Morrissey is a network systems programmer at Syracuse University and a Network Computing contributing editor. Send your comments to him at ppmorris@syr.edu.