The chosen performance service metrics for application services also will be response-time-focused. However, in these cases, we're usually more interested in end-user response time rather than network response time. This is because there is server processing time involved in addition to network delay. So our metrics should be based on actual user actions, such as time to download a file, send a message or access a particular URL.

Data Needs Analysis and Instrumentation If we're going to produce meaningful reports of how a service provider performs against baseline metrics, we first need to ensure that the right kind of management data is being generated. To instrument for tightly defined bandwidth service metrics, we need to make use of SNMP agents collocated with the pipes being measured. Fortunately, most routers and other network devices are available with SNMP agents that provide most of the information needed for baselining, including availability, utilization and frame relay CIR.

It's important to differentiate between baselining and planning. The information available from standard network devices is limited; it doesn't usually go beyond RMON (Remote Monitoring) Groups 1 through 4 because of the processing overhead that would be required. You probably won't need the additional information for baselining purposes, but you may need it for planning purposes to examine trends in network usage. If you do need greater granularity of information than what's available through the SNMP agents in your network devices, you'll need to install network probes in strategic locations in your network. These network probes, available from vendors like NetScout and Visual Networks, can provide detailed breakdowns of traffic, but they tend to be expensive.

SNMP agents perform the function of accumulating real-time data, and management systems periodically poll the agents to collect accumulated (and usually summarized) data. This approach works well for bandwidth services and may be used for managed and application services. For instance, you can use application-level monitoring from RMON2 and proprietary approaches like NetScout Enterprise RMON. However, a different instrumentation approach may be required for the response-time monitoring needed for managed and application services.

In general, response-time instrumentation requires a polling system that uses service-specific actions to measure the delay between sending a request and getting a response. For application-level response-time measurement, the polling system might perform a series of actions that are the same as those performed by a user--for example, connecting to a specific URL and using FTP to download a file or using HTML to download a Web page. In most cases, the polling system is built into the tool used to generate the reports, so we'll look at the specific tools.

But first, we need to address the main challenge with the polling approach to instrumentation--the difficulty of eliminating from the measurement those factors beyond the scope (and, therefore, control) of the service provider. When instrumenting managed services, this is best-achieved by locating the polling device as close as possible to the service being measured (as described above for our router/frame relay managed WAN). In such a case, the measurement factors beyond the control of the service provider are the performance and availability of the central LAN and the performance and availability of the polling system itself. Both of these factors may be eliminated by introducing a control measurement, such as the response time to another device on the same LAN; if the WAN performance/availability metrics change without significant change in the LAN performance/availability metrics, then that change is caused by the managed service from the service provider. The metric in this case should be the delta between the overall measurement and the control measurement (for both performance and availability).

In this managed services example, eliminating the "beyond scope" factors is relatively straightforward because the control measurement takes place across a network we own. Bear in mind that for some services, such as Web hosting, neither party may have significant control over the user to server data path. Eliminating factors that are beyond your control is more challenging and may not always be worth the effort.

Reporting Now let's take a quick look at some of the actual tools that can be used to collect and summarize the data. We're interested in two classes of reporting tools. The first class is used to collect and report on data from SNMP agents (and is mostly appropriate for bandwidth services). The second class of tools is those polling systems which we've just described as an appropriate solution for managed services and application services.