How We Tested Frame Relay Network Management
We tested frame relay network management systems at the MCI Developer's Lab in Richardson, Texas (www.mci.com/developerslab). From this location, we set up live T1 circuits to four POPs (points of presence) on the MCI HyperStream network, as noted in "Frame Relay Test Environment" below. Using Chariot 2.1 from Ganymede Software (www.ganymedesoftware.com/html/chariot.htm), we created a series of periodic IP traffic streams running at the NNTP, SAP, SMTP and CASTANET application ports between locations.

Our approach was to introduce a wide variety of common frame relay problems and assess how well the NMS (network management system) platforms tracked them, both in real time and in subsequent reporting. We targeted the most important frame relay metrics--availability, utilization, loss and delay--using NMS defaults and/or custom alarm settings as appropriate to the platform. By controlling the configuration within the HyperStream frame relay network itself, we introduced the critical problems that must be properly detected and correctly isolated for carrier service level verification. These include misconfigured or deleted PVCs (Permanent Virtual Circuits), excessive delay and access line failures.

In the first round of tests, we established 256-Kbps CIR (committed information rate) PVCs between all sites in a ring configuration (green lines in the diagram). We examined each NMS's ability to detect these PVCs, setting alarms on network utilization, frame discards and discard eligibility marking, availability and delay when possible. We also examined each platform's ability to identify and analyze the generated IP traffic.

Next MCI added two 256-Kbps CIR PVCs (blue lines) to create a fully meshed network, noting when each NMS detected these new circuits. Using a Wandel & Goltermann DA-30C Internetwork Analyzer (www.wg.com/products/da30/ da30.html) running the Flexmit application, we blasted a constant 1 MB of traffic via an undefined UDP (User Datagram Protocol) port between the Omaha, Neb., and New Jersey POPs. This simulated a new application that had been brought up at these two locations. We assessed each NMS's ability to detect this large, previously unknown traffic stream.

To evaluate local-loop problem reporting, we introduced bit errors on Seattle's T1 access circuit using an Adtech SX/12 Data Channel Simulator (adtech-inc.com/sx.html), generating enough errors to create an alarm condition without bringing down the access line completely. Finally, we fully disconnected the T1 link and examined each system's ability to track system downtime due to a hard-circuit failure.

After bringing Seattle back on line and allowing the alarms to clear, we added 200 milliseconds of latency to Seattle's back-haul circuit (again using the Adtech SX/12). We then waited for the new alarms to appear due to the excessive delay (the NMS alarm thresholds were set at 200 ms rising and 100 ms falling).

Next MCI dropped the three PVCs that supported the West Orange, N.J., location. We used this outage to assess each platform's ability to accurately report availability due to a misconfiguration or outage by the carrier. Finally, we brought the original four PVCs back up, this time at a degraded CIR (56 Kbps instead of the configured 256 Kbps), examining each NMS's ability to alarm on deleted PVCs and CIR changes.