How We Tested
The network test setup consisted of two "islands," simulated in the lab as two wiring racks separated by a spool of 25-kilometer (km) single-mode fiber representing our "undersea" link. Our test plan called for a 100-km link, but because of severe weather we had to make do with the equipment we had on hand. Each vendor brought appropriate equipment to connect two complete LANs across the wide-area single-mode link; a multimode OC-3 link simulated a backup satellite link. Each vendor's network had to support ATM-attached servers and clients, Fast Ethernet-attached clients, and MPEG-2 video streams via OC-3 ATM connections (see "Network Test Configuration," below, for a look at a typical network configuration).

We also required the network to support multiple ELANs (emulated LANs) across the infrastructure, and we expected routing services to be part of the LAN. Olicom failed to provide a router for its solution, forcing us to test it in a flat configuration.

We required each vendor to demonstrate its support for a VBR (variable bit rate) video stream across our test bed's OC-12 interisland link, preferably over SVCs (switched virtual circuits). To test the vendors' QoS guarantees, we used a mix of PC clients and Netcom Systems Smartbits traffic to oversubscribe the OC-12 link between Oahu and Maui. We watched for any video degradation and tracked the overall performance of the system, especially in relation to the video streams, Smartbits traffic and client/server throughput.

Very few difficulties cropped up concerning the QoS guarantees. Every switch we tested fulfilled its QoS guarantee without even a minor glitch in the live video streams. The only issue of note was getting the Hitachi ATM switches to accommodate the VBR SVCs. Hitachi engineers could not get the switch to set up an SVC-based VBR connection; ultimately, we settled for a soft-PVC (Permanent Virtual Circuit)-based connection, which enabled us to route PVCs through the core of the network using SVCs.

Next, we took a brief look at fault tolerance. In all cases, failover from the OC-12 link to the OC-3 link was nearly transparent. With the exception of 3Com, every solution we tested ran PNNI (Private Network-to-Network Interface) Phase 1 or PNNI version 1.0 code. PNNI enables quick rerouting of SVC-based calls in the event of a network outage. 3Com was beta-testing new PNNI-enabled code, but we did not accept beta submissions for our test; in lieu of PNNI, 3Com used a proprietary protocol called EIISP (Extended Interim Interswitch Signaling Protocol) to provide this same fault tolerance.

To test systemwide hardware performance, we measured the CDV (cell-delay variation) of each system using a Fluke Corp. OC3port Plus and Hewlett-Packard Co.'s Internet Advisor for ATM. We also measured the signaling performance of each vendor's solution using Smart Signaling software from Netcom.

CDV measures the delay that occurs when a burst of cells is delivered across a network. While the magnitude of the measurement is important, it is even more critical to note the difference in CDV on a loaded network as compared to an unloaded one. In our tests, all values were well within the acceptable limits for a typical ATM network (see "Cell Delay Variation Results," below).

Finally, we tested ATM signaling performance by measuring PCR (peak call-setup rate) (see "Smartbits Call Setup Signaling Performance," page 72). On a typical network, calls are set up and torn down as different devices on the network transmit data. If no data is transmitted for a specified length of time, the call is torn down. Thus, a typical network may see only five to 10 calls a minute on a busy network. However, if a failure occurs in the core of the network, then all of the interrupted calls must be rerouted, which sometimes involves hundreds or thousands of calls. In such a situation, calls pour into a switch from client devices. If the switch can't cope with the load, some calls will time out and reconnect. Some clients will remain unable to access the network until all calls have reconnected. We agree with most vendors that call-setup performance is an important gauge of systemwide performance in a mission-critical system.