FEATURE

THE BRADNER REPORT

Switches, Routers, Bridges:
Reality, Jellybeans and the Dog that Didn't Bark

by Scott Bradner

To download an Adobe Acrobat .pdf format version of all of the Bradner results charts, click here. (244k)

It's that time of year. While the vendor hype machines are always in full roar, it's time to refuel the engines. We've just finished another round in the Harvard switch and router testing series, and there are enough numbers for all. But the most interesting results may be what isn't here.

This round started, as the last one did, with Network Computing sending an invitation for participation. A fax went to each contact person listed for every switch and router manufacturer in the magazine's files. Vendors were invited to a two-month-long, open and free testing party at the Harvard Network Device Test Lab. There was only one catch: If a vendor arrived with a product to be tested, the results would be published in Network Computing and put online ( http://ndtl.harvard.edu/ndtl ). The vendor could not decline the honor of publication even if the results were not quite what they wanted or expected. Of course vendors could, and did, rent the test lab, in which case they could select the results to be published. The exact same procedures were used in both cases, the difference being the vendor could slink away into the dark of night and no o ne would know the vendor was ever in the lab. (Renting the lab during this time
and during the rest of the year is what allows us to keep doing these tests.)

There are two consistent features of scheduling these tests: the desire of too many vendors to come in at the last moment, so they will have the maximum time to get their devices to actually work; and the handful of vendors who wait until the last minute to try to schedule testing time. Things got more than a little crazy toward the end of the testing period. Conrad Nobili, who does all the scheduling and testing, did his best to coddle the laggards but was not able to accommodate everyone who wanted time during the last few weeks.

The results published here are cumulative: They include the results of the past two years of testing in the lab. All results of those two free public tests are combined with the results from the same tests run when vendors rent the lab (and authorize the release of the data) and private survey tests run for companies like Strategic Networks Consulting, Inc. (SNCI). If a vendor comes in with a new version of the same product, the new results replace the old; if not, the old results are reported. We do this to give as broad a view of the currently available products and to try to encourage the vendors to come back when they have a better story to tell. Results dated before 1994 have been discarded.

The Dog That Did Not Bark Like Sherlock Holmes commenting that the most interesting fact in a case was a dog that did not bark, some of the more interesting information in this report is who and what is not represented here. Newer products from some of the major vendors and products from some well-publicized wanna-bes are missing. For example, four of seven vendors in the SNCI ATM edge box test ser ies, conducted at the Harvard lab, refused to let us present their data. Most of these were production ATM devices. (By the way, they had good cause to be reluctant.) In addition, not many of the v endors of this sort of ATM edge device agreed to take part in the SNCI tests or have subsequently come by the lab on their own.

The data for those who did participate in the SNCI tests plus data from other ATM edge box tests is included here. (See http://www.snci.com for a full report of the SNCI tests.)

The bottom line is: If someone is trying to sell you a switch or router and cannot provide you with copies of third-party performance tests like the ones here, you should ask why. I suppose a few of the vendors have not heard of the Harvard tests, and for some of the smaller companies, the cost can be quite high. (Harvard does not charge for the lessons learned during these series of tests, but the airlines and hotels do.) But, there may be something a vendor would rather you not know. (Yes, I understand such an admonition is a bit self serving.)

Switches vs. Routers I'm not even going to get into the switch vs. router debate (some what reminiscent of the bridge vs. router debates a few years ago) other than to note that the big boys' routers are in the same ballpark as switches for performance--they are limited by the LANs they connect rather than the performance of the device itself. In other words, don't fall for the argument that you have to put in a switch to get around the "router bottleneck." If your network design works better with a switch, by all means use one. But if a router-based network gives you the compartmentalization features you would rather have, there are plenty of routers that will perform well.

Is Performance Overrated? Performance is clearly over-hyped, at least by vendors who score well, but it can also be under-stressed, often by vendors who don't make the grade. These days most switches and many route rs are limited by media, and even those that aren't are far faster than most network environments. Keep in mind that although the theoretical maximum rate for the smallest legal pack ets on an Ethernet (64 bytes) is 14,880 pps, the theoretical maximum rate for the largest legal packets (1,518 bytes) is only 812 pps.

A few years ago measurements on one of the too-busy backbone Ethernets at Harvard running at about 65 percent utilization showed a maximum packet rate of about 1,800 pps. Even for the 64-byte packets the theoretical limit on wide area links is very low--less than 120 pps on a 56-Kbps link, for example. The fastest router tested in the lab was able to forward almost one million packets per second--a bit of overkill if you were just connecting two Ethernets or an Ethernet to a wide area link.

I believe that the basic performance requirement should be that the device needs to exceed projected demands of the network in which it will be used. A router or switch that can deal with traffic at rates greater than 5,000 pps on each of its Ethernet links will not be a problem in any network that I'd want to run. You don't need a gigabit router on a 9.6 Kbps dial-up link (on the other hand, it would help a salesdroid make quota).

Once you determine that you have several devices that will meet your performance requirements, there are a number of things that you can take into account when narrowing your options: corporate strategic planning (that is, the boss said so), cost (your boss is cheap), vendor support (do you have to explain your problem to a secretary because customers can't talk to engineers?), network management (do you need to run their management station?), documentation (is it written in geek or your native language?), user interface (do you have to know binary arithmetic to turn it on?), security support (does it support traffic filters to keep Bob's nasty letters from getting out?), reliability (does it have dual power supplies because a repair call means climbing a 1,000- foot tower?), standards adherence (are standards for the other guys?) and interfaces/protocols (so you want to run ChaosNet on ARCnet LANs?).

Test Results, Late ncy: Differences Without Meaning In general, it takes far longer to transport a packet over a network than to process it in a switch or router, so the differences in processing time should rarely be an important selection criteria. It takes 1.2 milliseconds (ms) to transmit a 1,518-byte packet over an Ethernet and, as you can see from the table of latency results (page 130), most of the switches and routers have processing latencies far smaller than that.

A number of vendors are pushing cut-through switches. In store-and-forward switches and in routers, a packet is fully received before it is transmitted, so adding a device into a network guarantees that there will be at least one packet transmission time delay in the network. Cut-through switches remove that delay (at least in uncongested environments) by starting to transmit the packet as soon as the switch has received enough of the packet to know where to send it--usually 35 or so microseconds worth on an Ethernet.

One problem w ith cut-through devices is that they cannot weed out corrupted packets before forwarding them, since the corruption check is at the end of the packet. If you have a noisy network, you might think twice before enabling a cut-through mode; the decrease in latency might be outweighed by the increase in LAN traffic because of corrupted packets.

Test Results, Routers: Limited by Reality Routers are still an area where there still can be significant differences in performance. There is a series of low-cost products that have relatively low performance. Even the slowest of these might perform well in many environments, particularly over low-speed WAN links. They might not be able to deal with a too-busy Ethernet, or a 100-Mbps Ethernet connecting high-performance workstations or servers.

On the top end, router products are more often limited by reality (such as how many packets per second can fit on the wire) than internal performance. Despite this, a number of vendors harp on their a bility to forward traffic faster than the next. (I may not think that the devices would perform all that differently in an actual network, but if it keeps the vendors renting the lab, I'll claim the difference is meaningful.)

Test Results, LAN Switches: Perfection Is a Requirement LAN switches have become like jellybeans, at least in the performance arena: They are interchangeable. Most of the current generation of LAN switches run at wire speed for as many wires as you can plug into them. In fact, if you find an Ethernet switch that cannot handle the maximum theoretical packet rate, you are generally looking at a product that may not be around long. Their competitors will point out the failure to be perfect and imply that the switch is no better than a brick because of it.

The Future For the past year or two we've been trying to put together a series of tests for ATM switches. This has been difficult. We would like to split the difference between having a set of tests that differentiate between products on the basis of performance-related measurements and having a set of tests that a number of vendors will be willing to take: too much differentiation and the vendors with devices on the low end of the curve tend to shun the limelight; too little, and the results are useless.

We think we are about ready to start with the ATM switch tests. We now have an Adtech AX/4000 tester that looks like it will be able to perform some very useful cell latency and jitter tests along with some switched virtual circuit setup timing tests. We also have a modified Cisco LightStream ATM switch that we will use to produce multiple streams of ATM traffic at OC-3 rates. The plan is to include results from those tests in the report next year.

Testing Throughput, Packet Loss Rate And Latency

The tests used in th e Harvard lab are based on the work of the IETF Benchmarking Methodology Working Group (BMWG). This work includes RFC 1242 on testing terminolo gy and RFC 1944 on testing methodology.

Throughput The throughput test determines the highest rate at which a router or switch can receive, process and forward packets without losing any. This value is important because a pause of up to a few seconds can occur whenever a packet is lost from a data stream, since the application must then realize that data is lost and retransmit the missing data.

To test throughput, the tester sends a 30-second burst of traffic through the device at a rate that is half of what is theoretically possible for test conditions. The number of packets sent is then compared to the number received. If all of the packets are received, the data rate is increased and the trial is re-run. If all of the packets are not received, the rate is lowered and the trial re-run. This process is repeated until the rate is found at which all of the packets are forwarded

but where a rate of even one more packet per second causes packet loss. We performed the Ethernet th roughput tests using an Alantec PowerBits. The Harvard lab uses a PowerBits and a Wandel and Goltermann DA30 for FDDI throughput testing.

Packet Loss Rate The packet loss rate test is used to characterize the performance of a router over the full range of possible traffic loads. The test begins by sending a 30-second trial of traffic at the maximum theoretically possible rate for a packet size and media through the device under test, or at the maximum rate that the tester can generate if that is lower. The number of packets received by the tester is subtracted from the number sent to yield the loss rate. The rate at which packets are forwarded is also measured.

A graph of the results of the packet loss rate test for a range of packet sizes is the best way to characterize a device's performance, but since graphs for each of the protocols would take up more space than we have here, this information has been reduced to include only the maximum forwarding rate observed during the packet lo ss rate test. This forwarding rate is independent of the amount of packet loss. In a large complex network, the forwarding rate can be a good general indication of router or switch performance.

The maximum theoretical data rate depends on the packet size and type of media in use. For example, the maximum theoretical rate for the smallest legal packets on Ethernet (64 bytes) is 14,880 packets per second (pps). The maximum theoretical rate for the largest legal Ethernet packets (1,518 bytes) is 812 pps. The Ethernet packet loss rate tests reported here were performed using an Alantec PowerBits. A Wandel and Goltermann DA30 is used along with the PowerBits for FDDI packet loss rate testing.

Latency Latency is the measure of the length of time that it takes a device to process a data packet. In a real-world network, longer latency in a router, switch or bridge is frequently not noticeable and is overwhelmed by the transmission time of a packet on the network itself. Even the much longer tran smission latency has little effect on the operation of applications running over a network. The major exceptions to this rule are non-burst-mode Novell networks and some client/server applications. We used a Netcom SmartBits tester to measure the time between the packet transmission and its reception with 100 ns accuracy. A two-second burst of packets at a rate of 100 pps are sent in a latency trial with the latency being measured for the middle packet in the burst. Each test is repeated 15 times and the results are averaged.

Updated July 8, 1996