

ATM WANs: Cornering the Market on Wide Area Data
April 19, 1999
The Object of the Game
We based our TransPAC network on a 35-Mbps VBR-nrt ATM service provisioned as a single PVC (Permanent Virtual Circuit), extending from the ATM-based exchange point in Chicago, called Science Technology and Research Transit Access Point, (see "Bonus Points: STAR TAP," page 80) to the Tokyo APAN XP (exchange point). Additional bandwidth should become available once carriers upgrade the trans-Pacific infrastructure. Routers in Chicago and Tokyo provide Layer 3 IP services.
Several factors played into our decision to use ATM as opposed to, say, DS-3 service. First, we were working with a limited budget and the ATM VBR (variable bit rate) service was much less expensive than equivalent dedicated bandwidth. Second, we needed to be able to carve out several discreet pipes. For example, researchers on both sides of the Pacific are interested in collaborating on wide-area native IPv6 trials, but tunneling IPv6 in IPv4--the primary transport service offered over TransPAC--was not an option, since this would require end systems to run IPv4 and IPv6 stacks and would result in encapsulation overhead. Using ATM allowed us to provision a PVC to connect IPv6 routers on either side of the link.
Setting Up the Board
Most applications running over the TransPAC link will use IP as their transport layer. Although IP provides universal connectivity among heterogeneous systems and interconnecting networks, using IP over a long-distance link, such as ours, and ensuring that packet forwarding conforms to the vBNS acceptable-use policy present a number of interesting challenges.
For example, the speed of light tends to slow IP connections. The distance between Chicago and Tokyo is about 10,000 kilometers, or 10 million meters. The speed of light is about 300 million meters per second, so the delay between Chicago and Tokyo (commonly known as propagation delay) is about 10,000,000/300,000,000, or 33 milliseconds. Other factors compound the delay: Light travels through fiber at only 80 percent of its velocity through the air; router queuing delay and various other delays in the carrier's cloud must be considered. Taken together, delay becomes significant--in our case, it measures 100 ms one way. If a network node in Chicago starts sending packets to Tokyo at the rate of 35 Mbps (the maximum rate supported by TransPAC), the Chicago node would transmit 350,000 bytes before the first packet arrived in Tokyo. The number of bytes in transit on a link is referred to as the bandwidth-delay product.
TCP, the mainstay for providing error-free reliable communications over the Internet, is adversely affected by high bandwidth-delay product links. Through the efforts of Van Jacobson and others, TCP has developed a sophisticated pacing mechanism that regulates the rate at which data is transmitted over a given application connection. TCP attempts to adapt its transmission rate to available capacity, both as the connection starts its initial transmission and in the presence of network congestion as detected by packet loss. Communications links with a high bandwidth-delay product, like TransPAC, hamper TCP's feedback mechanism and typically impinge on an application's ability to ramp up transmission rates on startup or recover from congestion (seen as packet loss). Although most vendors have implemented modern TCP options that improve performance in high-bandwidth-delay networks, we could not assume that all end systems using this network had been upgraded, and thus discarded this as a possible workaround.
To combat the side effects of high bandwidth-delay product and minimize packet loss, TransPAC includes a Layer 3 buffering device (a router) between the TransPAC network and the STAR TAP ATM switch. TransPAC also provides support for end users who need to understand these issues and tune their applications and workstations to perform better over similar long-distance links.
|