COVER STORY

State of the WAN

by David Willis

Wide area network bandwidth is inadequate, overpriced and often out of reach.The service is hard to price, offerings are difficult to compare and youcan wait forever to get it finally installed. Moreover, carrier offeringsare not keeping up with our equipment. While we talk about

megabits on the campus, we talk about kilobits in the region. We can buildlocal, switched networks that run at 100 Mbps for less than $500 per desktop.Yet, we'll pay 200 times that amount in communications feeds to get lessthan half that bandwidth between two sites for just one year of T3 service.More important, corporations have to rethink their applicat ions when theydistribute them across long distances. The only thing that matters is thatapplications work the way your organization intends, but the great majorityof them are simply not designed to be distributed and providing the requiredbandwidth is extraordinarily expensive.

Times have changed. Legacy applications used to refer to PROFS, but thenew legacy application is dBASE. In fact, providing good performance forPROFS is infinitely cheaper. It was once fairly easy to analyze networkbandwidth requirements. Text-based terminal applications acted essentiallythe same, moving maybe 2,000 to 4,000 characters to display a screen's worthof data. File transfers could be scheduled in batch jobs, and you couldestimate how much data was on its way and size circuits accordingly. Sowe allocated circuits and installed new lines as applications came up.

Traditional bandwidth allocation techniques are straightforward, based onpredictive traffic analysis and possibly some basic queuing theory. If yourenv ironment is tightly controlled and is transaction- or batch-oriented,you can build a very accurate model of your traffic patterns. However, ifyou're in an anytime, anywhere, anything environment using today's commercialapplications, it's much harder to predict network utilization patterns.Managers tend to monitor general traffic flows and adjust the bandwidthbased on the unique traffic mix on the network as a whole--to watch theforest instead of the trees.

Putting large file-oriented applications over wide area networks is likeherding hippos through a garden hose. Desktop software is big, changes frequentlyand makes demands on bandwidth that vary widely. You can spend weeks modelingpotential traffic patterns, only to end up in the weeds when a new versionof an application gets installed, or when users discover how to run concurrentprocesses from their multitasking desktops (see "Hot Wide Area Servicesfor 1996," page 62). Just a few mouse clicks can stuff pipes like PrinceAlbert in a Can.

New network applications need cheap, elastic bandwidth, varying accordingto an application's needs as it occurs, not allocated as the result of amonths-old study. The good news is that it is easy to build this flexibilityon WAN backbones. The bad news is that your access into those networks isstill based on inflexible pipes that either overflow easily or sit mostlyempty. But you can make the most of the WAN links that you can afford.

For sure, big shifts have taken place on the back end of carrier networks.Fiber and fast cell switching build the new carrier backbones, and theseflexible networks are sold to us in a variety of ways: For some customers,they run voice; for others, packet-switched data; and still for others,they deliver video and multimedia. For many of us, they run data via theframe relay interface.

The Ethernet of the WAN Frame relay builds on widely used HDLC-basedequipment. You will find support in routers, Muxes, IBM Front End Processorsand inexpensive plug-and-g o Frame Relay Access Devices (FRADs). Interoperabilityis fairly good, except in compression algorithms, where a standard implementationagreement (FRF.9) was only recently ratified. Frame relay is enormouslypopular--in fact, too popular for most carriers to keep up with the demand.It takes weeks or months to get provisioned, with massive coordination delaysbetween local service providers and national carriers further compoundingthe problem.

Like all packet switching services, frame relay saves money by reducingthe number of circuits required to build a fault-tolerant communicationssystem. It's possible to use just a few access circuits to connect to thecloud, and then build on a carrier's sophisticated fabric of redundant fiberand ultra-fast switches.

Traffic is directed to a remote site via a Data Link Connection, noted bya Data Link Connection Identifier (DLCI), which is simply a number thatuniquely identifies the remote location from the current location. A DLCIdoesn't have to be unique acro ss the system, although giving it some meaningmakes management much easier (see details in the figure below).

Network protocols are mapped to DLCIs at a router's interface. It's possibleto map different traffic types to different DLCIs to provide a certain qualityof service, even if it's going to take the same route. End equipment, suchas routers or multiplexers, can move higher-priority traffic to the frontof the queue. In fact, router and multiplexer vendors are quite proud oftheir algorithms to provide priority queuing, which ensures that time-sensitivetraffic, like SNA, gets to the front of the line before the remote clienttimes out.

However, this does not solve the fundamental problem with frame relay. Withoutoffering levels of priority in the network, where some traffic streams takepriority end to end, you can't provide true quality of service. To addressthis problem, some providers will be offering priority PVCs later this year,and some will attach service guarantees.

Its Or iginal Intent Frame relay was designed as a data service, andinitially there was no provision to support time-sensitive traffic. Butthe enticement of cheap virtual circuits has pushed equipment vendors tointegrate voice into their gear, for use over a frame relay cloud. They'vemanaged to compress voice traffic down from traditional 64-Kbps PCM-codedvoice to as low as 4.8 Kbps--although most agree that sound quality suffersonce you go less than 7.4 Kbps. While that's an incredible feat, you won'tfind a major carrier that will endorse it on its public networks today.

The central problem is one of delay, and it is compounded by wild variancesin delay. A frame relay PVC with a Committed Information Rate (CIR) of 32Kbps means that the network won't throw away your traffic if you submitup to 32 Kb of data in a second. It doesn't mean that a bit placed on thenetwork arrives at the other end 1/32,000 of a second later. It could arrivehalf a second later and still be within the terms of your contract, and your voice will sound like an inebriated Scooby Doo over a CB radio. Ruh-Roh.

Voice is simply another type of data stream. If you can get low latencyfrom your carrier, voice over frame relay can save you big money. But youneed service guarantees written into your contract, and it is necessarythat you monitor it closely.

Monitoring FrameDelay For service monitoring, some equipment vendorscan track delay for you, or you can use external monitoring systems. Manypeople use ICMP pings, a la Unix trace-route, as a measurement of latency,but it is a poor metric. ICMP messages are low-priority, and they may giveyou a falsely high reading. Whatever you use, your carrier must agree onthe metric, as well as when, where and by whom it is measured.

Visual Networks builds a unique external monitoring system. Its Visual Uptimecontinually watches your actual network traffic--which is, after all, whatyou're trying to monitor--and keeps track of the results over time. Thinkof it as RMON for fra me relay (in fact, it's RMON compliant), with betterhistorical tracking. The probes can be integrated into a DSU/CSU, or asa "black box" sitting next to it.

Voice service needs less than 200 ms of network delay for any type of quality,and some people hear a difference with half that amount. MCI is the onlyvendor currently offering latency guarantees across its network, althoughit will not commit to supporting voice traffic (see the chart, "U.S.Domestic Broadband Services," on page 64 for details of carrier serviceofferings). MCI says it will consistently offer under 70 ms of one-way delayport to port.

As with all long-haul services, the greatest cost of a frame relay networkis not the end equipment or the frame relay network itself, but the costof getting to it. It takes a local service provider to bring a circuit fromyour site to the frame relay provider, and you often can't do it cheaplyor efficiently. For now, you have only two choices for nailed-up access:56-Kbps or T1 (1.544 Mbps) leased lines (occasionally you can find fractionalT1 in the local loop, but it's rare and may not save much money).

T1 lines typically run two-and-a-half to three times the cost of 56-Kbpscircuits, and even though 56 Kbps is just barely enough bandwidth to moveLAN traffic, it's what most people buy. So the much-talked-about bandwidth-on-demandfeature of frame relay networking doesn't buy you much because the portspeed ceiling is too low.

Of course, this may change with the competition for local service allowedby the Telecommunications Act of 1996. A price war over local service wouldbe great news, but it's not clear that this is actually going to happen,and many markets won't see a change for some time. In the meantime, don'tsign for long-term leased line service if you can avoid it, or at leastpositively index your rates to the tariffs.

Competition between service providers is pretty keen, so it may be attractiveto knit together carrier services by region. Many networks h ave Network-NetworkInterface (NNI) agreements between themselves. Proceed cautiously. NNI connectionsmay be slow--possibly limited to a few T1 circuits. We've talked to severalunhappy customers. Be sure that a single carrier is the prime contractorand will take ownership for any problems. If you can't get that, you mightas well be on the Internet.

Traditionally, the only way to get more speed than T1 was to inverse multiplexT1 channels using proprietary gear. After about 8 T1s, it's often cheaperto buy T3 service. Now, you're really going to need a strong case for thisone--one where high-speed networking makes or breaks your business. Annualservice rates are in the neighborhood of $50,000 per site. If you need this,you probably already have it.

Service Will Get Cheaper to Build. Will It Get Cheaper to Buy? Newtechnologies reduce the carrier's cost of delivering high speed to yoursites. Today, a T1 circuit requires the carrier to install repeaters every6,000 feet, and the last one can't be more than 3,000 feet from your location.Repeaters are expensive to install and maintain, so that keeps your costshigh. High Bit Rate Digital Subscriber Line (HDSL) technology will offerrepeaterless spans of 12,000 feet over existing copper loops, over two wiresinstead of the four required today. Unless your carrier has installed inductancecoils in the loop, which flatten frequency response to just the 4-KHz voiceband, they'll be able to deliver a 1-MHz channel to your premises.

T1 digital service won't be the only high-speed alternative soon. A cousinof HDSL known as Asymmetrical Digital Subscriber Loop (ADSL) will offerhigh-speed service over copper in an unbalanced configuration--typicallya multiple of 1.536 Mbps in the forward direction, and 16 Kbps or 64 Kbpson the return channel. A lot of attention is on ADSL for home multimediadelivery. Yet this unbalanc ed channel configuration fits many data networkingpatterns, where small requests for data, such as SQL queries, HTTP requestsand file open calls are met with large amounts of data to send back to theclient.

Several carriers have announced trials and possible commercial service thisyear, starting at T1 speeds and then moving up. Cable modems are yet anotheralternative that will be undergoing trials, possibly providing a 10-MbpsEthernet pipe directly to your site. But does your cable company and itsbackbone network have what it takes to deliver data service? These are indeedinteresting times.

SMDS: Maintaining a Foothold for Some Applications If frame relayis the Ethernet of the wide area, then Switched Multimegabit Data Service(SMDS) must be the Token-Ring. On many counts, it's a technically superiorservice to frame relay: It is connectionless, supports higher speeds, qualityof service and any-to-any connectivity via Switched Virtual Circuits (SVCs).Yet it's not widely available and requires more expensive equipment. Higher-speedframe relay is being rolled out now, and with priority PVCs and serviceguarantees, it will be t ougher to choose SMDS.

Its major remaining advantages will be SVCs and a larger address space,making it more appropriate for any-to-any connectivity in large networks.However, most client/server networks are designed for point-to-point communicationsbetween a small number of servers and a large number of clients. It's notnecessary to build any-to-any service in those environments.

ATM: Not a Question of "If," but "When and Where" ATM across a wide distance requires a serious dollar investment. Accessis limited. The current entry level is at T3 (45 Mbps), and you go up fromthere. This year, we'll see a downward shift to T1 speeds, with circuitaggregation capabilities (that is, inverse multiplexing) made availablesometime later.

The widespread acknowledgment of ATM as the integrating network platformis an incredible phenomenon. Ven dors from many camps have joined forces,especially those with traditional interests in data, voice, video or othermultimedia networ king. A common signaling system can be used across thenetwork, from stem to stern. Let's take a leap of faith and trust that newapplications will understand bandwidth utilization, request network servicesintelligently and handle blocked call requests. Let's also hope they emergefaster than network-aware LAN applications did.

It's important to recognize the complexity of ATM. It's a multidimensionallylayered suite of protocols, designed to accommodate many network traffictypes. It is much more than chopping data into 48-byte chunks, tacking a5-byte header on it and shipping it to the other side.

How Does It Become WAN-Friendly? Assuming you have local ATM service,how do you move it over the WAN? It depends on your installed equipment.If you're connecting an existing ATM switch, you'll need a Public-UNI (User-NetworkInterface) connection from the carrier, generally starting at T3 levels.

If you're connecting a router, with a high-speed (more than 45 Mbps) serialport and ATM route r software, you will have two choices: You can send HDLCframes to the carrier's FUNI interface, when they make them available; or,if you need to connect sites now, your only option is to send DXI framesto an ATM DSU, which go to the carrier's Public UNI (see diagram "HowATM Interfaces Fit," above).

Don't plan on hooking two different carriers together for this: ATM B-ICI(BISDN Inter-Carrier Interface) agreements won't be available until at least1997, so you need a single carrier. If you've got some sites running ATMand others running frame relay, look for a carrier that also supports ATM/FrameRelay Service Interworking (Frame Relay Forum Implementation Agreement FRF.8).Interworking has only been defined for Permanent Virtual Circuits, so youwon't have a standard way to do this if your carrier starts offering SVCsover frame relay. The bottom line: If you migrate sites to ATM and theymust interoperate with sites that route packets, you need a single carrierto integrate this interoperability f or you.

The Internet: Cheap if You're Already on It. You can't get very fardiscussing internetworking before you come to the Internet--the vast publicfrontier of bandwidth and information. If you don't need access to the informationthat's available on the Internet, there is little reason to jump on; you'rebetter off building your own network. The Internet is plagued by the sameproblem of all the internetworking techniques discussed here: Access isstill expensive, because you'll use the same digital services, from thesame carriers, that you need for the others. You may also find that it isnot a better value than private frame relay.

Assuming you need to be on it anyway, you can build virtual private networks(VPNs) over the Internet. The IPSec security protocols are being widelyadopted, and firewall vendors are furiously adding VPN functionality totheir systems. However, if you're not already on the Internet, but you haveframe relay, you're probably better off buying PVCs to the Internet forthose sites that need external access.

The Internet can extend your reach in another way: By allowing you to effectivelyoutsource your occasional dial-up connections. You can let your ISP worryabout supporting analog modem connectivity, which has caused many a networkmanager to go prematurely gray. With the right agreement, users will getlocal or 800 numbers to dial, and you can avoid the major capital and supportoutlay that dial-up requires.

Of course, you shouldn't rely on the Internet for mission-critical, time-sensitive,bet-the-company business. It can suffer from extreme delays, lacks securityand can have outage pockets (although generally the noticeable outages arebecause an end site or computer has gone down for some reason, not becauseof massive network failures). You aren't in control of the service, norcan you really complain to your service provider about most problems.

Hot Wide Area Services For 1996

Broadband services are widely available, and most ca rriers offer similartechnology. To compare offerings, you need to understand your applicationsand have some idea about budget constraints. Connectivity is fairly easyif you can afford to buy SONET services for all your sites, but, for therest of us, it's a little more work. Here are some new items to look forin the coming year:

Frame Relay

· Carrier backbone upgrades. Carriers can't support higher access speedsand meet demand for more customer ports without a solid high-speed infrastructure.Many public frame relay services are still built on T1s and routers, andlatency can be a big problem in these networks. If you have T1 access yourself,it's hard to feel comfortable about the carrier's backbone. Look for thelarge carriers to move from T3 to OC-3 (155.5 Mbps), OC-12 (622 Mbps) andeven OC-48 (2.488 Gbps) circuits in their backbones. Carrier edge switcheswill be upgraded to higher-speed, high-capacity switches such as StratacomBPX, Cascade 500, Newbridge 36170 or GDC APEX.

· Priority PVCs, with service guarantees. SNA, voice and even videocan be driven across a frame relay network if you can get the carrier tooffer stable service. Expect to pay extra for the service.

· Switched Virtual Circuits. SVCs will simplify the management of aframe relay network, creating more of a frame relay "cloud" insteadof the current point-to-point configurations. It is a better companion toswitched access service (that is, dial-up analog or ISDN). Announcementswill be made later this year, but few customers will actually have it untilsometime next year. It's likely to be billed based on usage, in contrastto the fixed-rate PVC charges that are currently the norm.

· Frame relay multicast. Frame relay multicast reduces the load ona broadcast source point by allowing broadcasts to enter the network onceand be delivered to a collection of sites, instead of r equiring the senderto deliver to each site individually. This could save access circuit bandwidthfor sites that have a large number of services that broadcast themselves(like NetWare servers), as well as support new multicast applications. Intheory, multicast applications can deliver data to "n" clientsin 1/n the time of conventional point-to-point methods.

· Asymmetrical PVCs. These let traffic in the forward direction havea different CIR than the reverse. If your traffic fits a small-request-met-by-large-responsemodel, this can save a few dollars.

· Frame Relay above T1 speeds will become available. At this time,only a few carriers are offering this service. Some restrict maximum CIRfor a given PVC to 1,024 Kbps, and may require expensive T3 access circuits.

· Switched access to PVCs via analog dial and ISDN. PVCs must be preallocated,and customers dial either a local or 800 number. Because leased lines areexpensive, take forever to provision and require term commitments, ISDNis a viable option for well managed, occasional connectivity. Analog isthe best mobile access choice.

· Voice over frame relay endorsements by carriers. Although frame relaywas intended to be a data service, it's viable for voice in carefully craftednetworks. Voice over frame relay is currently under discussion within theFrame Relay Forum.

· Standardized compression. Standardized compression in FRADs and routershas just been approved as the Frame Relay Forum's Implementation AgreementFRF.9. Many vendors use compression today, but they require their own equipmenton both sides of the PVC.

· FRAD service from carriers. Customers don't have to buy a FRAD forremote terminals. They simply install a serial access circuit, and frameencapsulation occurs over the carrier's equipment. However, FRADs are rapidlybecoming a commodity, with street prices dropping as low as high-speed modems.

· Frame relay to ATM Service Interworking. Defined as Frame Relay Forum Implementation Agreement FRF.8, it is a standard approach to mapping framerelay and ATM PVCs.

A ccess

Competitive access providers will give you new options to consider, hopefullyat lower cost. Look for offerings from traditional alternatives such asMFS or TCG, as well as new entries from the cable and utility industries.Best advice: Don't tie yourself to long-term contracts for local service.

We're predicting that the customer's cost for T1 circuits will drop, dueto competition and new technologies such as HDSL that reduce carrier costsand use currently installed local copper loops.

ISDN tariffs will stabilize, and service will be simplified. Equipment andsoftware vendors are joining the fight to finally bring it to the massesthe right way. It's still the only high-speed switched service that is affordablefor the occasionally connected user. However, it suffers from inconsistenttariffing and may be eclipsed by higher-speed alternatives.

Digital wireless (CDPD) access to data networks will become more widelyavailable in major metropolitan areas but will remain too expen sive forhigh-volume traffic.

ATM

Current offerings support only Constant Bit Rate (that is, voice, video)or Variable Bit Rate (frame relay, compressed voice) traffic.

Available bit rate service is designed for data traffic, particularly ifit is delay sensitive. It guarantees a maximum cell-loss ratio to avoidthe retransmissions of a large packet because a cell is lost, and allowsan application to place bounds on network throughput and delay. It's stillin the process of ratification by the ATM Forum, so nobody's supportinga standard ABR implementation yet.

Carriers will begin to offer lower-cost, lower-speed ATM UNI Interfacesin 1996. ATM access via T1 circuits is being supported by several carriersnow, with ATM Inverse Multiplexing (AIM) on the horizon, bridging the gapbetween T1 and T3.

When Bad Apps Meet Good Networks

If an application won't run across th e network that you can afford, maybeyou should fix the application. You may get lucky and b e able to optimizeit to respect slow and expensive links by placing much of it on a localresource. But it may require a deeper understanding of the program's internalarchitecture and the application development process than you counted on.

Here are some of the many things a bad application can do, along with afew tricks to work around the problems:

Problem: Synchronous, blocking RPCs. Applications can issue ping-pongrequests, even when using windowed transport-layer protocols. Here, theapplication requests data, waits for an answer, then requests a little moreand so on. The user is forced to wait while all of this occurs.

Redundant requests. Applications ask for data they already have. From theexamples listed in the chart below, the NDIR scan asked for the same information("NCP Get File Server Info") 414 times, sending 468,900 uselessbytes over the wire.

Possible Solution: Install local caches and/or virtual disks. Movethe program as close to the data source as possible. Host it on a remotecontrol application server.

Problem: Directory polling. Processes may scan directories frequently,as a method of triggering another action, such as an e-mail transfer.

Possible Solution: Minimize the number of files in polled directories.Reduce the polling interval. Place these processes local to the pollingtarget.

Problem: Lost applications. Applications may not know where to findfiles, and may follow the DOS or Unix path to find additional components.Each directory searched can mean long delays and some network load.

Possible Solution: You can force the application to look in the properlocation for files, instead of using the path. Put the target file locationearly in the path.

Problem: File thrashing. The application opens a file, reads a littledata, closes the file, then starts the process again.

Issuing ver y small data requests. This particular problem can generate hugeamounts o f tiny packets. Examples include character-oriented terminal applicationssuch as Telnet, or Novell's Remote Program Load process for booting acrossthe network.

Possible Solution: Switch to block-mode terminal emulation, or buffercharacters by application.

Problem: Asking for too much data. Examples include file-based databaseapplications, where all query filters are processed at the client. Anotherexample: badly formed SQL queries.

Possible Solution: Remote control hosting. Optimize queries or usestored procedures on your database server. Implement middleware solutionsdesigned for remote computing. Replicate data to the local disk.

Making Any Application Client/Server If other approaches don't help,and the application does not use bitmap images, consider hosting it on aslave processor using remote control software. Some remote control applicationsdesigned for dial-up use also support LAN connectivity, although most requirea dedicated processor fo r each concurrent user. While this will improveapplication performance significantly, it's difficult to scale across theenterprise--you need too many slave CPUs.

Citrix WinFrame is a more scalable application server platform that letsyou host DOS and Windows applications over slow links. Multiple concurrentusers can share a single server. Citrix WinFrame's ICA remote control protocolplaces a relatively light load on the network, requiring about 20 Kbps fora good quality session. If you have lots of legacy LAN applications, youmay want to check it out.

David Willis can be reached at dwillis@nwc.com.

April 15, 1996