Among the no-shows, Cisco Systems simply declined to participate. Network Appliance offered its full solution, from filers to caches, but balked at our review time line. Nortel Networks and Digital Pipe both said they could not free sufficient resources. Inktomi Corp. licenses its software to F5 Networks and therefore opted out.

Once we unboxed, plugged in and turned on all the systems, the vendor reps walked us through logon procedures and displayed their wares' features and functionality.

In the Beginning

ECDNs, like those from CacheFlow, F5 and Volera, evolved from efforts to accelerate the delivery of frequently requested Web content from the data center to end users on the Internet or at remote enterprise locations. Often applied as standalone solutions, these cache servers typically were used solely for caching Web objects. Today, they have been recast as content-delivery systems, with added intelligence from centralized controllers dedicated to moving content closer to users. Cache servers alone still serve static Web objects, but eCDNs also handle streaming media. The solutions we tested from CacheFlow, F5 and Volera use Microsoft's Windows Media Technology and RealNetworks' RealProxy to deliver streaming media.

InfoLibria's MediaMall eCDN stems from streaming media rather than caching. MediaMalls use full-blown Web and media servers, such as Windows Media Server, to deliver content and Web objects.

Running the Gauntlet

To test how well eCDN solutions relieve enterprise-bandwidth requirements and reduce latency for end-user applications, we emulated real WAN and Internet conditions--complete with delay, jitter and packet loss--using Shunra Software's Storm WAN-emulation tool (see "How We Tested eCDN Solutions").

In addition, we used LoadRunner's Virtual User Generator to capture a Web user's steps in downloading pages from an origin server over our emulated conditions. We compiled that session into a 3-minute-and-25-second script that could be replayed on multiple platforms. With LoadRunner's client software, we created a scenario and replayed the script on 10 500-MHz Dell Celeron PCs that acted as our 100 virtual Web users. All the Web objects under test were cachable, static objects that averaged 15 KB.

On a simulated T1 line, the virtual users received all Web objects within 13 seconds of their requests. When we ran the script over a simulated T3, the average response time dropped to three seconds. Next, we set up the eCDN solutions and retested. Average response time dropped to one second over both T1 and T3 lines. Over simulated Internet connections at 100 Mbps with 100-millisecond delays (10 percent standard deviation and 1 percent packet loss) in both directions, the average response time was one to two seconds. Using the same Internet links with 200-ms delays (20 percent standard deviation and 2 percent packet loss), average response time ranged from two seconds to four seconds per transaction.

Client HTTP Requests: Hits per Second Click here to enlarge

Rather than compare the products' response time in milliseconds, we translated the results to the language cache servers understand the best: hits per second (hps). The results (see "Client HTTP Requests: Hits per Second") show the total number of hits per second generated by clients using explicit proxy caches during 3 minutes and 25 seconds of activity. The baseline results used to compare the cache servers show the results of the same tests without proxy services enabled. We ran the baseline tests and the tests for each cache server over the four WAN/Internet emulations twice.

Cache Servers Meet Content Controllers

The cache servers we tested showed their maturity in ease of use and manageability. Hardware configurations for RAM and disk cache are scoped prior to purchase with presale technical support, making the server as much like an appliance as possible. Each cache server supports intuitive Web interfaces and easy-to-use CLIs (command-line interfaces) for device management. All include controls that limit overall bandwidth consumption in retrieving content from origin servers over slow WAN/Internet links.

But cache servers are not eCDNs unto themselves. An eCDN's true benefit comes from populating the devices with content to avoid the slow trip across a WAN or Internet link to the origin server. Each participant supplied a separate content appliance that pre-positioned Web objects on the cache servers. Although the controllers can be set up in a variety of places, we followed established best practices and set them up in a central location close to original content servers, where they can view and serve multiple cache servers in disparate areas.

Pre-positioning streaming media content provides huge savings in bandwidth and time. A file encoded at 25-Kbps can exceed 5 MB; at 150 Kbps, 35 MB. But if you want to see more than a talking head, the file could grow to more than 250 MB. Ten users accessing a 150-Kbps file will saturate a T1 link. If that stream were pre-positioned, the link would not be restricted and the streaming content would be served from cache.

To pre-position content, you must identify the content and pinpoint the cache server for delivery. For this task, each participant provided job-control and -scheduling features. The degree of control over the delivery process from the point of origin to delivery varied. Although none of the systems we tested work with back-end content-creation or -management applications, they all support Microsoft Windows Media technology and RealServer RealProxy as media extensions to cache servers. They also have controls that dictate when content is "stale" and needs to be refreshed from the origin site.

We did not test streaming media over the simulated network. Because our participants license Microsoft Windows Media and RealNetworks media extensions for cache servers, the available tools for reducing bandwidth requirements are similar across the board. However, Volera's Excelerator can stream media from origin servers at less than the encoded bit rate and deliver such media to users from the cache at the encoded bit rate in real time.

We also did not test cache servers for failover and fault tolerance. The servers we tested can rely on a round-robin DNS to supply these services. But the best practices for each rely on Layer 4 to Layer 7 switches or WCCP (Web Cache Communication Protocol)-enabled routers. Content switching applies filters to IP traffic and routes identified traffic to specified cache servers. For example, IP traffic using Port 80 can be filtered and sent to an IP address associated with a cache server. This content switching or routing is transparent proxy caching (see Proxy Primer").

WCCP, which was developed by Cisco Systems, specifies the interactions between one or more routers (or Layer 3 switches) and one or more cache servers that support the protocol. The purpose of the interaction establishes and maintains transparent redirection of Web services to cache servers. Rather than add another device to the test bed to examine fault tolerance, we set up explicit proxies between virtual users and cache servers.

Lab-Proven Bandwidth Savings

Once specified to control WAN and Internet activity, the caches serve as intermediaries or proxies that accept Web requests from clients and retrieve content requests on the Web servers' behalf. If the requested object is stored in cache (a hit), the cache server returns it to the client. If the object is not in the cache (a miss), the cache server retrieves that content from the Web server.

WAN Bandwidth Savings Click here to enlarge

With the help of these services, all our participants provided bandwidth-usage gains and application performance improvements in caching static Web objects; however, the degree varied from one product to the next. In the lab, Volera's Excelerator and CacheFlow's Security Gateway Appliance saved the most bandwidth along simulated T1 and T3 lines. In LoadRunner performance tests, Volera led the pack with more than 2,000 hps on T1 and T3 lines, and sailed through our Internet tests. CacheFlow's solution maintained just less than 2,000 hps on T1 and T3 lines and scored consistently better than F5's eCDN until "bad weather" hit: On the Internet emulations, CacheFlow's solution faltered and F5's performed like a trooper.

Performance-testing the cache servers depends heavily on the hardware under test. Volera had the benefit of a Dell 2550 server using dual Pentium III CPUs (1,000 MHz) and 1 GB of RAM. CacheFlow's Security Gateway Appliance had one Pentium III CPU, 768 MB of RAM and two 34.18-GB disk drives; F5's Edge-FX 540 had one 550-MHz Pentium III CPU, 768 MB of RAM and four 30-GB disk drives in a RAID 5 configuration. To level the playing field, the LoadRunner scenarios did not tax RAM on any of the systems. The total amount of RAM cache used by the servers during any single test did not exceed available memory. No systems needed to use disk cache.

Price Meets Performance

In the Internet simulations, the maximum network speed approached 100 Mbps before delay, jitter and packet loss. Hence, our test bed put the performance problems on the network, not on the CPU, bus speed or available RAM. To make all the ends meet, we created a price/efficiency (P/E) rating that considered price vis-ý-vis performance.

To calculate the P/E, we used LoadRunner's analysis of total hits scored by each of the participants in each of the timed tests across simulated WAN and Internet conditions. We added and averaged the number of hits across the tests, then divided the cache server's price by that number. The best scores approach zero. Despite the Volera Velocity CDN solution's high price, its 0.03 P/E rating was better than both CacheFlow's 0.12 and F5's 0.28. (We factored in the depreciated cost of the Dell 2550 server on which we installed Volera's software.)

The Victor: Volera Velocity

With great performance, the best P/E rating and superb content control, Volera takes our Editor's Choice award. Volera's Excelerator outperformed both CacheFlow's and F5's cache servers in price and performance. And unlike CacheFlow and F5, Volera includes all its caching services on the CD. Its Excelerator can be configured as a reverse- or forward-proxy cache (or a combination of the two). Both eCDNs from CacheFlow and F5 would require operating system-level tweaks for such a change.

Volera's System Controller and Content Controller, the content-delivery components, provide the greatest control over distributing Web objects to cache servers. They separate configurable content collections from the actions that could be performed on them according to schedules. Cache servers also can be managed and configured separately as publication points. In effect, content collections can be reused in multiple configurable actions on different schedules and delivered to various cache servers. Although F5's Global-Site can do the same, it lacks Volera's view at the edge and cannot manage and control the endpoints in the publishing process.