Fast and Good

Although we appreciate switches that process simple requests at lightning speed, we weren't nearly as concerned with rated performance as we were with consistent performance with a full configuration and under heavy loads. Our real-world content and concurrent-session-based test scenario shows these devices' real performance capabilities. No switch we tested matched its advertised performance metrics, mainly because those numbers are pegged using tiny files and multiple requests per connection.

Still, these devices performed over and above CMP's stated needs in terms of concurrent sessions and total throughput, with several products--especially F5's Big-IP 9 Application Switch and NetScaler's 9950 Application Delivery Switch 6.0--showing marked improvement over previous generations. We think they've evolved nicely.

Our biggest concern was CPU utilization. We're comfortable with a 60 percent maximum utilization threshold, but NetScaler's 9950 and Array's TMX3000 failed to stay under the maximum while under load. We noted with interest that though Array's switch fell down completely as its CPU utilization climbed to the 95 percent range, the NetScaler 9950 performed relatively consistently even at 100 percent CPU utilization, as seen during tests with its compression capabilities enabled. Although we appreciate decent performance at such high utilization rates, this makes us nervous because we can't predict capacity based on CPU load. The rest of the products we tested held near 30 percent to 40 percent CPU utilization under load, with spikes in the 60 percent range, which we found perfectly acceptable.

We also evaluated how overall performance affected latency. We attempted to reach 20,000 sustained TCP connections with our high-load tests, and the resulting metrics showed that this increased latency on all devices. F5's Big-IP handled this load the best with a TTLB (Time To Last Byte) of only 1,329 milliseconds, while the TTLB for Sun's N2120V Secure Application Switch and NetScaler's 9950 increased to more than 5,000 ms. Transaction and connection rates and TTLB were not an issue in our low-load scenario for any of the products except Array's TMX3000, which had problems performing up to expectations even under what we would term a low load of 2,000 established TCP connections.

It's interesting how a simple design decision can produce very different performance results. Because all entries were required, when capable, to be configured with an 802.3ad trunk, we could examine load-balancing capabilities at more than one layer. We discovered that the devices from Sun and Nortel load balance across the trunk based solely on MAC (Media Access Control) address. Sounds innocuous enough, but this setup caused no end of problems--distribution of traffic across the aggregated channel was uneven because our test equipment uses a limited number of MAC addresses, even though it generates a large quantity of IP addresses. In contrast, F5's Big-IP uses an IP/MAC hash to distribute traffic across the trunk. It had an advantage here. Lest you think this is only a lab problem, consider that internal networks and proxies--say, AOL's--would cause the same effect with large amounts of traffic. We're pleased to note that Sun reworked its hashing algorithm to include this type of functionality after being notified of our test results.

Although performance under load was an important concern, we found that the biggest differentiator among products is their implementation of rules. Rules route traffic and apply policies based on myriad variables, making it possible to architect efficient network topologies. We're always impressed with reusable rules, especially given the requirement to manage traffic for multiple CMP sites in a virtually hosted configuration. Although all the products we tested allow some form of reuse, F5's Big-IP offered not only reuse, but also extreme flexibility. The other products required a minimum of two rules per virtual server to achieve our desired functionality; F5's did the job with a single elegant rule and still performed like a champ.

Our biggest gripe? Case sensitivity. We abhor writing eight separate rules to cover all possible combinations to handle a mixed-case string of "bot." The devices from Sun and Array both failed to provide case-insensitive matching in Layer 7 rules.Products that don't offer both conditional branching and nested rules require implementing multiple policies. The way in which those policies were evaluated weighed heavily in the product's ease of configuration--or lack thereof. Array's TMX3000, for example, requires convoluted configurations to handle routing requests based on more than one HTTP header. Both Array's and NetScaler's products required a second proxy to handle our configuration. Array does not offer an if-then-else syntax to handle nested rule sets, something the other participants provided (and for which we are grateful). NetScaler's approach was also odd in that load-balancing duties are handled by a Layer 4 virtual server, and we needed a second Layer 7 virtual server to route traffic to the appropriate Layer 4 load balancer.

Layer 2-3 support for most of the products was what we'd expect from a switching device. All but Array's TMX3000 easily handled our trunking and VLAN tagging requirements; even NetScaler's 9950, which doesn't employ custom silicon to handle network processing, was undaunted by this task. Our only problems with Layer 2-3 on those devices that offered full support stemmed from configuration. F5's revamped interface was the most intuitive and navigable, while Sun's "Sun standard" interface made it nearly impossible to determine what to change or where to alter configuration of the trunk. NetScaler and Array held their own in configurability.

The Other Two Cs

Along with configuration, the products' caching and compression implementations differ in interesting ways. NetScaler's 9950 is not only easy to configure, it also includes in-memory caching and compression functionality. Sun, conversely, provides neither caching nor compression.

The F5 Big-IP's lack of caching stood out when measured against NetScaler's 9950. F5 told us in-memory caching will be available in a forthcoming release, as will hardware-based compression. NetScaler's 9950 and F5's Big-IP both provide software compression now; the former can compress any TCP traffic, while the latter is constrained to HTTP traffic only, but we're not convinced that the feature is worth the CPU cost anyway (see "Choose Compression Carefully,").Caching dramatically improved the performance of Array's TMX3000, doubling its transaction rate and halving its latency when compared against the same tests run without caching. NetScaler's performance improved in terms of latency, but the improvement in sessions and HTTP gets per second was negligible. Array's menu of caching options is not as broad as that of NetScaler, which has clearly put a lot of effort into offering a flexible, highly configurable caching scheme that improved performance, albeit not as much as the improvement seen by the Array device when caching is on. F5 needs to weigh the breadth and depth of its forthcoming caching offering if it expects to compete with NetScaler in this focused area.

Configuration differences were evident mainly in how easily we could navigate and view the associations between rules, virtual servers, groups and real servers. We liked F5's new interface, which makes navigation intuitive. NetScaler's Java console is also intuitive, though terminology differences and configuration quirks lowered its score. Sun's scheme is made more difficult by the need to first understand its virtual switch (vSwitch) model, but what irked us most was that we couldn't navigate easily between virtual and real servers. Array's Flash configuration is an eye-pleaser, but its responsiveness is lacking.

Only F5 and Sun let us configure either network- or serial-based failover connections between two of their devices; NetScaler and Array support only network-based failover. Statefulness also varies, with Sun's failover stateless, F5's and NetScaler's stateful, and Array's persisting in only TCP state across devices.

Array's and NetScaler's methods of synchronization between primary and standby units in our failover tests made sense, requiring little more than our indicating which device was primary and which secondary, then pushing a button on the GUI to synchronize the configurations. F5's and Sun's setup was nearly as simple, requiring only a bit of futzing with the configuration of secondary and shared IP addresses in the network-based failover deployment.Prices (as tested) ranged from $104,999 for NetScaler's top-of-the-line model to $41,985 for Array's TMX3000.

Wrap It Up, We'll Take It

When our tests were completed, F5's device emerged at the head of the pack and took our Editor's Choice award, with NetScaler's so close on its heels it was almost a tie. Although CMP's Internet Technologies group was impressed with the NetScaler, it was wowed by the flexibility of F5's new iRules and the Big-IP's ability to inspect, transform and route traffic based on any bit in the payload. CMP recently placed a Big-IP purchase order, and we'll be interested to hear about Colucci's experiences with the product down the road (we've including his comments about our top two finishers in the writeups below).

Although Sun's N2000 switch gave it the old college try, it just couldn't compete with the front-runners' triple threat of performance, flexibility and feature sets. And sadly, Array's TMX 3000 just wasn't up to the challenge. The switch performed fine in testing, but it lacks many of the features we feel are required of a critical edge device, including separation of management, more than two ports and the Layer 2-3 features necessary to earn the title "switch."

Colucci says: The available algorithms and rule sets were exhaustive. If we didn't find a canned iRule that matched our needs, F5's DevCentral site provided further resources. The F5 Big-IP 9 allows almost complete drill-down features to the TCP/UDP packet level with payload inspection, insertion and transformation control.Two years dark and a complete rearchitecture of the Big-IP platform have done wonders for F5's content switching line. A 44-GB backplane and a 4-GB uplink to its TMOS (traffic-management OS) equal greatly improved performance. The Big-IP handled an average of 50,000 gets per second while sustaining 20,000 TCP connections and processing 6,000 new connection requests per second, all with a sub-5-second total response time.

The Big-IP 9 sports 20 tristate copper ports and four SFP (small form factor pluggable) modules as well as a separate management port. Management is out-of-band and lights-out--even if the rest of the device is fried, you can still manage it--sharing only power with the rest of the switch. Even at spikes of up to 82 percent CPU utilization during a DDoS (distributed denial of service) attack test, the management console remained as responsive as ever. It also doesn't hurt that the interface is easy on the eyes.

Although previous incarnations of Big-IP have been stitchers--meaning they use delayed binding to intercept a request, determine which server to direct traffic to, and then bind the request flow to the correct server and get out of the way--this version is pure proxy, which furthers F5's vision of complete payload inspection for use in traffic-routing decisions.

VIP (Virtual IP) addresses, pools and nodes can be named, a small change we greatly appreciated when trying to find a server in a list of 300 nodes. The only gripe we have for this switch and the others we tested is that none could sort an interface list correctly. Interface 1.1 should not be followed by Interface 1.10, but rather 1.2.

F5 has always been a bit ahead of the game in terms of its rules capability, but it's really blown past the pack with this release. We liked being able to easily route traffic based on headers or payload and to perform actions based on these as well as internal counters, like CPU or memory. F5's new Tcl-based iRules language let us easily construct "contains" and negative rules, and lists of data--like the list of IP addresses in CMP's requirements to recognize some spiders and robots--can be created and referenced for cleaner rules. This was a far cry from Sun, which provided neither negation nor string-searching capabilities, and from NetScaler and Array, both of which required us to construct lengthy "if the IP address is this or this or this" rules to build our IP-based blacklist.

The biggest negative with iRules is that we still had to code by hand, and there's no validation mechanism. That's a dangerous combination: We wrote a rule that completely FUBAR'd the device's ability to process requests, making it reboot while traffic was flowing. F5 says it's working on a rule-builder GUI. This is one of the few areas where every other product but F5's shone--all included both a rules builder and a mechanism for verifying rules before they were applied.Big-IP 9 Application Switch, $69,990. F5 Networks, (888) 88-BIG-IP, (206) 272-5555. www.f5.com

Colucci says: The real surprise was the NetScaler unit, which I hadn't previously considered. I was impressed by the depth and scope of the NetScaler interface.

The NetScaler 9950 Application Delivery Switch is as much about massaging traffic as it is about directing it. The 9950 is the only device we tested to provide a nearly complete edge-traffic management system with compression, caching, link load balancing and an SSL VPN on top of its Layer 4 through Layer 7 load-balancing features. The only thing missing is true bandwidth management. Although we applaud the inclusion on both the NetScaler and F5 devices of rudimentary queuing to manage traffic, the functionality was just that--rudimentary. NetScaler offers no means to classify or truly prioritize traffic.

Still, the NetScaler switch lost out to the F5 only because of limitations with its policies, specifically those requiring a second internal VIP to handle content switching, and its slightly (and we do mean slightly) lower performance numbers in testing, mostly owing to our concern with high CPU utilization rates. The 9950 kept up with the Big-IP in terms of HTTP gets per second, but its latency was slightly higher and it ran at nearly 60 percent CPU utilization while doing so, twice as high as that of the F5 and Sun devices. Even so, we noted that high CPU utilization on the NetScaler switch had nearly no effect on our ability to manage it during testing, even though CPU load jumped to 100 percent when we turned on compression.

The 9950 is a four-port, tristate-copper device with a tristate copper management port. The management port is not separate--it can pass both ingress and egress traffic if desired, a design choice we find baffling. Sensibly, F5 and Sun completely separate management traffic from managed traffic; we hope NetScaler will follow this model in the future.NetScaler's Java GUI is easier to use than the CLI (command-line interface) and provides excellent graphing abilities, letting us monitor in real time just about every aspect of the device, from Layer 2 to Layer 7. One nit: NetScaler doesn't offer much help outside the manual, which we all know no one reads until it's too late.

Configuring Layer 7 rules was straightforward, and an included rule builder helped us create complex rules comprising multiple policies and expressions. Like F5, NetScaler let us recycle some rules into more complex rules, like the blacklisted IP addresses and user agents CMP required. And like Sun, NetScaler uses policy-based binding between services and content-switching rules.NetScaler's 2-GB in-memory cache has plenty of knobs, which helped us create caching policies based on anything from domain to the value of specific parameters within a request. Caching was not a CMP prerequisite but had been listed as a "nice to have," so we were glad to see the capabilities in the NetScaler and Array units. Caching can be manipulated from other programs using NetScaler's XML API. Enabling caching resulted in a minor performance increase, with most of the improvement showing as a decrease in latency with a slight increase in the number of HTTP gets per second.

NetScaler 9950 Application Delivery Switch 6.0, $104,999. NetScaler, (800) NETSCALER, (408) 678-1600. www.netscaler.com

Sun's N2120V Secure Application Switch was formerly Nauticus Networks' N2000. Sun acquired Nauticus in January 2004 and took the device off the market while manufacturing was brought up to Sun standards and some redesign and rebranding tweaks were made to bring the box in line with the rest of Sun's management look and feel.

Frankly, we're not thrilled with the result. We reviewed the N2000 when it still belonged to Nauticus and found it a solid little switch. Now, we worry about the future of this product, not because the technology isn't innovative or mature, but because of our experience with more than one recent release of bug-laden products by Sun and its disregard for integration needs outside its own family of products. For example, initial testing uncovered an ugly bug related to a last-minute change in the network processor code (provided by the device's network processor OEM) that caused the N2120 to forget to age entries in the flow table, leading to unacceptable latency as the cumulative number of sessions processed grew. Sun addressed the problem--and after a quick patch and then another to fix the same problem in the code handling trunking, performance was what we expected to see, near that of the devices from F5 and NetScaler in all aspects. Still, a bug affecting a core function should have been squashed before general release.

We've seen this acquisition/neglect cycle happen before, and though we hope Sun is the monolithic corporation that'll break the mold, we aren't holding our breath and hesitate to recommend this switch even though we liked some of its features. For example, when navigating through the GUI there is a CLI representation of the command used to show the same data on the CLI. It's a great learning tool. And Nauticus' vRouter and vSwitch concept is powerful, even if it is confusing to navigate at first.The N2120 has 12 SFP ports plus a separate management port. Because we tested the product as a half-NAT proxy, it was able to preserve the client IP address as CMP demanded without using the X-Forwarded-For header that other content switches would have required. However, this header is available to the N2120 when the switch is configured as a full proxy (see "Making Layer 7 Work for You,"). Like F5's and NetScaler's entries, the N2120 does TCP multiplexing, greatly reducing the number of necessary open connections to back-end servers--and the associated overhead of TCP setup/teardown.

Real-time monitoring/graphing and filtering are useful as well, if you can find them. We much preferred NetScaler's and Array's real-time monitoring implementations. They were easier to access and could present data in both raw text and graphical displays.We easily created object rules (Layer 7 rules) with the "object rule builder," a Flash tool launched from the GUI. It works well and lets rules be validated before they're used on the switch. The N2120 uses policies to tie a VIP to back-end resources. Response policies are available to modify HTTP headers or display a customized error page. This was unique to Nauticus and NetScaler two years ago, but now F5 also has added this functionality.

Sun N2120V Secure Application Switch, $47,495. Sun Microsystems, (800) 555-9SUN, (866) 596-7234, Ext. 18. www.sun.com/n2000

Array's TMX3000 is a pure proxy, converged-traffic-management system that offers memory caching, compression and an SSL VPN, among its features. The TMX3000 sports dual Gigabit Ethernet interfaces with no out-of-band management. CLI access is through serial or SSH/telnet.

In terms of configuration and management, Array's Web console is flashy, literally--it's based on Flash technology. As with the F5 and NetScaler offerings, we could control the TMX3000 using an XML API over HTTP. But unlike F5, Array eats its own dog food as the Flash interface takes advantage of the XML API under the covers. However, after using the Flash interface for a while, we decided that perhaps F5's decision not to use its own remote interface for managing the device was a good one. Although Array's interface is sexier than Sun's, it's also less responsive and more annoying than conventional Web consoles.

The TMX3000 is capable of 802.1q tagging, but it doesn't support 802.3ad trunking, and it lacks flexibility in Layer 7 switching. More than one rule can be applied to a VIP, but the first policy to evaluate positively is executed, with no opportunity to chain rules in an "if-then-else" scenario, which is what we needed to do to support CMP's traffic routing rule set. Because of this limitation, we had to create a bastardized chain of proxies and send traffic through not one, but two proxies to reach its final destination. We're certain this contributed to the TMX3000's lower performance numbers, because the device uses no custom silicon to handle network processing, leaving all the work to the TCP/IP stack.Array's dual Gigabit Ethernet interfaces limited us to a mere 2 GB of throughput, but after closely watching CPU utilization on the devices during testing, we're certain they can't handle that much traffic anyway. Array engineers were surprised when the TMX3000 handled only 6,618 HTTP gets per second with more than a 15,000-ms delay between the initial SYN and the final FIN. Marketware claims the device can handle up to 50,000 HTTP gets per second, but when we pressed them, engineers admitted that their marketing numbers were achieved using 1-KB files during internal testing.

Array's device impressed us in speeds and feeds with its in-memory caching. With caching enabled, the device served up twice the traffic (13,500 gets per second) with only 2,200 ms from request to full response. We'd suggest the TMX3000 for those with basic, low-volume Layer 7 switching needs, because it's easy to manage and less expensive than our review leaders.TMX3000, $41,985. Array Networks, (866) MY-ARRAY. www.arraynetworks.net

Lori MacVittie is a Network Computing senior technology editor working in our Green Bay, Wis., labs. She has been a software developer, a network administrator and a member of the technical architecture team for a global transportation and logistics organization. Write to her at [email protected].

When we heard our parent company, CMP Media LLC, was interested in purchasing a new Layer 7 switch, we suggested that its architecture and requirements could form the basis of our testing scenario for this review. After all, a previous joint venture was a win-win situation (see "Web Analytics Services: Inside Information,").

We tested switches from Array Networks, F5 Networks, NetScaler and Sun Microsystems and found they've matured--they're able to dig deeper into payloads and understand traffic flows within the application context, rather than on the packet-by-packet basis of their predecessors.

After testing performance, manageability and ease of configuration, we recommended that CMP choose F5 Networks' Big-IP 9 or NetScaler's 9950 Application Delivery Switch. F5's Big-IP won our Editor's Choice award by a nose, thanks to its consistent performance, flexible rule set and simplified configuration. CMP agreed, and recently purchased a Big-IP switch.We based our test bed on CMP Media's hosting environment, which includes the online presence of more than 10 magazines and 12 technology pipelines. CMP IT was evaluating content switches to replace its aging Nortel Alteon 184s and was concerned about support, performance and rules flexibility. Its base requirements included:

• Routing based on HTTP headers in a virtual hosting environment with the ability to perform limited pattern matching and IP address lookups

• 802.3ad (trunking)

• 802.1q VLAN tagging

We connected two Hewlett-Packard ProCurve 3400cl series Ethernet switches over a 10-Gbps link. To create a load for performance tests, we hung our Spirent Communications Reflector 2500 off one ProCurve switch to simulate back-end servers and our Spirent Avalanche 2500 on the other simulated clients. We deployed each content switch in a sidearm configuration, as per CMP's desired topology, using an 802.3ad trunk with five Gigabit Ethernet links, except in the case of Array Networks' and NetScaler's devices, which provide only two and four interfaces, respectively. We used five separate VLANs and enabled 802.1q tagging.Our servers hosted an assortment of images and articles pulled from multiple CMP publication sites, including InformationWeek, Network Computing and Intelligent Enterprise. We scripted the Spirent Avalanche to reflect a variety of requests, including downloaded binary content, and built our Layer 7 switching rules around differentiating between robots, spiders and real users using a number of techniques, such as HTTP header evaluation and blacklisted IP addresses and user agents. The average page comprised a text file and 15 images, ranging from 60 KB to 100 KB. The exception: downloaded binary content that weighed in at 500 KB.

All products were configured to support 13 virtual hosts on a single VIP and three additional hosts, each on individual VIPs. Each device was then configured with the requisite real servers, groups and rule sets. Throughout our configuration and tests, we evaluated the management consoles, both CLI and GUI, and took into consideration real-time monitoring and the switches' ability to integrate with an existing logging and systems-management infrastructure. We looked for features meeting CMP's requirements by configuring each device to log to an external SYSLOG server and looking for SNMP support options.

We then ran a series of tests on the base configuration before turning on features, such as caching and compression, where available. After additional features were enabled, we reran the same tests to determine the impact on latency, bandwidth savings and CPU utilization on the DUT (device under test).

To gauge the effectiveness of failover for each DUT, we configured a second DUT in an active/passive failover configuration. We then set off on a long test run, during which we killed power to the primary device. After the secondary device took over and had resumed a level of service equal to prefailover measurements, we restored power to the primary device and let it pre-empt the secondary, again evaluating how well the devices handled the situation. We tested serial- and network-based failover for those devices supporting both methods, but the primary scenario during our tests was network-based.

All Network Computing product reviews are conducted by current or former IT professionals in our Real-World Labs® or partner labs, according to our own test criteria. Vendor involvement is limited to assistance in configuration and troubleshooting. Network Computing schedules reviews based solely on our editorial judgment of reader needs, and we conduct tests and publish results without vendor influence.One easy way to enhance the end user experience is to compress HTML and associated text-based Web content.

It seems simple enough: Reduce the size of the content delivered over less-than-fat pipes, and you'll have faster response times and happier end users. Plus, you can save money through reduced operational costs. The downside? The overhead required to perform compression.

Compression in software is as CPU-intensive as SSL bulk encryption in software was five years ago. Our tests showed that enabling compression increased the mean CPU utilization on edge devices to the 90 percent range, an unacceptable rate for most data centers. So though compression might save your end users some time and might save you a few bucks on your ISP bill, it could result in higher out-of-pocket one-time expenses if you must add extra edge devices to handle the load. Given that WAN costs are recurring, do the math and determine whether the long-term ROI is worth the expense.

Even if software compression doesn't add up for you, there is hope as compression moves into hardware in the same manner SSL did. F5 Networks, like Array Networks before it, has introduced a PCIX compression card to assist its edge device with compression and take some of the load off the CPU. But it's not perfect. F5's technology combines software compression with hardware compression, depending on the state of the Big-IP at the time the request for compression is made.

In addition, hardware-assisted compression is limited by factors such as PCI and internal bus speeds, over and above the CPU hit. And vendors raise the price of their devices to cover the cost of hardware compression.Aside from a few updates in WebOS 22, Nortel Network's 2424-SSL isn't much different from the device the company acquired when it purchased Alteon WebSystems in July 2000. The Alteon was a strong performer, built on a solid architecture, but time waits for no switch: Until Nortel invests the resources to bring this device in line with rivals, we cannot recommend it.

The 2424-SSL's quirks signal that it's still targeted at carriers, not enterprises. The GUI hasn't been significantly improved in years, and it's an eyesore compared with the sleek interfaces provided by Array, F5 and NetScaler. The CLI is still a familiar friend, however.

Although Nortel engineers worked closely with us to complete our tests, the company's dedication to the Alteon line seems to be waning; we suspect Nortel's overall health has adversely impacted its ability to support this product correctly. Robert Colucci, a network engineer in CMP Media's IT department, echoed our concerns, saying support for CMP's existing Alteon devices has been sorely lacking.

Bottom line: Although the Alteon may still be viable, Nortel's inability to support and enhance it means its days of being on purchasing managers' shortlists may be numbered.

Content Switches

Sorry,
your browser
is not Java
enabled

Welcome to NETWORK COMPUTING's Interactive Report Card, v2. To launch it, click on the Interactive Report Card ® icon above. The program components take a few moments to load.

Once launched, enter your own product feature weights and click the Recalc button. The Interactive Report Card ® will re-sort (and re-grade!) the products based on the new category weights you entered.

Click here for more information about our Interactive Report Card ®.