Our test plan was straightforward and deceivingly simple: Deploy pairs of firewalls in a redundant fashion, configure them for HA stateful failover, blast thousands of real TCP sessions through them, pull the plug on any single device and see which fails over gracefully. Unfortunately, this was easier said than done, and our seemingly innocent plan turned out to be, once again, mission impossible. Three months later we found ourselves pounds lighter and tons wiser.

Have you ever wondered why veteran network administrators seem to have an unwavering sense of loyalty toward particular vendors? For anyone in the trenches, where real-world performance makes the difference between sleep and 3 a.m. pager calls, the answer is simple: proven reliability. Reliability to a network administrator isn't based solely on the device's tenacity for staying alive. Reliability is the device working the way you expect it to work. Reliability is your being able to easily instruct the device to do what you want it to do. Reliability is being able to access the device in a secure, unfailing manner. Reliability is the accessibility of tools and utilities an administrator can use to solve problems. Reliability is the knowledge that your security products are truly concerned about security, not just about performance. Reliability is knowing that your HA firewall solution is not going to drive you insane, keep you up at night, flake out, crash or give you another reason to bang your head against the wall.

Online Only Check out our Detailed Test Results for more in-depth vendor comparisons.

It is with these ideals in mind that we went into our testing, and it is based on these ideals that we recommend the Cisco PIX 535 and the NetScreen-1000 for HA firewall needs.

The Great Testing Debate

While the firewall industry is rapidly maturing, the state of public firewall testing is not. You'll find the Internet littered with firewall-performance tests based on odd packet sizes, bizarre traffic mixes, TCP sessions that are mangled or incomplete and other things you won't find in real-world environments. Part of this mess is due to testers' not giving their test beds a lot of thought, and part of it is due to a dearth of well-designed testing tools. While there is no substitute for live network traffic, there are ways of designing tests around real-world environments so synthetic traffic comes close to that of real-world conditions.

For our HA firewall testing, we turned to Caw Networks' WebAvalanche and WebReflector products. WebAvalanche is an HTTP client emulator capable of simulating thousands of clients simultaneously. WebReflector, which can run independently of the WebAvalanche, is the HTTP server side. WebReflector can service millions of HTTP requests and essentially function as a Web server on steroids.

We used the Caw units for five primary reasons. First, the traffic generated by the Caw devices is real TCP traffic: It uses real TCP sequence numbers, it opens and closes sessions properly, and the payload is actual HTTP traffic -- just like on a real network. Second, the Caw boxes let us create an unbelievable amount of valid traffic. Although we didn't focus on throughput performance testing, we were able to use the Caw units to generate thousands of concurrent TCP sessions. We maxed our tests out at 200,000 concurrent sessions, but the Caw devices are capable of more than a million. Third, the Caw boxes have robust reporting engines. Not only could we view the round-trip times of all sessions, we also could view TCP errors, unsuccessful HTTP requests (to measure orphaned sessions) and other statistics.

Fourth, the Caw units can emulate thousands of unique IP combinations. This is important for taxing state tables. We found that Class B client ranges (65,000 unique IP source addresses) knocked over most firewalls, so we stuck with six Class C ranges (1,500 unique source addresses). Finally, the Caw solutions are capable of supporting NAT (Network Address Translation) connections. NAT breaks many testing suites, so this functionality is important.

Getting the test bed stable was a challenge (see "Lessons Learned"). We had to tune and tweak the Check Point Solaris machines, the Nokia units, the StoneGate servers and our Cisco switches. We also had to construct scripts for the Caw units to run that wouldn't kill our firewalls. While we kept all our tests well under 100 MB per second, we made sure our session counts were high. However, it wasn't hard to topple even the gigabit-equipped firewalls with the Caws. Readers beware: just because it says gig on the box doesn't mean it can handle a full gig's worth of real-world traffic.