We also invited IBM Corp., which declined to participate because the company is between releases, and Lucent Technologies, which never responded to our invitation. As for Network Associates, it put its left foot in, pulled its left foot out, put its left foot in and shook it all about. But all we got was hokey-pokey.

We wanted to look specifically at distributed multiunit management. All the firewalls we tested had some capability for remote management, but we found a wide variance in implementation. Secure Computing's Sidewinder could be managed securely by redirecting X Window over a client VPN, while the Cisco Secure PIX's Cisco Secure Policy Manager (CSPM) offers security-policy management regardless of the underlying infrastructure. With the exception of the Check Point product, the firewalls we tested required us to use their VPN clients for secure management, which meant using a Windows-based management station. We placed the firewalls on our Real-World Labs® network and lived with them. We found that, for the most part, remote configuration was fairly painless. We could hardly tell the difference between managing a firewall on the local subnet and managing it over our frame relay connections. However, what differentiated one firewall from the next was logging and reporting. Being unable to troubleshoot connections locally, we had to rely on the reporting facilities in the management stations.

We found sizable differences in this area. Reporting is crucial to firewall management because of the wide variety of attacks that can appear at the firewall. What at first may seem like anomalous behavior, such as a few rejected connection attempts, can turn out over time to be a slow port scan designed to evade detection. Until automated event correlation is developed, a network's best defense is the administrator who regularly monitors the logs. While tracking successful connections may not be terribly important on a daily basis, an increase in blocked attempts can signal anything from an active attack to misbehaving or misconfigured devices. Good logging should present enough information to administrators for them to know when to quickly scan for important events. More detail should be available if needed. The Check Point firewall beat the other products in its reporting capabilities because of the level of detail presented to the administrator. During testing, we found that we could easily determine where our configuration problems lay by examining the logs, finding the relevant entries, and then going to the rule that triggered the log entry.

All the firewalls we tested are ICSA-certified (www.icsa.net), and we configured the firewalls and underlying OSes to be as secure as possible by turning off services and patching the underlying OS. To test for vulnerabilities, we chose a few well-known attacks found in the wild to strike the firewall-protected servers, instead of using commercial products, such as ISS' Internet Scanner or Network Associates' CyberCop Scanner; vendors have hardened these products against such obvious ploys.

Our first step in security testing was to profile each firewall. We wanted to determine how much we could discover about the firewall and the services it was protecting, and whether we could bypass the firewall and attack the servers directly. When we scanned the firewalls directly with Fyodor's Nmap application, we discovered every firewall except FireWall-1, which remained dark to us. It's interesting to note that even though Nmap wasn't able to detect the operating systems of some of the firewalls, we were able to determine whether we were scanning a proxy server. This bit of information was revealed because a scan of the network protected behind a proxy firewall will come back with identical information for each IP address. Remember, proxies accept and make network connections on behalf of clients and servers, so proxies will always respond to connection requests for all the services they are proxying. Nearly all the firewalls detected the port scans.

With the exception of Raptor, the firewalls let us successfully attack our internal Web server, gaining a shell through the firewall (see "Acknowledging Firewall Limits," www.networkcomputing.com/1106/1106side3.html).

We used an attack that sent non-HTTP traffic over TCP Port 80 to our IIS 4 Web server. The traffic should have been stopped by the application proxies because the traffic didn't conform to the HTTP protocol. Axent's proxy did just that. Secure Computing's Sidewinder, also an application proxy firewall, let the attack through. Secure Computing acknowledged the problem and told us the company has known about it since October. By the time you read this, a patch should be available. The inspection firewalls didn't stop the attack because inspection firewalls don't examine the application traffic.

NetScreen-100 was the only real shocker in our performance tests. We had heard a lot about how fast it was in VPN performance, and it sure was: It hit a whopping 126 Mbps with 3DES and MD5 preshared secret IKE. That's nearly twice the throughput of any other VPN we've seen. The VPN tests also confirmed our assumptions that software VPNs can't keep up with hardware-based VPNs because they lack cryptographic acceleration or have bus limitations to deal with. Cryptography is resource-intensive, and hardware-based devices such as NetScreen-100 will outperform software VPN. And don't confuse hardware with appliance; the PIX Firewall 520 is an appliance, but the cryptography is done in software as well. We also expected to see a performance difference between the inspection firewalls (VPN-1 Gateway, NetScreen-100 and PIX Firewall 520) and the proxies (Raptor and Sidewinder), and we weren't disappointed. The inspection firewall throughput was around 150 Mbps, while the proxies came in around 60 Mbps, largely because the proxies have to check each packet in the application layer, which requires more processing. Thus, the trade-off is that proxies afford more in-depth processing, while the inspection-only route yields higher performance.

Once the dust settled, VPN-1 Gateway came away with our Editor's Choice award, primarily because of its mature management interface, excellent reporting and logging facilities, and good performance. PIX Firewall 520 came in a close second, based on its CSPM, reliable performance and good logging facilities.