Deep Inspection Firewalls

Once we'd laid in a few cases of Red Bull, we put on our enterprise firewall admin hat and focused on answering these questions: How effective at stopping attacks are the new DI-capable firewalls, and what impact does DI have on performance? How robust is the failover component? What capabilities are provided for those who need to manage a large number of firewall rules? What audit, logging and troubleshooting capabilities can we expect? And finally, what do these bad boys cost?

Our invite list reads like a who's who in the enterprise firewall arena. However, after reviewing our testing criteria, only six top-level vendors were both qualified and willing to participate. We were pleased to welcome Check Point Software Technologies' Next Generation With Application Intelligence, CyberGuard Corp.'s TSP 7100 Security Appliance, Fortinet's FortiGate-3600 Antivirus Firewall 2.8, Juniper Networks' NetScreen-ISG 2000, Secure Computing's Sidewinder G2 and Symantec Corp.'s Symantec Gateway Security 5460 Appliance into our labs and consider these offerings squarely within the upper echelon of the enterprise firewall market.

3Com-TippingPoint, McAfee and the latest Gartner Group-anointed "visionary" of next-generation firewalls, iPolicy Networks, didn't have products that fulfilled our required feature set. Lucent Technologies sent us the wrong product and Nokia did not respond at all, while Cisco Systems, Crossbeam Systems, ISS, Nortel Networks, SonicWall and WatchGuard declined to participate.

The Testing

When we started developing our testing criteria, we thought the hard part would be building a test bed that could mimic an enterprise network and traffic load. But after a month and a half of testing, it was the third component--enterprise-scale troubleshooting--that came back to bite us. Problems ranged from misconfigured firewall rules to a demonically possessed switch that killed two days of testing before we finally replaced it.

This is not the first time Network Computing has taken on a large-scale firewall bake-off (see "Defense Mechanisms,"), and we've learned from past errors. Given the complexity of our test infrastructure, the myriad firewall rules involved and the complexity inherent in tweaking the DI functionality in these firewalls, we decided to provide all required test configurations and rule sets to the vendors in advance, hopefully eliminating the risk of misconfiguration. Still, only Check Point and CyberGuard delivered their firewalls on time, with rule sets configured correctly. CyberGuard was the standout here: Its firewalls were completely ready to go, with our different configurations on separate hard drives, making our job easy. The company also sent three servers, just in case something went wrong. As Murphy's law would have it, one of the firewalls did get damaged beyond recognition, thanks to FedEx.

In addition, we let vendors have an SE on site for half a day before testing commenced to ensure that everything was working. But despite our precautions, most vendors had problems configuring their firewalls in accordance with the required rule sets. Some were never set up for high availability, some were missing large segments of the rule sets, and one firewall, Symantec's 5460 Gateway Security, showed up without any configuration at all.

Today's enterprise firewalls are loaded with a dizzying variety of features and capabilities that can confuse even the most well-educated IT decision-maker. Therefore, we put together test criteria that went beyond throughput to address the most pressing question raised by firewall administrators, IT directors and CTOs: What firewall will optimally protect my environment while meeting all my fundamental criteria? Our key testing areas were:

• Manageability: Although most firewalls have decent management interfaces, our definition of "decent" changed dramatically when we went above 100 firewall rules. First and foremost, we wanted to investigate how administrators who manage enterprise firewalls where logs are large and rule sets are larger would fare with each device under test. Among our considerations: What browsing capabilities are available to track down that one pesky customer IP address and validate whether a transaction successfully passed through the firewall to the Web server? How easy is it to audit logs? Can administrators browse the management interface and intuitively track down possible rule set modifications to isolate a particular issue?

Of course, the management interface doesn't stop at the GUI. So we asked what additional components are available to the firewall administrator for troubleshooting and firewall management. Do simplified command-line and console tools exist? Another component we investigated, after running into problems multiple times during our tests, was being able to view and modify firewall rule sets in readable text format. Having to modify 100-plus rule sets through a proprietary GUI interface on a line-by-line basis isn't pretty.

Check Point's Next Generation device was tops at managing large rule sets--in fact, its Secure Console was easily the best management GUI tested. All the right tools are in the right places, creating firewall rules was easy, and managing more than 100 firewall rules through the GUI was convenient. The Secure Computing Cobra management interface also provided a very nice interface for managing large firewall rule sets and multiple firewalls. Juniper's NetScreen-ISG 2000 was a mixed bag here, though if you've managed a Cisco router you'll be right at home with its intuitive CLI, which provided easy-to-use network troubleshooting capabilities as well as backup and import features. However, as with most of the devices tested, after we progressed past the 40 firewall-rules mark, things got difficult. Overall, vendors seem to hit a wall when it comes to representing rules that span multiple screens. Many opted for the multiple-page route, a poor choice if you regularly troubleshoot client connectivity problems early in the morning.

Bringing up the rear in this category is Symantec's 5460 device, which made extensive use of Java in the interface, lagged in troubleshooting and lacked a CLI. An overall observation: Even though the ASIC-based devices from Fortinet and Juniper had impressive CLIs, they need improvement on their graphical management interfaces.

• Performance: These are all enterprise-class firewalls, so we subjected them to enterprise-level stress conditions--a large number of NAT requirements, hundreds of concurrent clients that needed to perform transactions to the DMZ traffic and crowds of users executing various live Internet transactions. In addition, we were curious about the performance and firewall impact inflicted by enabling the DI features.

Our performance testing matrix consisted of running Spirent Avalanche-Reflector pairs configured to request or serve live content using a step-based load profile. We set the Avalanche load profile to ramp to a particular transaction-per-second rate; hold it for a steady period; and then ramp to the next transaction-per-second rate, again holding steady. We repeated until a throughput of 1 Gbps of combined throughput was achieved across all three firewall interfaces--DMZ, Internet and internal. We performed this test in three consecutive runs with DI disabled, then three with DI enabled, and calculated the average to determine the firewall's actual performance. Here no one could catch Juniper's NetScreen-ISG 2000. Although the CyberGuard TSP 7100 nipped its heels with no DI and the Fortinet FortiGate-3600 held its own with DI enabled, for consistent top speed the NetScreen was untouchable.

If a firewall could not sustain the traffic necessary to reach 1 gigabit, we noted the point of failure, then performed a binary search to find the exact throughput level that could be sustained with all transactions being successful. We implemented the binary search by varying the load profile until the device could sustain the traffic without serious packet loss. We experienced the full array of failures as we pushed the higher echelons of throughput, including extremely high latency, firewalls unable to establish new connections and even some unable to respond at all until the test was stopped.

• IDS/IPS: The next phase of testing flexed the devices' IDS/IPS features by sending a moderate level of traffic through the firewall, then running malicious content from the Internet/external segment at vulnerable hosts, which were located in the DMZ. Why only 10 percent of the overall rating appointed to inspection depth? We thought that if inspection depth were to get a higher rating on the overall scale, presenting only five known vulnerabilities would not have earned the scoring, and ramping up much beyond that would have gone beyond the scope of this article. It was important to verify not only that the IDS/IPS components were working, but how they were doing so.We threw LSASS, RPC DCOM, Unicode, IIS ISAPI, Warftpd and WuFTP attacks at the firewalls. Only Symantec's, Fortinet's and Check Point's devices stopped everything. The Symantec 5460 performed very well in this test and was armed with an arsenal of IDS signatures, application proxy capabilities, anomaly detection and antivirus functionality. The FortiGates also performed well from a DI perspective. If there was an IDS/IPS feature out there, the FortiGate had it. Fortinet provided not only hundreds of IDS signatures, but also a packet anomaly engine; both the signatures and engine were fully customizable.

The FortiGates easily detected, blocked and logged all the vulnerabilities we tested. In contrast, we dinged Check Point for a half point here because its firewall does not have the extent of signature and IDS capabilities of the Fortinet and Symantec devices. On the far end of the spectrum, CyberGuard's TSP 7100 blocked only three out of the five attacks. We wanted to see how Juniper's long-awaited IDP module would fare in this department, but it wasn't available in time for our testing.

• VPN: VPNs are an essential component of an enterprise. Because all the evaluated firewalls have VPN capabilities, we decided to investigate the capacity at which these firewalls can function as VPN gateways. Our tests consisted of configuring the devices as VPN tunnel destination points in a site-to-site configuration, then using 72- , 1,024- and 1,518-KB files to test their VPN capabilities. The 72-KB file represented IPsec overhead, the 1,024-KB file mimicked a standard VPN data packet, and the 1,518-KB file forced the VPN devices to fragment the packets prior to IPsec conversion. Here, as with performance testing, Juniper's NetScreen blew away the pack, with CyberGuard's and Fortinet's products running second and third, respectively, and Secure Computing's Sidewinder hobbling along in the NetScreen's dust.

• Failover: There are two primary types of failover: stateful, in which existing sessions should not be dropped, and stateless, in which sessions are dropped but connections can be re-established. With the complexity and demands placed on enterprise firewall reliability, we opted to test stateful firewall failover with the expectation that the devices would successfully fail over with no packet loss. If only the rest of the testing went as smoothly as this! Failover tests were conducted under a medium traffic load; we simply pulled the uplink cable on the main firewall and verified traffic loss and failover time on our Spirent gear. We had to perform a little tweaking to ensure that all the rules, especially HTTP traffic, were configured for stateful failover. To our delight, all the firewall clusters failed over successfully within three to four seconds with zero packet loss, giving us the opportunity to focus on another important criterion, namely logging.• Logging: We consider logging an essential component of any firewall--after all, what good is a security device if you can't easily audit it or track down possible events to verify whether you've been scanned or even hacked? We believe that essential firewall logging features (many of which we validated on a day-to-day basis while troubleshooting) include the ability to search logs from a specific source, destination and protocol; easily validate whether a client can perform a transaction; determine the last unsuccessful authentication attempt; and manage logs without undue effort. Heck, many of these items aren't just desirable anymore, they're required by rapidly evolving regulations. Being able to schedule automatic firewall log rolling and alerts is also becoming a necessity for firewall administrators. Check Point led the pack here, letting us easily search for, troubleshoot and monitor specific log entries. The other devices performed reasonably well, though Juniper's NetScreen and Symantec's 5460 both lacked effective log-search capabilities.

• Price: Pricing is one of those limiters that sideswipe you, derailing many a project. As much as these devices varied by feature set and performance, so was cost all over the map. One requirement for participation was that the vendors provide us with pricing for the firewalls as tested. This included two devices with Gigabit Ethernet cards, accelerators and any additional licensing costs. Because these firewalls are sold as security gateways that perform DI, we thought it important to rate them on a "bang for your buck" basis, so we came up with a table of dollar cost per megabit of DI throughput (see chart left). Because we required vendors to provide redundant hardware so we could test failover, this "as tested" pricing includes two devices. We looked at product price only after all testing was done, so as not to exert undue influence on our subconscious.

Fortinet led in bang for the buck; a pair of high-end devices will set you back just $59,990. Only Check Point quoted a lower price, but its Next Generation firewall lagged well behind Fortinet's in performance. The majority of the vendors demand additional costs for additional features. Although the IDS signature subscription was a relatively affordable option for all vendors, antivirus and spam-protection modules were often cost-prohibitive.

• Intangibles: Readers who don't deal with firewalls day to day may shake their heads at the doglike devotion some veteran administrators feel toward a specific brand, even though it might not be the fastest or have the largest feature set. The logic behind this loyalty is actually quite simple: It's the comfort factor--knowing the quirks and benefits of a specific device, and being able to troubleshoot client issues at 3 a.m. without having to first ingest three Red Bulls outweighs having all the latest bells and whistles.And by "comfort," we don't mean just availability or performance, but the knowledge that the device is working the way you expect and has the security coverage necessary to protect your environment. In the world of DI firewalls, dependability harkens to a sense of security in knowing you can easily respond to a possible intrusion or customer issue, and can validate your troubleshooting through logs and audit trails.

With these ideals in mind, we award our Editor's Choice to Juniper's amazingly fast NetScreen-ISG 2000. The NetScreens just kept on going, like big square Energizer bunnies. Fortinet and Check Point took the second- and third-place slots, respectively. Fortinet's very affordable FortiGate blew us away with its DI capabilities, and Check Point's NG provided a good management interface and great value for the money.

The CyberGuard is a nice device as well, but it was hindered by a high price. Secure Computing's Sidewinder needed some extra oomph on the performance side, while Symantec's downfall was its difficult management interface. All prices are as tested and include two devices.

Juniper provided us with two NetScreen ISG 2000 firewalls and an NSM (NetScreen Security Manager) server. These appliances sport a module-based architecture and were the first to be run through the gauntlet. They remained the class of the test, leaving nothing but dust in their tracks and outperforming rivals in both our firewall and VPN performance tests.

Configuration was a breeze after we figured out where to disable the autonegotiation feature on the interface. The NetScreen's native HTTPS interface was easy to use and gave us all the necessary tools to manage tasks from load balancing to firewall rule creation, but we performed most of our firewall management using Juniper's new NSM interface, which was designed to make management of multiple firewalls easier within a large enterprise environment. We found NSM easy to navigate and replete with all the necessary functionality.However, a couple of areas could have cost Juniper the crown. First, managing large (more than 100) rule sets was a tiresome process that involved manually navigating through multiple pages. Auditing a firewall over multiple pages of firewall rule sets got extremely tedious as well. In addition, the logging environment did not provide all the desired functionality. For example, searching for specific log entries was difficult and cumbersome, and we still don't know how to roll the logs. On the other hand, only the Juniper and Fortinet devices let us export their firewall configurations to text format for editing and modifications, saving us a lot of time.

Juniper Networks NetScreen-ISG 2000, $108,100. NetScreen Security Manager software costs $4,600 to manage 10 firewall/VPN devices. Juniper Networks, (866) 298-6428, (408) 745-2000. www.juniper.net

The FortiGate-3600 firewalls have a lot of geek appeal. Of all the HTTP/HTTPS interfaces we encountered during this review, Fortinet's HTTPS interface was the easiest to navigate and use. Although we still were stymied when trying to browse 100-plus rule sets, we liked Fortinet's intuitive method of portraying rules by traffic flow.

Unfortunately, Fortinet shipped the FortiGate devices with half our required configuration rules missing. This would have been a daunting problem for the SE had it not been for the slick text-viewable firewall configuration that Fortinet uses. A little script here, a little Vi there, and we had all the NAT addresses and rule sets configured in a moderate amount of time. Although the log viewer had a searching capability, we missed a decent real-time log-viewing and troubleshooting interface.

The FortiGate led the pack in DI--we found the device packed with IDS/IPS features plus hundreds of IDS signatures and a nice packet-anomaly engine, all customizable. In fact, we ended up burning the midnight oil playing around with the wealth of IDS signatures.From a throughput perspective, the FortiGate seemed solid until we reached approximately 750 Mbps of throughput, at which point the firewall could not open new transactions. Interestingly, when the FortiGate devices were pushed to the limit, they couldn't recover for a significant amount of time even after we stopped testing. Ten minutes after the test was stopped, the FortiGates were still processing more than 300,000 entries within the state table, with CPU usage above 86 percent.

The FortiGate came in first on price and in our "bang for your buck" value calculations. This firewall is simply affordable, coming in a close second overall, thanks to decent troubleshooting tools and exceptional DI features. Just make sure you don't push it past its comfort zone.

FortiGate-3600 Antivirus Firewall 2.8, $59,990. Fortinet, (866) 868-3678, (408) 235-7700. www.fortinet.com

Our experience with Check Point's Next Generation (NG) firewall was positive overall; the device turned in consistently good numbers except for performance, and the Check Point Secure Console GUI let us manage huge rule sets with ease and had more tools than Bob Vila. If anything, we thought there were too many features and applications packed into the interface, but it was still the most useful management app we tested. On top of that, the NG logging interface was also a winner: We could monitor real-time logs, search for any possible field available within a log, and easily filter data to troubleshoot and monitor for specific log entries. In addition to an effective GUI interface, the Linux-based OS under the hood provided a familiar playground when we needed to do packet captures, network testing and high-availability monitoring.

Although it detected all our tested vulnerabilities, the NG does not have the extent of IDS signatures we found in Fortinet's and Symantec's gear. In addition, the NG firewalls earned just fourth place in performance: We couldn't get close to even half a gigabit of throughput, even without the DI rule set.From a cost perspective, though, the as-tested price of these firewalls made them extremely attractive. At $41,556 for the cluster, soup to nuts, we found the NGs very competitive. Between its management interface, IDS and cost scores, Check Point has a real opportunity to keep the ASIC contenders looking over their shoulders.

Check Point Next Generation with Application Intelligence, $41,556. Check Point Software Technologies, (800) 429-4391. www.checkpoint.com

These boxes were huge--it took two of us to put them in the rack. If it were possible to intimidate attackers by sheer bulk, the TSP 7100 would be our pick.

The Linux-based 7100 has some nice features as well. For example, all the functions available on the GUI can also be reached from the command line, and we liked the parsing interface, all organized within a single directory. We could either manage the firewall remotely or run the GUI locally on the server. Ethereal, which we used extensively while troubleshooting a bad blade on our switch, is also available right from the local GUI. We wish all firewalls had this feature--it's amazingly helpful.

From a logging perspective, the CyberGuard firewalls were right behind Check Point. During our vulnerability testing, we got the opportunity to enjoy the device's extensive auditing and logging capabilities. We could run numerous audits to investigate possible intrusions or malignant firewall traffic, and at the same time take advantage of convenient filtering parameters. Both local and remote management is done through an HTTPS interface, which had limitations when we tried to manage more than 100 firewall rules. There was no effective way to browse and view large rules within the screen-size interface, making it difficult to find and modify rules lower down the rule set.From a throughput perspective, the 7100 was the fastest of all the non-ASIC firewalls. CyberGuard firewalls support application proxy and stateful packet filtering functionality, and though the company does not advertise the 7100 firewall as a DI appliance, these units definitely held their own. Without the DI rule set (application proxy features in this case), we could push the devices to well over a gigabit and 160,000 sessions. However, the minute we used the DI configurations, the firewalls took a significant hit. Ultimately, we couldn't nudge them past 90,000 sessions when we had the DI features enabled.

Be aware that big honking boxes and fast performance come at a cost--what we consider an excessive $156,000 for a clustered firewall landed the CyberGuard 7100 device near the bottom in terms of price. If not for that, the product would have given Check Point some serious competition for third place.

TSP 7100 security appliance, $156,000. CyberGuard Corp., (954) 375-3500. www.cyberguard.com

We really had the opportunity to get our hands dirty on the Sidewinder setup. High availability was not configured, firewall rules were missing, and the entire configuration for the non-DI rules set was MIA. We got on the phone with Secure Computing and found out that the Sidewinder firewall essentially does intrusion detection when using application proxies; the vendor provided us with a nonapplication-proxy firewall rule set a couple of days later.

Secure Computing's proprietary GUI, which can be installed on Microsoft Windows or Linux, was simple and intuitive. Although it was better than most of the HTTPS interfaces tested, we ran into a few problems performing what should be simple tasks, like saving rule-set modifications (where is that button?) and performing in-depth log searches. We found the command-line tools easier to use.The Sidewinder had various settings in regards to TCP anomaly detection and was the most sensitive to SYN-flood attacks. The Sidewinder stopped the majority of our vulnerabilities, but it lacked an extensive signature list. On the performance side, however, we couldn't push the device past 16,000 consecutive sessions while running the DI firewall rule set. The Sidewinders came in last in performance as we kept on hitting the ceiling on the state tables.

Sidewinder G2 Security Appliance Model 2150, $71,800 for two appliances. Antivirus, spam filtering and Web filtering are on a per-workstation basis. Secure Computing Corp., (800) 379-4944, (408) 979-6572. www.securecomputing.com

Symantec's 5460 hardware scored high on the aesthetic geek-appeal-o-meter: These silver-banded devices sport small front panels that let us easily monitor system status and configure them for quick deployment. We managed the 5460 using a well-organized HTTPS interface. Had it not been for the extensive use of Java, we would have rated this area higher, but as it is the interface is slow and sluggish, making troubleshooting difficult. For example, when we had the firewall under significant load, we sometimes couldn't determine whether the last submitted command was being processed. Unfortunately, the 5460 also lacked a command-line interface, forcing us to use the unresponsive GUI.

If all we wanted to do was basic functions, managing logs was easy. Trying to resolve a problem with the logs was a different story, however. The cryptic descriptions made our heads hurt, and we ended up spending more time trying to troubleshoot the Symantec firewall than any other device in this review.

During our DI testing, the 5460 kicked butt. We were pleased with the complete IDS signatures and application proxy, anomaly detection and antivirus capabilities. From a DI administrative perspective, we could easily enable and modify each signature individually, and we had to flip just one switch to turn the firewall from packet filtering to DI mode. From a non-DI performance perspective, we were able to push these firewalls to 24,000 concurrent sessions, at which time CPU usage hit 100 percent and successful transactions became extremely sporadic.Symantec Gateway Security 5460, $68,000 as tested. Symantec Corp., (800) 441-7234, (408) 253-9600. www.symantec.com

Adrian Peters is an engagement manager for Chicago-based security consultancy Neohapsis. Write to him at [email protected].

Michael Jones is a lab technologist for Chicago-based security consultancy Neohapsis. Write to him at [email protected].

For this portion of our Firewall Blowout, we tested six Gigabit-capable network firewalls that support high-availability stateful failover, VPNs, centralized management and DI (deep inspection) in our Chicago Neohapsis partner labs. We closely examined management capability--for example, how easily the products let you create and manage large rule sets. Performance was also critical--after all, what good is VPN and DI functionality if it chokes throughput?

Check Point Software Technologies' Next Generation, CyberGuard Corp.'s TSP 7100, Fortinet's FortiGate-3600, Juniper Networks' NetScreen-ISG 2000, Secure Computing's Sidewinder G2 2150 C and Symantec Corp.'s Symantec Gateway Security 5460 devices all offered decent management. It was performance that separated the best from the rest. Our Editor's Choice award went to Juniper's blazing NetScreen-ISG 2000, while Fortinet's well-priced FortiGate-3600 held up under our bruising tests to come in a close second.How We Tested

We used Spirent Avalanche and Reflector, SolarWinds' Engineer's Edition Toolset, and vulnerable Linux and Windows VMWare images to test the various components of the performance infrastructure. We sent all required test configurations and rule sets to the vendors in advance, hoping to save time tweaking the devices and reduce the chance of a misconfiguration. The test setup was derived from actual enterprise infrastructures we've encountered, cobbled together to simulate a simple yet large enterprise network implementation. We required vendors to provide redundant hardware so we could test failover.

Our three-tiered test-network architecture comprised an Internet, a DMZ and an internal network. The Internet segment consisted of thousands of external clients, external customers and external Web servers. The DMZ comprised hundreds of servers: Web, DNS, SMTP and FTP were all represented. The internal network consisted of Web, DNS and SMTP servers. The firewall rule set provided the following access: HTTP/HTTPS, FTP and SMTP access for external clients to the DMZ; HTTP/HTTPS, FTP, DNS and SMTP access for external customers to DMZ and internal servers; SMTP access from the DMZ to the internal network; and HTTP/HTTPS and SMTP access from internal clients to external servers.

All HTTP/HTTPS transactions were live Web content sized at 4 KB, 16 KB and 64 KB. Content (text/ HTML only, no images) comprised a 4-KB Google home page, a 16-KB AltaVista home page and a 64-KB CNN home page. FTP transactions were each standard 10-KB files, and the SMTP transaction consisted of a 2-KB e-mail file.

Our next step was to accurately depict the average enterprise traffic mix of the various protocols. We used CAIDA (Cooperative Association for Internet Data Analysis) statistics of Internet usage and traffic mixes from 2001 and 2003 to create a percentage-based hierarchy of mixed protocol traffic. After investing a small fortune in Red Bull, we had our testing environment ready for action.We conducted our VPN throughput tests using Spirent's SmartBits SmartFlow positioned behind each firewall, creating IPsec tunnels in each direction. The testing matrix consisted of a binary throughput search of each of the particular packet sizes until packet loss occurred. Our next task was to perform step testing within the binary results to narrow down the actual point of failure.

All Network Computing product reviews are conducted by current or former IT professionals in our Real-World Labs® or partner labs, according to our own test criteria. Vendor involvement is limited to assistance in configuration and troubleshooting. Network Computing schedules reviews based solely on our editorial judgment of reader needs, and we conduct tests and publish results without vendor influence.

R E V I E W

Deep Inspection Firewalls

Sorry,
your browser
is not Java
enabled

Welcome to NETWORK COMPUTING's Interactive Report Card, v2. To launch it, click on the Interactive Report Card ® icon above. The program components take a few moments to load.

Once launched, enter your own product feature weights and click the Recalc button. The Interactive Report Card ® will re-sort (and re-grade!) the products based on the new category weights you entered.

Click here for more information about our Interactive Report Card ®.