NIP Attacks in the Bud

That left us with three potential vendors: NetScreen Technologies, Network Associates and TippingPoint Technologies. TippingPoint declined to participate in our tests, claiming it didn't have a device for testing because all available units were promised to customers. (And we thought Dell had a tight supply chain!)

We set up NetScreen's IDP 500 and Network Associate's McAfee IntruShield 4000 in our Syracuse University Real-World Labs® and scrutinized their detection, management, performance and reporting capabilities (see "How We Tested NIP Devices"). We were impressed--though each device had some quirks, we were able to work around them without too much pain.

The Setup

We installed each product sensor in both inline and one-arm mode. The sensors were configured to communicate with one management server, and we used a management client on a workstation. Our criteria were simple: Products must detect attacks and anomalies accurately while minimizing false positives (alerts generated by legitimate traffic) and false negatives (letting attacks go undetected).

Effective intrusion prevention depends on effective intrusion detection. Any vendor claiming to have zero false positives is feeding you a line. There are too many attacks and variations of attacks, and too much legitimate traffic that looks like attack traffic, for a claim of "no false positives" to hold water. It's laughable, really.Sure, on our controlled test bed, we didn't find any false positives, but we didn't expect any because the traffic was manufactured to our exacting specifications. When we placed the devices on our live network, however, we initially had numerous false positives, including supposed application-protocol anomalies, such as a TTL of 255 in a RIP packet, various SNMP messages, HTTP header-length messages and apparent Unicode encoding generated from the Mac OS X AOL Instant Messenger client. Tuning reduced the false-positive rate considerably.

We are convinced both of these products will block known malicious traffic, though we did need to carefully choose which alerts to block because of the potential for blocking legitimate traffic.

However, we are not convinced these products are sufficiently accurate at spotting unknown attacks through protocol-anomaly detection. Application vendors don't always write network-protocol stacks that conform to standards, thus causing legitimate traffic to be pegged as anomalous and malicious. We can't stress this enough: You must know what constitutes malicious traffic, then use existing signatures or develop your own signatures to block it.

So should you buy a NIP device? If you're planning to implement a new IDS or replace an existing IDS, we recommend choosing a NIP appliance instead to take advantage of easy deployment, good conformance to stated performance claims and flexible network integration. Plus, you get the option of blocking attacks. But we don't recommend NIP in addition to an existing IDS deployment. Better to spend the cash patching server vulnerabilities--that's something you should be doing anyway. Throwing money in that direction is a more direct way to ensure that your servers are protected because even if attack traffic gets through, patched hosts should be immune. Blocking attacks is not as critical as proper patch management (see "PatchLink Helps Keep Windows Closed,", and "Patching Patch Problems").A properly tuned policy will reduce false positives, but tuning a detection policy means customizing the detection-signature set to suit your environment. Examples of tuning include:

• Enabling signatures that pertain to your servers. If Apache isn't installed on your network, do you really want to see Apache-specific alerts? We think not.• Ignoring alerts from specific hosts or subnets. For example, a vulnerability-assessment tool will light up a NIP device like a Christmas tree, so it's probably safe to ignore attacks from that host. Network-discovery tools that use ICMP, port scans, NetBIOS and SNMP to enumerate hosts will trigger a high number of alerts as well.

Although both NetScreen's IDP Manager and Network Associates' IntruShield Manager have robust tuning capabilities, they take radically different approaches. It's a Scylla and Charybdis thing: IDP Manager, which uses a rule-based paradigm, is familiar and easy to use but can easily lead to rule overload. IntruShield, on the other hand, relies heavily on policies applied to subinterfaces or virtual IDSs, raising the potential of lots of small, highly customized policies. We prefer a rule-based approach because it's the sea monster we know, but we recognize that rule bases can get complex. Individual policies are simpler conceptually, but changes need to be made multiple times, which adds complexity.

Reducing false negatives is a different matter. Factors contributing to false negatives include timeliness of signature updates, poorly written signatures and nonstateful packet inspection. If a signature doesn't exist for an exploit, it won't be detected, so clearly, it's critical to update signatures as quickly as possible. We'll qualify that statement by saying protocol analysis may spot unknown exploits by detecting, for example, violations in a protocol sending binary data, such as shell code, in an HTTP or SMTP header. But protocol analysis can just as easily raise false positives, so though it's a good early warning system, it's not a reliable method of detecting attacks.

An alternative to rapid updates is custom signatures. Although signature development is complex and best left to experts, a little knowledge of signature writing can let you block fast-moving worms or create custom signatures for your environment. Each product has its own signature-creation method, so there will be a learning curve to negotiate, but we found documentation pretty helpful.

Stateful packet inspection is almost a given in IDSs, and it's no different in NIP systems. Common games that attackers play when trying to evade signature-based systems include fragmenting packets into tiny pieces, splitting the protocol headers or attack into multiple packets, sending packets out of order and sending packets with unusual combinations of options. At the application level, encoding of HTTP URLs is a common way to obfuscate HTTP-based attacks. Packet reassembly and protocol canonicalization strive to present the IDS with the packets that will be seen by the destination system. IDP and IntruShield both perform stateful packet inspection and normalize traffic prior to signature matching, so these tricks were generally thwarted.Our expectations on detection were straightforward: The attacks we threw at the NIP devices should have been detected properly, even when we used evasion techniques like fragmenting and reordering. We weren't disappointed, exactly. On our test bed, the devices performed flawlessly, detecting all the attacks we threw at them. What we didn't expect were the rather odd false positives that cropped up when we installed the products on our live network.For example, we use SSL over SMTP because we authenticate to our mail server before sending e-mail. IDP triggered on there being binary data where SMTP commands should have been because it didn't recognize the STARTTLS command. We added a new signature that detected the STARTTLS command, created a rule that detected STARTTLS and then ignored the rest of the flow. We placed that rule at the top of the rule base so it would be triggered first.

Next, we had to deal with IDP thinking that our NTP server was UDP-scanning a remote host that was trying to synchronize time. We used WildPackets' EtherPeekNX to analyze traffic between the two hosts, which was NTP traffic while the IDP was alerting us about a UDP port scan!

IntruShield had a few quirks, too. For example, it claimed that SNMP traps (UDP 162) sent to the broadcast address were overly long NetBIOS name queries, and that SNMP responses were another form of UDP port scanning. Once we had things tuned (a never-ending process, really) we were able to track all those pesky attackers trying to break in. False positives happen, so we strongly urge you to thoroughly understand why an alert is being generated before you decide to block that traffic.

Something To Talk About

Before we decided to block traffic, we had to investigate the alerts and log entries we were seeing. In the deluge of events, information management is key. IntruShield excelled in this area with its simple but powerful tools to filter alert presentations and delve into specific areas. For example, IntruShield Manager let us drill down into the alerts via a variety of avenues. We had more than 15,000 entries, but we could sort them by individual protocols, eight of which carried alerts. That's a manageable starting point. After several weeks of monitoring, IntruShield detected 33 individual attacks, with the bulk of the attacks in the top 10 of the total 33 discovered. Individual views could be sorted by any column, and packet capture was available and predefined for certain alerts. You will need to install Ethereal or another protocol analyzer capable of reading pcap files.

IDP defined filters on the real-time event viewer to pare down the information, and it let us save the filters in a view, which could be used time and again. If you know what you're looking for, you'll be able to locate it easily. Unfortunately, we could not get a view of all the attacks detected, besides a Top 10 list. The predefined reports left a lot to be desired, and there was no scheduling facility to generate reports periodically--a feature offered by IntruShield.

After extensive testing, we found both products' protection measures reasonable. When installed inline, both IntruShield and IDP can drop packets and streams or send resets to the client, server or both. We enabled autoblocking on our live connection for those alerts we determined had a very low probability of false positives. These were generally HTTP-related encoding, directory traversal and command-execution attacks. We did not autoblock for servers where we expected some false positives. Your comfort level, and therefore mileage, may vary.

We gave our Editor's Choice nod to Network Associates' McAfee IntruShield 4000, largely because of its top-notch detection and report facilities, for which you will pay a premium. For sites with less than 500 Mbps of network traffic, NetScreen-IDP is certainly a contender. Its firewall-like rule base will be familiar to most administrators, and its reporting, after some massaging, was adequate.Scorching performance, robust attack detection and great analysis tools are the hallmarks of IntruShield 4000. Although it took us a few days to grok the management console, once we did, unearthing the appropriate data was a snap. The configuration can be simple, but once we started tuning policies to groups of hosts, we found the paradigm unwieldy. Not being able to specify arbitrary address ranges, addresses that are not within a CIDR block, and apply multiple policies to the same hosts made management a bit more complex. IntruShield performed fine up to the limits of our test equipment--1.2 Gbps, with latency averaging 1 to 2 ms.

IntruShield can be installed inline using a port pair or in one-arm mode taking traffic off of a switch span port or network tap. IntruShield can drop packets and flows while inline, but in one-arm mode, it can block traffic only through TCP resets or ICMP unreachables sent via a response port. Each interface can be configured to capture traffic using a different method, a level of flexibility that is not available with the IDP.

Policies, in IntruShield parlance, are where attack signatures and DoS (denial of service) detection are enabled and disabled. Each attack is defined to detect a discrete event, like Unicode-encoded URLs or binary traffic in a protocol header. Attacks are organized by protocol, so it's a simple matter to drill down into a policy and see what is enabled. Unfortunately, Network Associates doesn't let users see what constitutes a signature. When we asked about this, the company said it didn't want to help people develop evasion techniques. The Exploit Alert Detail dialog on the Alert Viewer reveals text matches for a given alert, but that one match could be a subset of all possible matches.Given time, we could have puzzled out most of the signatures via exhaustive searches, so we think Network Associates is just being difficult. In comparison, NetScreen opens signatures for review and editing--an approach we prefer.

The lack of signature information quickly became frustrating, and it complicated troubleshooting when a match was based on a protocol anomaly because there wasn't enough information to know why a match occurred. We had to send packet traces to Network Associates to determine why an SNMP packet was being detected as a NetBIOS issue. It took a few days, but the company resolved the problem and provided an update to the signatures. Signature updates are automated, but you need to buy a support contract to get them.

Policies All Around

IntruShield's policy assignment is very flexible, which is a double-edged benefit. We could apply policies to individual interfaces or, when in inline mode, to interface pairs, and there were policies defined for both outbound and inbound traffic, so asymmetric detection was possible.

Policies could be tuned in two ways. A rule set could make wholesale limitations in what types of attacks were detected by limiting the policy to signatures that match a set of categories, protocols, OSs, applications and other classifications. For example, we created a rule set that included all categories of attacks against IIS Web servers over HTTP. We then defined an IIS Web Server Policy using that rule set. All signatures that did not match the criteria were not selected for the policy. The alternative to using rule sets is to enable/disable individual attacks in the policy manually; however, if you want a symmetric policy, you must edit both the inbound and outbound attacks.Policies can be applied to individual hosts or groups of hosts, but this process was more complex than with IDP. Subinterfaces are used to group hosts by CIDR blocks or 802.1Q VLAN tags. Unfortunately, all hosts may not be in the same CIDR block or virtual LAN. So we had to reconfigure our host IPs to group similar hosts within a CIDR block or add hosts individually to the subinterface using a 32-bit subnet mask. We chose the latter because renumbering is a difficult process. Also note that CIDR blocks can be defined only once per interface (see "Policy Assignment Per CIDR Block,").

We had a vulnerable Microsoft SQL Server 2000 on 192.168.2.40. We successfully ran the Resolution Service Stack Overflow against that server, and it was detected by IntruShield under the policy assigned to the CIDR block 192.168.0.0/16. We then ran the same exploit against our Sun Solaris box, which is governed by the Solaris policy assigned to the subinterface containing the CIDR block where our Solaris server resides. That attempt was not alerted because the signature governing that address was not enabled.

The moral of the story: Tuning policies reduces false positives and lets you focus on relevant alerts.

Reporting

Once we completed our configuration, we ran a number of exploits against our target hosts. This gave us an opportunity to examine IntruShield Manager's reporting and data-mining features.The Alert Viewer provided us with real-time and historical views of alerts, while consolidated graphs showed rollups of alerts by severity, top attacks, and source and destination IP addresses. The graphs are context-sensitive and changed as each bar was highlighted. Double-clicking on an attack bar, for example, brought up a new window that showed just those attacks.

The Real-Time Attack Details showed each event as it was logged. Double-clicking on an event brought up details pertinent to that event, such as an attack description; addressing information; and, if there was a signature match, the data that was matched on. Most of the alerts also captured the offending packets, which we then could view using Ethereal. Some attacks generated multiple alerts. For example, a Unicode-encoded directory traversal attack triggered three alerts--for Unicode encoding, for directory traversal and for running cmd.exe.

Real-Time Attack views can be frozen, and the columns can be sorted. The drill-down function provides a high-level categorization view from which to start. For example, selecting the attack drill-down showed us the 20 or so detected attacks and the counts for each. Clicking one of the listed attacks brought up a filtered dialog showing those attacks.

The reports are full-featured, and the scheduler offers daily and weekly report generation.

The IntruShield product line worked well as an IDS and, with administrative forethought, as a NIP device. The reporting is top-notch and, for the most part, informative. The policy configuration, while initially cumbersome, provides for granular tuning where required.McAfee IntruShield 4000, Network Associates, (972) 963-8000.www.networkassociates.com

The NetScreen-IDP 500 has many features comparable to those found in the much pricier IntruShield 4000 and has more straightforward management to boot. Firewall administrators will feel right at home with IDP's rule-based policy, which is flexible and tunable. IDP's sensor, like IntruShield's, can capture packets inline, using an external tap, or from a switch span port; however, it can capture packets using only one mode. Also like its rival, IDP can drop packets and streams while in inline mode, and it can send TCP resets or ICMP unreachables to block malicious traffic in one-arm mode as well.

The device's reporting leaves much to be desired, however, and narrowing down alerts to specific sets of data is not nearly as straightforward a process as with IntruShield. Signature updates are free but must be initiated manually.

NetScreen seems to have strived to make the policy-definition process familiar and easy--and it has succeeded. We barely cracked a manual during testing. Rules are defined using a set of specifiers, such as source and destination addresses or ports, optionally defining a server or configuring an action. Each rule can be applied to all sensors or to a specific one. We could have had one large policy with specific sensors defined per rule, or we could have defined multiple policies and assigned them to individual sensors. We chose to define individual polices per sensor for our testing because we prefer to segregate policies.

Rules Rule

We found IDP's rule-based paradigm especially useful when tuning the policy for our production network. Our goal was to have the broadest policy enabled while keeping false positives to a minimum. However, our production network is not what you'd call static: We have a wide variety of technologies running all the time, and new servers are always being installed and upgraded. For example, we have a number of network managers that periodically scan and enumerate Syracuse University's Class B network. These guys triggered tons of alerts, so we created a host group that contained those managers, added a rule that ignored them, and applied the policy, making the alerts disappear faster than audiences at a screening of Gigli.Unlike firewalls, which dispatch traffic on the first positive match, packets processed through IDP can fall through the rule base and be processed multiple times. But we were able to set rules as terminal, meaning once the rule matched, the action--be it "none," "drop packet," "drop sessions" or "send reset"--would be taken and processing would stop. Order matters and is intuitive.

We did have to add some signatures to the rule base to allow certain kinds of traffic. For example, as we noted earlier, IDP doesn't recognize the STARTTLS command within SMTP, which is used to start TLS for traffic encryption. Every time a TLS-enabled MTA (Message Transfer Agent) or MUA (Mail User Agent) negotiated with the mail server, an alert for a bad command would trigger, and subsequent SMTP packets would be alerted as having binary data in the header. We created a signature that looked for STARTTLS using a regular expression and then ignored the flow that resulted.

Unlike IntruShield's, IDP's signatures are viewable and editable. For example, the smaller IDP 10 product, which we included in our live network testing, kept alerting on HTTP traffic between IntruShield Manager and the laptop running the IntruShield management client. The signature was looking for binary data in the entire TCP payload and triggering. We located the signature and changed it to match only HTTP packets with binary data in the header portion. This edit ability let us easily refine the signatures so that they conformed to our normal traffic. Of course, we would never be cavalier about editing signatures because we know we could just as easily render them useless. But used wisely, NetScreen's signature editing is a powerful capability and a clear advantage over Network Associates' closed signature model.

Reporting Miasma

Although there are lots of things to like about IDP, alert viewing and reporting aren't among them. When we knew exactly what we were looking for, such as specific attacks, ports or hosts, we could define filters in the real-time alert viewer that narrowed the information flow to a manageable level. These filters could be assigned to views so that they could be reused, and the real-time alert viewer provided a filterable log of all events processed. Double-clicking on an event brought up a detail dialog box, which provided a brief description of the event and any applicable references. Right-clicking on the event brought up a context-sensitive menu with options, such as Filter, Show Data and Locate Attack in the attack objects or policy. The Filter options could filter on the column field selected, and subsequent uses of the filter narrowed the data further. For example, we were troubleshooting a POP3 issue, so we filtered first on the POP3 Command-Failed alerts, and then filtered those results by the host we were interested in.However, there's no decent top-level summary of all the attacks that were seen. The reports weren't interactive, for example, to let us drill down deeper; and there's neither a report scheduler nor a way to customize reports. These are significant weaknesses, given the large amount of data generated by IDP.

The IDP 500 also is limited to 500 Mbps; during testing, we hit 461 Mbps with latency reaching 1.7 seconds, which hurts performance.

NetScreen-IDP 500, NetScreen Technologies. (800) 638-8296, (408) 543-2100. www.netscreen.com

Mike Fratto is a senior technology editor based in Network Computing's Syracuse University Real-World Labs®; he covers all security-related topics. Prior to joining Network Computing, Mike worked as an independent consultant in central New York. Write to him at mfratto@ nwc.com.

Post a comment or question on this story.We devised two different scenarios: We started testing by placing the NIP system into a controlled lab network with known characteristics. Once we operated the system and learned how to use it, we placed a second system in one-arm mode on our production network with live traffic.On the clean test bed, we tested both signature detection and performance. Security testing involved grabbing public script-kiddie tools associated with network-based vulnerabilities in the SANS Top 20 (current vulnerabilities found in recent months). We modified some of the attacks so that they were still successful, but not stock.

We used Nessus and nmap for reconnaissance and ran the exploits across a router, ensuring that the NIP systems properly detected and alerted. Next, we used fragroute to split packets into tiny fragments and reorder packets in an attempt to evade detection. We then ran traffic through the NIP devices and reran all the tests to see if the attacks would still be detected. They were.

To test performance, we measured performance up to the rated capacity--or until we exhausted our rated capacity. We used Spirent WebAvalanche as a Web client and a Spirent WebReflector to mimic a set of Web servers. Each user requested a page consisting of a main page 8 KB in size and 10 subelements at 6 KB each, for a total HTTP payload size of

68 KB. We initially used HTTP/1.1 with keep-alive and ramped up the connections per second. Both devices under test passed all traffic with additional latency. We then switched to HTTP/1.1 without keep-alive so that each GET required a network TCP connection. Again, both products passed traffic with the same result.During our tests, we deployed a number of intrusion-prevention systems on our production network, taking some lumps in the process. Here are our notes from the school of hard knocks:• Disable all blocking until you understand exactly what will be blocked.

• Err on the side of caution. If your IPS has the option to block just a single packet or stream, let it do just that. If your IPS can shun an IP address for a period of time, be careful that you don't set it to block for too long, potentially shunning future legitimate traffic. Remember, many client IP addresses are randomly assigned.

• Don't shun IP addresses based on connectionless traffic, like UDP, ICMP or TCP traffic that is not part of an existing stream. You're asking for a denial of service.

• Be sure you understand what constitutes legitimate traffic, and don't accept a vendor's claim that a signature has a low false-positive rate. Your traffic is unique to your network; only you can assess what constitutes a false positive.

• Test signatures that are blocking candidates for false positives.• Tune your rule base to match the applications and services that will be protected by the NIP device. This will reduce the false-positive rate.

• If you can't afford any blocked connections to a service, such as SMTP, don't use NIP. Harden your server instead.