Why You Need A New-Generation Intrustion Protection System

Signature-based Intrusion Prevention Systems (IPSs) are being out-evolved. New vulnerabilities and new exploits provide attackers with a window to launch fast-spreading worms and malware that blow by signature-based detection systems. Enterprise networks are vulnerable until their IPS vendors capture, analyze, and identify the new attack. And while new signatures can be deployed within hours, that's still too slow in the age of SQL Slammer, a Windows worm that doubled the number of compromised hosts every 8.5 seconds and infected approximately 75,000 computers in 30 minutes.

The fact is, today's threats have advanced beyond the capabilities of signatures to address them alone. IPS vendors, especially those in the Host-based IPS (HIPS) field, are stepping up to the challenge with a range of technological approaches that close this window of vulnerability.

Venerable server and desktop HIPS technology from Cisco Systems, McAfee, and Sana Security can monitor host behavior at the kernel level and shut down processes that tip toward malicious or unwanted actions. These products, with their ability to stop both known and unknown attacks without the use of signatures, represent the incumbent HIPS technology today.

However, a new generation of products is emerging to challenge the notion that any piece of software can effectively discern bad behavior from good. These upstart HIPS products, also aimed at servers and desktops, take a variety of approaches, including enforcing essential programming conventions to prevent buffer overflows, creating a protected subsystem to run untrusted programs without affecting a host's critical files or registries, and implementing strict enforcement agents that only allow approved executables to run. These products also avoid some of the problems of behavioral analysis, such as the need to train and retrain agents and the risk of false positives.

Both the incumbent and start-up technologies aren't attempting to get rid of signature detection, but they do provide what signatures can't--namely, advanced protection against unknown exploits, extra layers of defense for critical servers and desktops, and protection for assets during the lag time between newly announced vulnerabilities, patch releases, and patch deployment. That said, these products all have drawbacks that architects must consider when evaluating a HIPS solution to bolster existing defenses.YOUR BEST BEHAVIOR

Host-based behavioral monitoring came to market several years ago, driven primarily by start-ups Entercept and Okena. These two companies have since been acquired by McAfee and Cisco, respectively. Sana, a third company, also plays in this space. All three vendors' products can run on servers and desktops under a variety of OSs (see "Protection Beyond Signatures" table for a complete breakdown).

As the figure to the left shows, behavioral monitors are software agents that get inserted between an OS's user space and kernel space.They monitor system calls made by applications and correlate those calls to rules, policies, or lists of known behaviors. The agents can block any system calls that they deem suspicious or that violate rules or policies. The result is that zero-day or unknown attacks can be thwarted without having to wait for attack signatures to be developed.

IT architects are often leery of deploying host-based protection that might degrade application performance. This is certainly possible for HIPS software that must examine large volumes of system calls to detect potentially unwanted behavior. However, vendors say their agents generally exact only a 3 to 5 percent overhead on server and desktop performance.

"In general, we look at the large majority of the system calls, but depending on the system call, the calling process, and the kind of protection policy applied, we only do the necessary processing," says Vlad Gorelik, CTO at Sana. "This allows us to minimize CPU overhead." For performance-sensitive applications, however, predeployment testing is a wise precaution.

The McAfee, Sana, and Cisco agents also claim to stop a variety of malicious spyware, such as keystroke loggers and dialers. For instance, Sana's Primary Response agent uses Active Malware Defense Technology to observe suspicious behaviors for various processes. For instance, does a process start automatically at boot? Does it have a way to communicate to an outside location? Does it perform unwanted behaviors such as keystroke logging? Does it try to keep itself out of Windows Task Manager? Such behaviors are strong indicators of malicious programs and can be shut down automatically.Note that McAfee's Entercept doesn't do away with attack signatures completely, but rather uses a limited set to bolster its behavior-based detection. A software module monitors traffic coming in from the network card and looks for pattern matches to known attacks. Entercept also adds signature protection for Apache, IIS, and SQL in the form of HTTP and SQL injection and directory traversal attack prevention. Cisco's Cisco Security Agent (CSA) and Sana's Primary Response don't use signatures.

IT architects use these products as part of a strategy for defense in depth, especially for essential business systems. James Cupps, until recently the CISO for multibillion-dollar paper manufacturer Sappi Fine Paper, used two HIPS products on critical servers in addition to a network IPS. He says the HIPS products catch maybe three or four attacks per week. "If you look at the signature-based attacks of the network-facing systems, that's tens of thousands of attacks daily, but the majority are filtered out before they get to the server," says Cupps. "That's why you do a layered approach."

HIPS software also provides coverage while companies test and roll out critical patches. "If Sana is on a server, we don't have to patch right away--we can go through our change control process," says Eben Berry, manager of information systems operation for Network Health, a Medicaid health plan that provides health insurance for more than 60,000 Massachusetts residents. Berry has standardized his Windows server configurations to include Sana agents to protect high-risk servers in the DMZ.

The catch is that these products must understand good behavior before they can identify unwanted actions or they'll cause false positives by blocking harmless transactions. Entercept, CSA, and Primary Response all come with default policy templates to provide protection against basic attacks such as buffer overflows, but these products are most effective when they can be trained and refined.

Cisco requires administrators to create rules for its agent. Administrators can run CSA in test mode with default policies in place. In test mode, the agent monitors behavior and then reports back to a central management console. The report shows what processes the agent would have allowed and what it would have blocked. For instance, the report might show that a particular process tried to open or read a file and that the behavior would have been blocked in protect mode. If this is a known good behavior, the administrator can use the report to override this rule. Rules can be assigned to various servers or desktops according to user group or computer group (for instance, all IIS servers).Entercept operates on a similar principle, offering a warning mode that lists events that would have been blocked. Administrators can create exceptions and then set the agent at one of three protection levels. Sana's Primary Response is also first deployed in learning mode. Administrators run an application's functions so that the agent understands the behaviors that are allowed.

The problem with behavioral monitoring is that applications aren't static. New versions are rolled out, and patches and software fixes are continuously deployed. That means these behavioral monitors must be retrained for application changes or new application deployments, a process that may represent significant effort.

"In order for the Sana agent to learn, it had to see certain functionality in regard to user and application behavior," says Berry. "The internal Web testing team had to exercise all the functions of the application to know what was good and what was bad."

Berry also says that after deploying patches on servers, he switches the agents from protect mode to learning mode as part of a rigid change control process. "It's a good idea to make sure there are no false positives," he notes.

THWARTING BUFFER OVERFLOWSBuffer overflows are a common class of software vulnerability often exploited by malware. Attackers often use buffer overflows as the first step to executing their own code on a target computer.

Two technologies address this critical attack vector. The behavioral analysis agents from Cisco, McAfee, and Sana count buffer overflow protection as one of their core offerings and have been the standard protection mechanism for several years. Once the malicious code included with a buffer overflow begins to execute, these agents will detect unusual system calls and prevent the code from running, thus preventing an attacker from gaining control of the target.

However, these products don't actually prevent the buffer from being overflowed. They must wait until the malicious code begins executing, at which time they detect unusual system calls and stop the code in execution. What's more, if the buffer overflow itself causes the application or computer to crash, the behavioral engine won't be able to stop the crash.

A new technology addresses this problem by dealing with the buffer overflow at a more targeted level inside the OS. The Memory Firewall from start-up Determina grew out of research conducted at MIT. The research demonstrated that buffer overflows violate basic programming conventions. These conventions, known as the Application Binary Interface, are the same among all compiled applications. Thus, rather than analyze system calls for potential buffer overflow activity, the Memory Firewall simply enforces these basic conventions.

In a traditional buffer overflow, the attacker aims to point the computer to injected code by modifying the return address of a function call. The Memory Firewall will detect that change in the return address and stop the transfer of control to injected code.More specifically, when a program executes, the Memory Firewall loads its instructions into memory, including all possible instructions for that program. At this point, the product creates a small virtual machine environment to control the program's execution down to individual instructions. Because all the instructions of the program have been loaded into the virtual environment, the Memory Firewall can verify which instructions should come next and that the instructions are part of the original program loaded at start-up.

"The issue with system call monitors is that they aren't watching every single instruction in the user space," says Charles Renert, head of security research and development at Determina. He emphasizes that attacks aren't blocked until the malicious code executes and then issues the first system call. By contrast, the Memory Firewall doesn't allow a single instruction of malicious code to run. In addition, while the Memory Firewall doesn't prevent the buffer from overflowing, in most cases it'll keep the application or computer from crashing.

Once a block of code is inspected, it's moved to a secure cache so that it doesn't have to be reinspected. Renert says the software will add 5 to 15 percent latency to an application in the first few seconds of operation. Out of the box, the Memory Firewall protects all Windows services and server applications, including ISS, Exchange, and SQL Server. Administrators can also add custom applications.

Cupps used both the Memory Firewall and Entercept on the same IIS Web servers and also extensively tested CSA. He used the Memory Firewall for buffer overflow protection and Entercept for protection against SQL injection and directory traversal attacks. The buffer overflow feature was turned off in Entercept.

One reason he did this was that he preferred Determina's approach to buffer overflow protection. But just as critical was the lack of configuration it required."We get a high level of protection [from the Memory Firewall] with almost no oversight and a reasonable cost," says Cupps. "There's no specific configuration necessary. With CSA, you have to choose what you want to allow to happen. With Entercept, you run it in learning mode, choose things not to alarm for, and establish a baseline. And whenever you change the baseline, you have to touch Entercept again. With Determina, you don't worry about that."

However, the Memory Firewall only protects against memory-based attacks such as buffer overflows. It doesn't provide the complete range of protection against malware and other attacks offered by Entercept, Primary Response, and CSA.

ATTACK DETECTION: A FOOL'S GAME?

Two vendors have looked at the efforts of behavioral blocking companies and concluded that trying to get a software agent to discern between good and bad behaviors is a fool's game. These vendors, GreenBorder and SecureWave, believe that regardless of the detection mechanisms employed, either some malware will get through or there'll be false positives.

"We are getting to the point where it won't be possible to identify malicious threats," says Jim Fulton, vice president of marketing at GreenBorder. The growing number of zero-day exploits and polymorphic malware, combined with numerous infection vectors (including e-mail, Internet, IM, portable storage devices, and mobile laptops), can thwart or bypass detection systems. "Bad stuff will get through any detection-based scheme."However, while GreenBorder and SecureWave agree on the premise, they diverge widely when it comes to implementing their own solutions. The GreenBorder software creates a protected subsystem on individual desktops and laptops to run untrusted applications and executables away from the host's files and registry. By contrast, SecureWave's Sanctuary creates a white list of allowed applications and refuses to let any other executable run, locking down the computer against unknown or untrusted software.

Although it relies on a protected subsystem that acts as a virtual environment for the OS and user files, GreenBorder isn't traditional virtual machine technology because it doesn't create distinct systems with their own virtualized hardware, memory, and so on within a single machine. Instead, GreenBorder creates a shadow environment in which to run untrusted content that comes from Internet Explorer or Outlook. This content will run normally, but have restricted access to the OS. GreenBorder defines untrusted Web content as any Web traffic that comes from servers outside a specified range of IP addresses. E-mail trust can be configured based on source domain or specific mail servers.

Web sites and e-mail messages that run in the isolated environment appear with a green border around them (hence the name) to alert the user that the content is untrusted. Any malware contained in this untrusted content is permitted to run. However, API and system calls from programs inside the protected subsystem are intercepted and prevented from reaching the actual kernel, and any changes to files or registries are made on shadow copies of those entities--the real files and registries are left unchanged. The software can also be configured to block access to the network or Internet from the protected subsystem.

In addition, if content inside the protected subsystem launches another application, that application is likewise run in isolation. For example, if a user copies information from an untrusted Web page into a Word document, that Word document will run in the protected environment as well. If the user saves that document, GreenBorder will tag the file so that the next time the user opens the file its application will also run in the protected subsystem. If the user e-mails the document, the tag will accompany it so that if the recipient uses GreenBorder the document will run in the protected environment, too. If the recipient doesn't have GreenBorder, the file will run normally (including any viruses that may have been embedded in it).

When a user logs out at the end of the day, the protected subsystem is flushed of whatever content was shunted there, including cookies, any downloaded applications, and executables. All processes are terminated, and any changes made to shadow files or registries are removed. Also, if users notice any performance degradation as a result of malware loaded, they can reset the protected subsystem on demand to remove any processes that are hogging system resources.While the product is configured out of the box for Internet Explorer and Outlook, other programs can also be tagged to run in GreenBorder. For instance, an administrator could tag the Firefox browser to run in the protected subsystem.

Anthony Shields, systems administrator at The Epstein School in Atlanta, GA, has been running GreenBorder on 250 Windows XP computers since January 2005. Approximately 50 are for dedicated staff, while the remainder are available to students and staff in computer labs and classrooms.

"It works great, and it's transparent to the user," says Shields. He especially likes GreenBorder because it doesn't query the user about whether to allow an action or not. "A third-grader isn't going to know what's safe to run, nevermind an adult," says Shields. "GreenBorder just lets it happen, and when a user exits Internet Explorer it's done."

That said, Shields is still using anti-virus software on the desktops to protect against infections from trusted areas. "GreenBorder won't protect you if someone plugs in a floppy or thumb drive that has a virus on it because it's coming from the trusted network," he says.

GreenBorder may also not protect hosts from network-based worms that attack a computer through a network-available service other than Internet Explorer or Outlook. Administrators should run GreenBorder in tandem with a software firewall to limit network availability, as well as anti-virus software to ensure that malware isn't loaded onto the machine through trusted sources.MAKING A LIST, CHECKING IT TWICE

At the other end of the spectrum is SecureWave's Sanctuary product. Rather than monitor behavior or create an isolated environment in which to run programs, Sanctuary creates a white list of allowed executables; any other program or application that tries to run gets blocked.

"It's far easier for an administrator to identify what's good than keep up with what's bad," says Dennis Szerszen, vice president of corporate development and marketing at SecureWave. "That includes not just malware but things like Kazaa, which has potential bandwidth and performance issues."

The Sanctuary product, which can be deployed on servers and desktops, is an agent written as a kernel-mode driver that intercepts requests for executables to run. The agent takes a snapshot of the executable, including RPCs and DLLs, identifies the entity, and creates a hash of it. The agent then compares the hash to a master list that comes from a central server. If the current hash signature isn't on the white list of approved applications or executables, access is denied and the end user gets a message saying that the application isn't authorized. The result is that any unauthorized program, including malware that tries to load itself onto the computer, is blocked.

Agents are controlled from a central management console where administrators can create master lists. The agents will also log every executable that tries to run, giving administrators a global view of network activity. The product can also track the usage patterns associated with approved applications, enabling administrators to save money by removing extraneous licenses for software packages that aren't being used.There are two difficulties with the Sanctuary product, however. One is updating and deploying valid applications, and the other is creating and enforcing enterprise-wide application policies.

Administrators must be sure to update the master white list with new applications or application upgrades before rolling out software to protected clients. Otherwise, the Sanctuary agent will block any attempts to deploy new software or patch existing programs.

Perhaps even more problematic is that the majority of organizations don't have a uniform set of applications. Sanctuary integrates with Active Directory so that policies can be built around existing user groups, but even within those groups it's likely that a large number of exceptions will have to be created and managed. In addition, administrators may find it difficult to wrestle control back from end users who have become accustomed to loading whatever applications they want onto their machines. "The hard part is building allow lists the first time around," says Szerszen. "The effort is up front."

Szerszen says the first step is to establish a written executable control policy that states what users can and can't run and inform users of the policy. The next step is to deploy agents and simply have them monitor what's being run for a week or two. Administrators can then build application groups with permission lists and notify users of a specific date when unauthorized applications won't work anymore.

The Sanctuary agent can also be configured to allow local delegation, which lets the end user run whatever programs or applications he or she wants. However, local delegation essentially removes all the protection offered by the agent in the first place. Sanctuary is most powerful when applications are firmly controlled by a central authority.Andrew Conry-Murray, technology editor, can be reached at [email protected].

Risk Assessment: HIPS Technology without signatures

System call-based behavioral monitoring is the most mature type of HIPS software that doesn't use signatures. The market was essentially created in late 1999 and early 2000 by start-ups Entercept and Okena, companies that have since been acquired by McAfee and Cisco, respectively. Newer HIPS technologies eschew behavioral analysis in favor of various proactive prevention strategies, but they don't yet have the track record or customer penetration of their elders.

HIPS software isn't nearly as widely deployed as anti-virus software. This is in part because of a fear of false positives, but also because traditional HIPS software may require a significant amount of training or rule creation up front. These concerns are amplified because HIPS software is usually deployed on business-critical systems where self-inflicted outages won't be tolerated. Newer generations of HIPS technology may alleviate some of the difficulties associated with policy creation.HIPS software can have an immediate impact in two key areas. First, it provides an extra layer of defense on critical servers, especially against new attacks that may slip by signature-centric security gateways. Second, HIPS software acts as a backstop against potential new exploits during the lag between patch releases and patch deployment. It provides coverage so that organizations can follow proper change management procedures.

When it comes to prevention products, security and business professionals must balance the risk of attacks with the risk of self-inflicted transaction losses. At present, fast-moving malware outbreaks have tipped the risk equation in favor of prevention. However, security professionals must use care when designing rules and policies to ensure low false positives.

The Busted Buffer

Buffer overflows are a common weapon in the attacker's arsenal. In a basic buffer overflow, the attacker sends a specially crafted attack to a computer running software that's known to be vulnerable to buffer overflows. This specially crafted attack has more data than can be contained in a section of memory known as the buffer. The excess data flows out of the buffer and into another area of memory and changes the normal process by which the computer operates. The computer will then execute the attacker's code as if it were part of the regular application or program.

If the attacker has written the attack code correctly, the computer will follow whatever instructions are in the code, such as enabling remote access, executing a program, or getting the attacker closer to complete control over the target. Even if the code is flawed, the application--and possibly the computer--will crash. Thus, even an unsuccessful buffer overflow attack can still disrupt a service or otherwise harm the target.The best defense against buffer overflows is to write code properly to prevent overflows in the first place. Unfortunately, a great many software applications are still created with overflow vulnerabilities, which means other defenses must be employed. Many HIPS products include buffer overflow protection as part of a larger defense against malware.

The classic paper describing buffer overflows is entitled Smashing the Stack for Fun and Profit by Aleph One. You can find it online at www.phrack.org by searching for issue 49-14. Also check out the books Security Warrior (O'Reilly, 2004) by Peikari and Chuvakin, and Building Secure Software (Addison-Wesley, 2001) by Viega and McGraw.