Three-Way No Calling
HIP vendors have implemented three basic models: behavior-based policies, signature-based policies and user-based policies. While you can achieve similar functionality with any model, the general concepts are vastly different. In all three cases, it's important to understand that these products protect the OS from attacks that exploit applications. They won't stop attacks that operate by interacting with an application, such as someone trying to manipulate your database through SQL injection by fiddling with your Web application, nor will they stop servers from accessing files to which they should have access.
Behavior-based models require you to define the normal behavior during an application's run time. For example, Apache on Linux needs to read and write files in its directory root, read its configuration file, read and execute its own executable, read shared libraries, read files from webroot, execute cgi-bin files and bind to Ports 80 and 443. A properly behaving server doesn't require any access beyond that, so a policy that limits its access blocks all abnormal behavior. Of course, what is normal for one server may not be normal for another, so each server must be profiled and its normal behavior thoroughly understood before you can define an enforceable policy.
Sign Here, Please
Signature-based HIP products use signatures to detect and block malicious activity. Of course, signature products suffer the same problems as IDSs and virus scanners--they are only as current as their latest updates. But used in conjunction with behavior-based HIP, signatures provide another layer of security by being able to selectively block attacks at the application layer, and they also aid in attack identification because an attack alert message can provide a name, rather than alerting on a generic "buffer overflow."
Finally, user-based models are similar to user-based access control, where user name or group membership is used in conjunction with ACLs on system objects to determine if access is allowed or denied. This model works well in the Unix world, where services can be run under unique user IDs. However, in Windows NT, where critical services, such as inetinfo.exe, can run as System\Local, it's a different story; you can't effectively limit an activity that is running as System\ Local because other services running as that account require so much access. The lack of individual user IDs is also problematic.
As for policies, HIP products fall into two categories: those limited to predefined polices and those that allow customization. In general, HIP products with predefined policies are easier to install and manage, provided your applications are supported. These packages use discovery or rely on an administrator to customize the configuration for each installation. For example, if you put your webroot in a nonstandard location, the HIP application will need to know about it. The obvious limitation is that you can't protect anything beyond your supported applications.
The second category of products offers methods for securing any application, provided you can define the necessary resources and the types of access for each resource. Customization offers the flexibility to support any service your business needs, and you're not tied to a vendor's limited set of supported applications. The downside is that application profiling and policy development are complex and time-consuming processes because you have to cycle through resource and access discovery, policy development and deployment, and testing, and repeat until the policy is defined to allow only the necessary access and the application runs without problems.
Clearly, there is a trade-off between ease of management and flexibility. If you go with a general-purpose HIP, budget to send your administrator to training, or at least give him or her time to learn the systems. We spent between 25 and 40 hours per product just learning the ins and outs, and we consider ourselves moderately skilled in many host OSs. We also used some handy references, which you can find in our "Recommended Reading" list, below. While programming skills are not required to effectively profile applications and develop policies, the more you know about how the host OS and applications work, the better off you'll be.