Rootkit Detection

New products are emerging to make it easier for security professionals to unearth rootkits on compromised machines, but identifying those machines and removing the malignant software remains frustratingly difficult. Attackers still have the upper hand if a machine gets compromised. Malicious software incorporates full rootkits or rootkit-like capabilities to entrench itself on compromised PCs and evade detection. The use of stealth techniques by malware has increased 600 percent since 2004, according to McAfee, and the use of custom rootkits, which are difficult if not impossible to detect with signatures, is also on the rise.

The security community has responded to these developments with standalone rootkit-detection tools that attempt to find rootkits by examining low-level data, such as the raw file system. Some vendors also are adding enhanced rootkit-detection capabilities to their security software suites. Anti-rootkit tools generally do one of two things: detect and block rootkits before they compromise a PC, or attempt to find and remove them after they've burrowed into the OS.

Toward the goal of prevention, security vendors recommend a cocktail of techniques that includes signatures, heuristics, behavioral analysis and generic exploit blocking. If a machine has been compromised, the most common approach is to use a standalone rootkit-detection tool to probe the infected host.

At the same time, rootkits continue to evolve. For instance, rootkit authors are designing their programs so that they don't modify system information, thereby thwarting some rootkit-detection tools. Rootkit authors also are exploring new stealth techniques, such as hiding files using ADS (Alternate Data Streams), an NTFS capability designed to facilitate compatibility.

Tracing The Roots

Rootkits go back to the Unix OS, in which the "root" account provides administrator-level access to all functions and facilities. The goal of a rootkit is to hide the presence of an attacker and malicious tools. Rootkits exist for Unix and its variants, but most rootkits--and anti-rootkit software--focus on the Windows OS because of its ubiquity.

Windows rootkits can be divided into user-mode and kernel-mode. User-mode rootkits run as an individual application or may modify an existing program. Kernel-mode rootkits run in the kernel of the OS, and are often loaded as a device driver. Both types hide by intercepting and changing system-status information requested by an application.The first Windows rootkit, NTRootkit, emerged in 2001 as a proof-of-concept by security researcher Greg Hoglund. Since then more potent versions that target the Windows OS have emerged, including publicly available rootkits, such as FU and HackerDefender. And a growing number of criminals are buying prepackaged exploit tools.

Researchers at the University of Michigan and Microsoft recently described a proof-of-concept virtual machine rootkit called SubVirt. SubVirt installs a virtual machine monitor under the Windows OS of a compromised machine, letting it boot the OS into a virtual environment and operate undetected by security software running inside the virtual environment.

The Searchers

A variety of security products can prevent a rootkit from gaining a foothold on a computer, including antivirus, anti-spyware and HIPS (host intrusion prevention system) products. Standard signature detection from antivirus and anti-spyware software still plays a key role in prevention. The great majority of malware uses binaries or code snippets of known rootkits, which means signatures and heuristics can spot variants of known rootkits before they hit the hard disk.

HIPS software also can provide a measure of detection. Rootkits are often bundled into the payload of an exploit, but if the HIPS stops the execution, the rootkit won't be installed. For more on HIPS, see "Probing Questions" at nwc.com/channels/security/showArticle.jhtml? articleID=193005679.McAfee VirusScan 8.5, which is due to ship this month, will include a kernel-based scanner that can scan kernel- or user-mode memory for known rootkits.

Microsoft also has included a security feature called Kernel Patch Protection, or Patch Guard, in the 64-bit versions of its Windows OS. Patch Guard monitors the kernel and detects attempts by other code to intercept and modify kernel code. Microsoft says this feature is designed to help protect the OS from malware and from legitimate software that may destabilize the OS. At press time, Microsoft was meeting with third-party security software vendors about APIs to allow security software to work around Patch Guard.

Uprooting The Problem

Security vendors are developing methods to uncover rootkits on compromised machines. Many standalone tools use a technique called cross-view differential detection. This technique relies on the fact that a rootkit manipulates registries, APIs and system calls.

Cross-view detection mechanisms scan system components, including files, registry keys and processes, using the APIs on machines suspected of being rooted. This produces a "tainted view" of the system. It then runs a second scan of the computer--the trusted view--without exercising the APIs by examining lower-level data structures, such as the raw contents of a file system or the registry hive that aren't manipulated by the rootkit. It then compares the two scans to identify instances where system information may have been manipulated.

Standalone cross-view tools include F-Secure's Blacklight and SysInternals' Rootkit Revealer. F-Secure includes Blacklight in its consumer security suite, and plans to incorporate it into the next version of its enterprise security suite, F-Secure Anti-Virus Client Security. Symantec also has created a new tool, VxMS, that uses a method similar to the cross-view differential technique. The VxMS technology will be included in forthcoming enterprise editions of Symantec Client Security and Symantec AntiVirus.

Note that these tools look for generic rootkit activity, not rootkit signatures. That means an experienced IT administrator must examine the results to determine if the files represent a threat.

Warning Signs

Rootkit detection also is complicated by the number of desktops under IT administration. It's simply not conceivable to run a standalone tool on every PC--you'd have to touch each PC individually. As security vendors integrate cross-view differential detection tools into suites with a management console, this will be less of a problem.Meantime, there are signs that indicate a compromise and, if encountered, standalone tools are warranted. First, if you find a machine that has been infected with spyware or adware, you should also run a rootkit scan. Second, some rootkits can cause PCs to freeze up. If you've got machines on your hands falling prey to Blue Screens of Death for no apparent reason, a rootkit scan should be included in your diagnostic analysis. Other indications include the typical behavior of a machine infected with malware, such as high volumes of e-mail or Web traffic and back-channel communications using unusual ports or protocols.

If you can, also track how the machine got infected. Too often it can be tied back to employee behavior. That must be corrected to prevent future infections.

Technology Editor Andrew Conry-Murray can be reached at [email protected].