Using Open Source Tools For Malware Detection

I started my technology career in the late '80s, working as a bench tech at a small computer repair shop. The first major malware infection I remember was a virus called Stoned, which spread when users shared floppy. It was hidden in the master boot record, but was not terribly difficult to find and repair. A computer infected by Stoned simply displayed the message, “Your computer is now stoned, Legalise (sic) Marijuana.”

Times have certainly changed. Malware has evolved into a multimillion-dollar criminal enterprise, sophisticated to the point that it's difficult to detect and even more difficult to remove. Today's malware is excellent at evading antivirus by using rootkits and obfuscation. It lurks in memory, often attached to a system process or device driver, and can siphon out personal and company secrets that are sold to the highest bidder.

How can a small business stand up to this kind of sophisticated malware? The good news is that there are great open source tools available that allow anyone to find, study and eliminate these threats. It just takes a little time and effort to turn some spare hardware into a commercial grade malware detection system.

To give you an idea, I'll describe how I used open source tools to hunt down a particularly malevolent strain of the Zeus Trojan, which primarily targets financial accounts. Zeus captures keystrokes on financial websites to steal credentials, which it forwards to criminals. The source code was leaked in 2011, and modern strains of Zeus rely on peer-peer communication; this makes finding the command and control servers for the Zeus botnet very difficult.

pfSense and Snort

Zeus entered the network in late June, either via a phishing email or a user visiting a compromised website. It used a zero-day flaw in the Java runtime, then bypassed an up-to-date antivirus system and hid itself via a rootkit in the system recycle bin. The nightly antivirus scans ran without a hitch, so there was no hint of the criminal activity that was taking place on the infected machine.

My company had just implemented a new firewall using the open source pfSense. This easy-to-use firewall based on FreeBSD is a great security solution for a limited budget. It will run on fairly low-end hardware and is very simple to install. Don't let the price fool you--it offers features that are competitive with almost any commercial offering.

[Security researchers were busy demonstrating new vulnerabilities and exploits at the recent Black Hat USA Conference. See what was on the firing line in "9 Technologies Security Researchers Will Break At Black Hat."]

One of the best features of pfSense is that it can run the open-source Snort intrusion detection system. Snort is one of the most useful tools available for discovering malware running on the network. Here's why: Malware can easily bypass antivirus programs unless a specific signature exists to detect the threat; after bypassing antivirus protection, the malware is almost impossible to find once it has used a rootkit to avoid future detection. However, all malware needs to communicate out across the Internet to download instructions and upload payloads of ill-gotten information; this is where Snort catches it.

I configured pfSense with the open source Snort feed, which is 30 days behind the registered rules but still works well enough for SMB networks. I also added the emerging threats rules subscription because it is full of malware signatures that are contributed by the community almost in real time. Instead of configuring Snort to listen on the external network interface, I configured it to listen only on the LAN network interface to intercept traffic destined for the Internet. This allowed it to see the source addresses of infected machines running on the LAN.

(click image for larger view)

pfSense and Snort are so well-integrated that the system can block traffic based on the source or destination IP address the triggered the alert. This worked well when dealing with Zeus because the infected PC could not communicate to the rest of the Zeus botnet, blocking any possible information leak until the PC could be pulled from service.

(click image for larger view)

The infected PC was identified through the Snort alerts, but there was an interesting surprise: It had another virus--ZeroAccess, which is used as a “click-jacker.” A click-jacker is used to generate fraudulent clicks on ads that are per-click based. Although not as dangerous as Zeus, it is still a type of criminal activity that uses company assets. pfSense contains a diagnostic packet capture that is adept at capturing malware that uses recurring traffic patterns. For example, ZeroAccess communicates to the rest of the botnet using UDP ports 16464 and 49154. pfSense was able to capture this traffic by defining filters on these specific ports.

Next Page: Open Source Forensics ToolsOnce captured, you can export the traffic as a standard pcap file to one of the best open source network tools available: Wireshark. It was obvious from the packet capture that ZeroAccess was extremely active and had contacted 50 different hosts in about a minute. The traffic looked to be some type of a keepalive method of distributing information to the PCs that were part of the botnet. The infected PC would send out a 60-byte UDP packet (probably a type of keepalive) from a source port of 49154 to a destination port of 16464 on the target PC. The target PC then would respond with a 610-byte UDP packet from port 49154 (probably instructions from the botnet) back to port 16464 on the infected PC.

(click image for larger view)

The next step in this forensic analysis was to find the actual malware executable. The best way to find elusive program files is by using a forensics-focused Linux distribution. There are several great Linux distributions that allow for deeper analysis and malware analysis. The ever-popular Kali Linux is typically used for penetration testing but contains forensics tools, as well. The SANS Institute provides the SANS Investigative Forensic Toolkit, or SIFT, which hasn't been updated in a while but still provides excellent documentation on how to conduct a forensic analysis. A lesser-known but powerful Linux distribution focused on reverse-engineering malware is REMnux.

The best practice is to remove the hard drive from the PC and attach it to another workstation with a USB-based write blocker. There has been debate in the forensics community over whether live CDs modify the drive contents, even when mounting read-only. A USB-based write blocker is the sure way to prevent any modifications to the target drive.

In this case, I mounted the infected hard drive to a USB-based write blocker and attached it through a virtual USB connection directly to REMnux running inside of the VirtualBox open source virtualization tool. I quickly identified the Zeus Trojan inside of the “$Recycle Bin” folder on the infected drive using tools available in REMnux Linux. I then used VirtualBox again to infect a virtual Windows XP image to study the outbound network connections more thoroughly and to document the infection process.

Malware will continue to become more sophisticated and stealthy as the profits it generates increase. Small businesses could become easy prey for criminals wielding this malware because they cannot always afford proprietary security tools. pfSense, Snort, Wireshark, Kali Linux, SIFT, REMnux and Virtualbox provide capabilities that rival their commercial counterparts. These tools will continue to evolve as more people use and improve them. Open source is all about community, and community may the best approach to solving the malware problem.