Network Computing is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Using Open Source Tools For Malware Detection

I started my technology career in the late '80s, working as a bench tech at a small computer repair shop. The first major malware infection I remember was a virus called Stoned, which spread when users shared floppy. It was hidden in the master boot record, but was not terribly difficult to find and repair. A computer infected by Stoned simply displayed the message, “Your computer is now stoned, Legalise (sic) Marijuana.”

Times have certainly changed. Malware has evolved into a multimillion-dollar criminal enterprise, sophisticated to the point that it's difficult to detect and even more difficult to remove. Today's malware is excellent at evading antivirus by using rootkits and obfuscation. It lurks in memory, often attached to a system process or device driver, and can siphon out personal and company secrets that are sold to the highest bidder.

How can a small business stand up to this kind of sophisticated malware? The good news is that there are great open source tools available that allow anyone to find, study and eliminate these threats. It just takes a little time and effort to turn some spare hardware into a commercial grade malware detection system.

To give you an idea, I'll describe how I used open source tools to hunt down a particularly malevolent strain of the Zeus Trojan, which primarily targets financial accounts. Zeus captures keystrokes on financial websites to steal credentials, which it forwards to criminals. The source code was leaked in 2011, and modern strains of Zeus rely on peer-peer communication; this makes finding the command and control servers for the Zeus botnet very difficult.

pfSense and Snort

Zeus entered the network in late June, either via a phishing email or a user visiting a compromised website. It used a zero-day flaw in the Java runtime, then bypassed an up-to-date antivirus system and hid itself via a rootkit in the system recycle bin. The nightly antivirus scans ran without a hitch, so there was no hint of the criminal activity that was taking place on the infected machine.

My company had just implemented a new firewall using the open source pfSense. This easy-to-use firewall based on FreeBSD is a great security solution for a limited budget. It will run on fairly low-end hardware and is very simple to install. Don't let the price fool you--it offers features that are competitive with almost any commercial offering.

[Security researchers were busy demonstrating new vulnerabilities and exploits at the recent Black Hat USA Conference. See what was on the firing line in "9 Technologies Security Researchers Will Break At Black Hat."]

One of the best features of pfSense is that it can run the open-source Snort intrusion detection system. Snort is one of the most useful tools available for discovering malware running on the network. Here's why: Malware can easily bypass antivirus programs unless a specific signature exists to detect the threat; after bypassing antivirus protection, the malware is almost impossible to find once it has used a rootkit to avoid future detection. However, all malware needs to communicate out across the Internet to download instructions and upload payloads of ill-gotten information; this is where Snort catches it.

I configured pfSense with the open source Snort feed, which is 30 days behind the registered rules but still works well enough for SMB networks. I also added the emerging threats rules subscription because it is full of malware signatures that are contributed by the community almost in real time. Instead of configuring Snort to listen on the external network interface, I configured it to listen only on the LAN network interface to intercept traffic destined for the Internet. This allowed it to see the source addresses of infected machines running on the LAN.

Global Config (redacted)
(click image for larger view)

pfSense and Snort are so well-integrated that the system can block traffic based on the source or destination IP address the triggered the alert. This worked well when dealing with Zeus because the infected PC could not communicate to the rest of the Zeus botnet, blocking any possible information leak until the PC could be pulled from service.

Blocked (redacted)
(click image for larger view)

The infected PC was identified through the Snort alerts, but there was an interesting surprise: It had another virus--ZeroAccess, which is used as a “click-jacker.” A click-jacker is used to generate fraudulent clicks on ads that are per-click based. Although not as dangerous as Zeus, it is still a type of criminal activity that uses company assets. pfSense contains a diagnostic packet capture that is adept at capturing malware that uses recurring traffic patterns. For example, ZeroAccess communicates to the rest of the botnet using UDP ports 16464 and 49154. pfSense was able to capture this traffic by defining filters on these specific ports.

Next Page: Open Source Forensics Tools

  • 1