When It Comes To Anti-Spyware Tools, Accuracy Is Key

How good is your anti-spyware? Can yours “detect 40,000 parasite definitions”? Can yours search for 53,248 spyware components?” Only 22, 984? Wimp! But wait, how can the wimp be rated fifth best out of 20 in a comparative review? What are we counting here? Are we all using base 10 arithmetic?

If you’re confused by the disparities in claims of numbers of spyware detected, and nervous that the anti-spyware software you just purchased doesn’t measure up, join the club. Dozens of anti-spyware software companies are waging war on two fronts. To the east, software engineers and spyware hunters battle against spyware developers. To the west, marketing wonks wage a competition amongst themselves to catch your attention and ultimately, another sale. In our “super-size it” society, what better way than to pile up the statistics?

The Issues In Defining Spyware

In security technology circles, numbers are never more deceiving than when they are applied to intrusion and malicious code detection. In the case of spyware, the numbers are doubly deceptive.

As you may have already surmised, the first deception lies in what’s counted as spyware. I know of no standard definition of what constitutes “one spyware” (if you find one, send it to Congress). Is each ad cookie one spyware instance? Each OLE object? DLL? Executable programs? Can we count a program stored on disk and a process running in memory as two instances? Is each registry item added by a spyware installer package one spyware instance? What if the spyware changes a registry item: can we count that? What if two spyware use the same registry value or substitute their own DLL for a legitimate one? Can I count my competitors by labeling them scamware?Common sense tells me that this is all nonsense and borders on deceptive advertising. But how often does common sense prevail in a competitive market? When I asked several anti-spyware vendors how they counted, I discovered “what counts as spyware” is quite a hot button. So I decided I’d compare how anti-spyware vendors count spyware myself. I also decided to forego formal, methodical testing. Instead, I would “inspect my system for spyware” the way an average consumer might.

A Spyware Inspection Approach

The one test area where I did impose some rigor was the method of infection. I asked Aluria Software’s Research, Analysis, and Response Team to provide me with some spyware samples. I visited sites RAR suggested to further infest the test PC. [Disclosure: I have done consulting work for Aluria Software and earned their trust. Under normal circumstances, they do not distribute spyware samples.]

I began with a laptop running a clean install of Windows XP SP2 and downloaded 10 “free anti-spyware scanners” at random. I installed each scanner and disabled any active protection provided by the product. I ran a full system scan from each scanner to be certain they all detected no spyware. This in itself was an interesting exercise, as several products identified competing products as scamware; humorously, some products point accusing fingers at each other. Adjusting for this behavior proved non-trivial. I didn’t want to remove the scamware because I was fairly confident these products would help me prove my point. Moreover, I was only interested in obtaining coarse measures, so I simply added the counts of scamware detected to the total counts.

This was a very informal test so I do not intend to publish the product names nor the results. Suffice to say that the range in the numbers of spyware infections reported was between 14 and 187. By my count, the number should have been 19. At the high end, I suspected several false positives but it was evident from the way the scan results were presented that the objective were to deceive and persuade the consumer to purchase the product.Unofficial Testing Conclusions
Now that I had a basis for comparison, what conclusions could I draw? The first is that raw numbers of spyware detected are deceiving. Without standards for what constitutes one (1) spyware infection, it’s impossible to say whether one scanner is superior to another. Without certification to assure that products comply with such standards and hence compete on a level field, deception is too often rewarded: unsophisticated users can easily be misled or frightened into purchasing products that claim to detect the most spyware. Lastly, new spyware appears frequently, and existing spyware is morphed to evade detection even more frequently. Counting can actually conceal the fact that a product isn’t keeping pace with new threats.

Rating anti-spyware products based on claims of the number of spyware detected diverts attention from what I believe is the more important metric for scanning: accuracy. Scanning accuracy is extremely important, especially for large-scale deployments. Accuracy can be measured in terms of false positives and completeness. False positives – crying “Wolf!” – distract IT from productive tasks. If you’re about to deploy anti-spyware to hundreds of desktops, you don’t want to be barraged with false alarms.

Comprehensiveness of detection is even more important. Products that do not identify all the components of a spyware infection and cannot keep accurate track of components as spyware morphs can be dangerous. In large desktop deployments, you don’t want products that perform automated removal upon detection to do 'The Wrong Thing', remove a required dll, and cause hundreds of PCs to crash and burn.

Ultimately, you run the risk of getting exactly – or less – than you pay for when you rely on the performance of free scanners as the sole basis for purchasing anti-spyware software. You have no useful basis for comparison. You are also overlooking two equally important features of anti-spyware software: blocking and removal. Unfortunately, there are too few reliable comparative tests for anti-spyware, and too many web sites that post contrived and biased test results. If you’re searching for effective anti-spyware software, I recommend you search for product tests performed under editorial supervision by reputable trade publications.

A second, reliable source is to follow the leads of Internet Service Providers who offer anti-spyware software to customers: top-tier ISPs aren’t going to recommend or offer software that’s going to increase support calls.0