Filters Take a Bite out of SPAM

Posted by Ron Anderson on May 7, 2004

Tags: antispam,  government regulations,  legal,  mail,  spam laws,  technological,  Anti-spam,  Client/Server Protection,  Data Protection,  Other,  SPAM,  Security and Privacy,  Software,  Software and Web Development,  Spam

Channel: Data Protection, Other

Weighty Matters

Because our weighted accuracy rating determined the products that made it into this review, it's important for you to understand our definition of accuracy. We used both false positives and false negatives to determine an accuracy score for each spam filter because both measurements represent classification mistakes. But because false positives are more costly to your organization than false negatives, we took our accuracy ratings a step further by weighting each false positive by a factor of five (for our definitions of false positives and negatives and other spam-related terms, see Glossary). We include the nonweighted accuracy in our table (page 62) for comparison but used the weighted ratings to determine which vendor would make the final cut.

The Long List Click to Enlarge

Note that our weighted accuracy scores are lower than the accuracy ratings published by antispam vendors. This is due, in part, to our giving more weight to false positives. In addition, procedural issues had a larger effect on some products than others. For example, Postini complained (after the fact) that our test methodology caused it an unduly large number of false negatives because its transport heuristics were rendered useless. Postini uses transport heuristics to examine the content of the SMTP conversation prior to the data command in the SMTP protocol and drops up to 30 percent of inbound SMTP connections as spam before any message content is received. Because our messages were mirrored from our production e-mail server, Postini's transport heuristics didn't come into play, forcing its content filters to do 100 percent of the spam detection. Likewise, vendors that rely on customer training for their Bayesian engines fared worse than vendors with Bayesian engines that ship with an extensive pretrained database.

Let's Talk SPAM Join us Tuesday and Thursday (May 18th and 20th) at 12:30pm eastern to talk live with Ron Anderson about his recent review of 35 Anti-Spam hardware and software solutions.

Another reason our accuracy numbers are lower than the vendors' is because their stats look at only part of the picture and are based on best-case scenarios. Vendors usually report their tuned catch rate, which counts only true positives and reflects customer-specific tuning to help increase accuracy, or their false-positive rate. For example, Brightmail reports its product to be 99.9999 percent accurate based on its claim of 1 in 1,000,000 false positives, with no reference to false negatives.

Accuracy Test Results Click to Enlarge

Finally, our test bed used real e-mail directed to NETWORK COMPUTING editors, including scads of press releases, HTML-formatted industry newsletters and other spammy-looking legitimate missives that are tough to analyze correctly. Remember that this is a point-in-time test that emphasized out-of-the-box performance and defined accuracy in a certain way--your mileage may vary.