RSS
Email
Print
ShareSpam Filters
Posted by Christopher T. Beers on November 18, 2005
Of the 16 vendors we invited, Cloudmark, Commtouch Software, MailFrontier, Red Earth Software, Sunbelt Software and Symantec delivered products that met our testing criteria. Unfortunately, our tests of Sunbelt's iHateSpam yielded unusable results; our Sunbelt cache file became corrupt during the test, which left its spam engines without the data needed to determine if a message was spam. Sunbelt support worked with us on the problem, something the company said it had not seen in the past, but we ran out of time and had to regretfully exclude iHateSpam from this review. Clearswift, GFI Software, Hexamail, Lyris Technologies, McAfee, NetIQ, SurfControl, Sybari Software, TrendMicro and TumbleWeed Communications declined our invitation.
Accuracy Remains King
We designed our accuracy test to assess antispam technologies out of the box, without reliance on customer tuning (see "How We Tested Spam Filters"). In fact, our methodology highlights the two most critical criteria for selecting a commercial antispam product: low administrative overhead and content-analysis accuracy. Unless you're a spammer, junk e-mail doesn't add a cent to your bottom line--it just eats into productivity--so limiting the cost of controlling it is important.
On the other hand, a misidentified customer e-mail could cost your company, so we weighted accuracy, at 30 percent, as the most important measurement of an antispam product's performance. It's important to understand our definition of accuracy--it differs from vendor marketecture that pegs accuracy based on how well engines and signatures identify spam messages, without reporting how often they classify legitimate mail as junk. We've found that spam engines are all too likely to identify as spam legit bulk e-mail, such as mailing-list traffic, newsletters and user-requested product-offer e-mail.
For our purposes, we called spam that made it to our inbox a false negative, while legitimate e-mail that was identified as spam and stopped was considered a false positive. Because false positives are much more detrimental to organizations, for scoring purposes we considered them five times more costly than false negatives (one false positive = five false negatives). We included our nonweighted numbers for comparison, but we stuck with the weighted information for scoring (see "Spam Filter Accuracy Results," page 60).
Vendors report accuracy numbers that can exceed 99 percent; some even tout a zero false-positive rate. Our testing shows these claims are mostly marketing hype based on results of "tests" with custom-tuned software--something we avoided to see how accurate these products were out of the box. Not all shops have the wherewithal to tweak spam rules before mail is tagged incorrectly. As always, caveat emptor; we recommend test-driving any product before entering into a software agreement.
Symantec's Mail Security for Exchange misidentified only 12 messages we were supposed to receive, whereas Red Earth Software's Policy Patrol misclassified a whopping 643 false positives! Using our weighted accuracy, Cloudmark Server Edition and Symantec's Mail Security for Exchange scored 93.5 percent and 92.4 percent effectiveness, respectively, while Red Earth Policy Patrol posted a -10.7 percent score. Its score was so skewed it made us question our testing methodology--briefly. Red Earth relies heavily on outbound e-mail to tune its policies in real time, but we sent no outbound e-mail so its offering was severely hampered. In our invitation, we specifically stated that no training of Bayesian filters will be done. In addition, sending e-mail through spam filters is much less common than receiving it through them. Spam filters should be able to identify spam without relying on outbound e-mail--that's what we pay for. All other competitors survived the test bed.
Our test bed presented the products with another difficult challenge because we used real e-mail directed to Network Computing editors, including press releases, HTML-formatted industry newsletters, expletive-laden notes to deadline scofflaws and other spammy-looking legitimate missives that were tough to analyze correctly. Remember, too, that we emphasized out-of-the-box performance and defined accuracy in a certain way--your environment may not be as demanding.
Ease of administration and configuration are important with antispam software. Moving the burden to determine what is and isn't spam from users to IT administrators is a recipe for disaster--one person's spam is another's ham, so end users are best qualified to make this decision. Antispam software should assist users in doing so and give them full control over their message stores.
All the software suites we tested can deliver spam to a folder in the user's Exchange account. The only software configured out of the box to not provide the end user with access to spam-identified messages is Red Earth's Policy Patrol; here the burden falls on the administrator, though the admin can reset the configuration, a tedious process.
Web-based quarantines are becoming a popular selling point; only the Commtouch and MailFrontier offerings have Web quarantine interfaces. Both of these and Symantec's Mail Security also can notify users of spam with a summary e-mail. Even if you currently just tag messages and allow users to write rules, or automatically place the e-mail in their inboxes, that configuration choice may change over time.
Reporting is critical to justify the purchase of antispam software and can give the administrator an overview of how well the product is working. Well-polished trend reports help justify the yearly expenses associated with the care and feeding of your antispam strategy. Reports also give those pesky users who complain of a spam or two in their inboxes a reality check. We found MailFrontier's Gateway Server the only product to offer a full range of reports we would feel confident sending to upper management.
Virus scanning is a critical component of every e-mail and desktop security strategy. Although not all viruses slither in through e-mail, stopping any that do should be a basic function of e-mail security software. Exercising the products' virus-scanning engines was beyond the scope of these tests, but we expect them to work as advertised. Commtouch's Anti-Spam Enterprise Solution is the only package that doesn't include a virus-scanning component--a serious deficiency in a mail gateway scanner.
If you use a simplistic approach to filtering, as in Red Earth's Policy Patrol, you may as well just delete every other message and hope for the best. RBLs (real-time blackhole lists) are becoming increasingly useless because spammers now use infected computers connected to high-speed Internet connections to do their dirty work. And just scanning e-mail for common words or phrases found in spam will cause more false positives than any business should be willing to accept. Just because we send an e-mail that has "click here" or "unsubscribe" in it does not mean it's spam--avoid at all costs products that simply classify words or phrases. Red Earth uses both of these concepts in its product, which resulted in a poor showing in our tests.
Rather, look for an engine that computes hashes of body and header contents to make decisions. Most of these engines deliver updates locally through an active update process; however, Commtouch and Cloudmark take a different approach, requiring hashes to be sent to their servers for analysis. Results are sometimes sent back stored in local caches, which generally adds only seconds to the delivery process. Either approach will work. Because e-mail can't be delivered without an Internet connection, don't shy away from a service that relies on that same connection to process spam fingerprints.
A lot of spammers use semi-random word generation to get around filters that use hash-based approaches, but Commtouch and Cloudmark do not hash the entire content; because spammers get paid for clicks or image views, hashing these items and ignoring the rest makes their approaches effective.
If your data center includes clustered Exchange servers, make sure the product you select is cluster-compatible. Be careful of lingo like "cluster-aware"--that usually means the software can detect a cluster but can't work out problems that arise. And if your company has multiple entry points for incoming e-mail, be sure the software package can replicate its policies; otherwise, your administration efforts will increase and may be unacceptably error-prone.
What'll It Cost
The average price for the antispam software in this review is about $12.50 per user, per year. Our as-tested price reflects a 1,000-user license on a two-year agreement, extrapolated to per-user, per-year figures. Red Earth's Policy Patrol was the only software that did not follow the common per-user, per-year pricing structure. It has a one-time price of $3,423 for a 1,000-user license with a two-year commitment, which we normalized to $1.71 per user, per year.
Commtouch, Cloudmark and MailFrontier all hovered near the average while Symantec's pricing for its Mail Security for Exchange was almost triple the average. Red Earth's price of $1.71 is by far the least expensive, beating its nearest rival by nearly $7 per user. Note that Commtouch's price of $8.50 per user, while second lowest, does not include antivirus support--that's why we scored it fourth in our price comparison; we figured another software add-on would raise the overall price by at least $3 per user. Cloudmark Server Edition costs $9 per user, and MailFrontier's package is $11.60 per user over two years.
Symantec does offer a more aggressive discount if you purchase more products, but it's still an inflated price. Meanwhile, Red Earth's low-ball pricing teaches you to look at features and accuracy as well as cost when evaluating antispam software. Sometimes, you do get what you pay for.
After all the numbers were tallied, MailFrontier's Gateway Server ran away with our Editor's Choice award. This easy-to-configure software provided the best reporting features, turned in a weighted accuracy rate just less than 90 percent and offered a very competitive per-user price. Cloudmark made up for its lack of reporting capabilities with the second-best price per user. Symantec and Commtouch were not far behind, turning in scores close to our runner-up.
MailFrontier installed in a snap using a wizard installation routine common to Windows-based software. Gateway Server includes copies of Sun's Java Runtime Engine and Apache's Tomcat for the administration console. The installation wizard warned us about Gateway Server's capacious space needs--it requested 40 GB of free space for use with its Web-based quarantine, dubbed "Junk Box."
Once installed, we logged in to the Web-based administration console using the default user name/password. Here we entered the license keys provided by MailFrontier to switch on various parts of the software. We changed the Microsoft SMTP service to run on a different port than its default, and enabled MailFrontier's Gateway Server to bind to the standard SMTP Port 25 and forward messages to the Microsoft SMTP service. Some simple configuration was left, including tagging e-mail subject lines and writing Outlook rules for our tests. In all, installation and setup took about 20 minutes, including one reboot.
MailFrontier's spam engine design is simple, but highly effective. The software looks at three parts of each message: senders and their reputations, content of the message headers and bodies, and contact points (URLs, phone numbers, mail-tos) and their reputations based on their e-mail traffic. For each e-mail, the software determines the spamminess of each part; if two or more report it's spam, the message is tagged as spam. If one reports the e-mail as spam, it's tagged as likely spam. If no classification can be performed, the software considers the e-mail legitimate.
Once Gateway Server determines that a message is spam or likely spam, it can take various actions. We set the software to tag the subject lines for both spam and likely spam, but its default configuration will put these messages into a Web-based quarantine. We also could set the software to delete the message, return it to the sender, send it to another person or deliver it to the end user's inbox. This is the only software we tested that specifically works to detect phishing attacks by classifying their reputations in a different category. Because phishing attacks are fraudulent in nature, MailFrontier warns users prominently so they don't accidentally act on a legitimate-looking e-mail in the spam folder--a useful safety feature.
MailFrontier says administration of Gateway Server takes at most 10 minutes per week and that most customers spend closer to five minutes. We consider these numbers realistic. Bottom line, long-term administration of this product will not require you to hire more people, but expect an initial time investment to maximize accuracy through custom tuning. This is true of any antispam system.
In our tests, MailFrontier Gateway Server turned in the third-best weighted accuracy score, 89.6 percent. The software identified only 37 false positives and 150 false negatives, and had the second-best unweighted score.
|
|
Gateway Server's reporting features were the best in class. All reports were driven directly from its Web interface and were polished enough to pop directly into reports for upper-level management. The initial dashboard view provided a quick overview of how the software had performed over the past 24 hours. We also liked the canned reports that showed top senders and receivers and the domains from which they originated, plus bandwidth savings, ROI projections, spam/not spam graphs and more. If none of the more than two-dozen canned reports fits your needs, try the custom reports wizard.
MailFrontier Gateway Server 4.1, $11.60 per user (as tested). MailFrontier, (866) 3NO-SPAM, (650) 461-7500. www.mailfrontier.com
Another easy-to-install piece of software, Cloudmark Server Edition required almost no effort on our part. We simply created a user account and granted Enterprise Administrator and Delegated Exchange rights. Running the installation program required us to enter our activation code, the server IP address, and domain and user credentials, then check a box to configure the software to automatically place e-mail into each Exchange user's spam folder--a nice touch. Installation kept pace with other products tested, taking less than 30 minutes.
Cloudmark Server Edition uses a plug-in to the Microsoft Management Console, so Windows administrators will get up to speed quickly. The MMC application is incredibly simple. It integrated with Exchange easily, and Cloudmark says most administrators set options once and forget about it--seems likely, since there aren't many options to play with. This will please some, but we would have liked more opportunities to customize. Limited features and very limited reporting cost this otherwise accurate spam filter the top spot in our tests, but shops that do not require advanced graphical reporting or have very limited IT resources should give Cloudmark Server a walkthrough. According to the vendor, the software scales to around 2,000 users per install, and its sweet spot is customers with 25 to 500 Exchange mailboxes.
We found Cloudmark Server's filtering service similar to Commtouch's in that it has no locally updated content filter definitions. In fact, Cloudmark takes an interesting method of identifying spam: It relies on more than 1 million users of its Cloudmark Desktop antispam product to send spam that hits their desktops back to the company. Using a custom fingerprinting approach, the Cloudmark software computes hashes for various headers, body content and MIME attachments and asks the Cloudmark spam catalog servers to judge the spamminess of this e-mail. These servers report back to the Server Edition software, which takes appropriate action. Cloudmark Desktop users who report spam back into the company's servers gain credibility; the more times they're right, the more they're trusted. Users of Server Edition cannot participate in this trusted user feedback model yet, but should be able to in the next release, which will report back to Cloudmark when a user moves spam inside Exchange.
|
|
We guess 1 million end users can't be wrong: Cloudmark turned in the best weighted accuracy performance in our tests, at 93.5 percent, as well as the best unweighted accuracy, at 97 percent. Its software was second best in false-positive and false-negative scoring, with 28 and 69 messages respectively. There's no doubt its unique approach to identifying spam is effective. Reporting, however, was a weakness; Cloudmark Server offered us only the ability to see how much time and money we saved with the product. A Web-based quarantine was also missing. Cloudmark says it believes this confuses users more than it helps. We disagree but see where the company is coming from, given that it aims its software at very small shops.
Cloudmark, $9 per user. Cloudmark Server Edition 1.6, (415) 543-1220. www.cloudmark.com
Symantec leverages its recent purchase of Brightmail's antispam technologies for the Mail Security for Exchange with Premium Anti-Spam Service the company sent us for testing. The Premium Anti-Spam Service is essentially the Brightmail service integrated into Symantec's Exchange product. Mail Security can integrate with Microsoft Intelligent Message Filter and Spam Confidence Level settings as well, a feature that was missing from all other competitors.
Installation of the software was simple. We answered a few questions, chose not to restart IIS, set the IP and port for the Web-based administration console, configured an administrator e-mail address and installed the two licenses we received from Symantec--one for Mail Security and one to enable the premium module. Mail Security for Exchange includes a Spam Folder Agent, which can automatically direct spam messages identified by the premium antispam module to a user's spam folder using server-side rules, though this was not tested. Don't be tempted to cheap out on the premium service--you'll save money, but you'll end up with a basic heuristics engine, updated infrequently. Accuracy will be drastically different from our test results.
We did basic configuration of Mail Security for Exchange using Symantec's admin console. Configuration included configuring Live Update to get virus and premium antispam updates every hour, telling the Premium Anti-Spam Service not to tag suspected spam, and setting Premium Anti-Spam to tag subject lines of spam messages for delivery into our spam folders, all based on recommendations from Symantec. We also configured the software to save spam report data so it would be available later and told the virus scan engine not to notify senders or receivers. The entire setup process took less than 30 minutes.
Symantec uses 2.5 million probe accounts with major ISPs to identify spam outbreaks soon after they start; its Brightmail operations center constantly monitors these probe accounts to create and test definitions and update its spam filter database, which is kept in sync with Symantec's Live Update software. The company supplements that with more than 20 filtering techniques, including URLs, signatures and its own RBL lists. In Symantec's own testing, the Brightmail software boasts one of the highest accuracy rates at 99.99 percent--that's one false positive per 1 million filtered messages.
In our testing, Symantec tagged the fewest messages incorrectly as false positives, identifying only 12. Not as accurate as the vendor's own numbers, but twice as good as the next closest competitor. Overall, Symantec posted a weighted accuracy rate of 92.4 percent, second overall. Mail Security for Exchange did have 184 false negatives, indicating that it errs on the side of caution.
|
|
We found reporting in the Mail Security product weak, offering only very high-level reports. We had to export data to CSV for more advanced queries. The company says the next release will include the ability to e-mail reports on a scheduled basis.
All in all, the as-tested price of $32.25 per user, per year for two years seems pretty steep. Companies already doing business with Symantec will receive additional discounts.
Symantec Mail Security 4.6 for Exchange, $32.25 per user. Symantec, (408) 517-8000. www.symantec.com
The Commtouch product was one of the easiest antispam suites to set up--a plus for small shops. It took just 20 minutes. Using the wizard, we accepted all defaults, including an MSDE engine used to store spam statistics. Commtouch also can automatically configure an existing MS SQL installation, if chosen during initial installation. Customer sites with a large volume of mail will want to budget for MS SQL licenses to overcome MSDE's limitations (a 2-GB database size limit). After installation, we had to restart our computer.
CommTouch Anti-Spam Enterprise uses a polished Web interface for administration. We were impressed with its layout and think most IT administrators will like its intuitive interface. Commtouch recommended that we set up our Anti-Spam Enterprise software to ignore the Received From header lines from our CommuniGate Pro mail server, integrate via LDAP with our Active Directory install and adjust some report data-retention times. We also set the software to tag the subject line for spam and bulk messages, then wrote Outlook server rules to process all messages with those tags into our spam folder. Commtouch differentiates between spam and bulk messages, allowing administrators to set different actions for those types of messages, including redirecting them to a Web-based quarantine that users can manage.
The Commtouch suite uses a unique spam engine design that relies on proprietary RPC (Recurrent- Pattern Detection) algorithms to identify the key characteristic of spam: messages sent in bulk. Commtouch does not update typical content-filter definitions found in many spam engines; it searches out patterns instead.
When the Anti-Spam Enterprise gateway receives an e-mail, it attempts to match it to local policies that can be set enterprisewide or by individual users. If the e-mail does not match a policy, Commtouch will search the local cache of answers it previously received from the Anti-Spam Detection Center. If it still can't match a rule, the gateway software queries the Anti-Spam Detection Center housed at Commtouch's facilities; if the center is not available, the message will be placed in the user's inbox. Once a message is categorized as spam or bulk, the gateway takes the appropriate action, as configured by the administrator. Legitimate e-mail is delivered to the end user's inbox. Using its RPC technology, Anti-Spam Enterprise can classify e-mail as suspected spam and hold it for a predetermined time while it checks in again with the Anti-Spam Detection Center to see if the message has been classified.
In our tests, the Commtouch software allowed the fewest false negatives--only 48 messages reached our inbox that we would consider spam. Overall, however, Commtouch posted a weighted accuracy of just 74.5 percent, fourth in our tests, because of its 154 false positives. This accuracy rate is abnormally low, according to Commtouch, and was likely a result of the unique e-mail flow of bulk messages, like press releases, newsletter subscriptions and product announcements, our editors receive. The company claims a spam detection rate higher than 97 percent with custom tuning, which we specifically disallowed.
|
|
Reporting in the Commtouch Web interface was adequate; we could produce reports of top spammers and top recipients, with detailed views of each. We also could set reports to generate daily and save for later perusal. We missed having a custom reporting interface, which would greatly improve the product's reporting. More than anything, however, Commtouch was hurt by its lack of an antivirus engine; this brought down its features and price scores, as we would have had to purchase a separate e-mail AV system.
Commtouch Anti-Spam Enterprise Solution 4.1, $8.50 per user (as tested). Commtouch, (650) 864-2000. www.commtouch.com
Red Earth's Policy Patrol was a little more difficult to configure than its rivals. The problem was that we wanted all spam messages delivered to the user, but by default the software is configured to place most spam into a folder on the server. An administrator must then choose whether to delete or deliver the messages. Configuring the software to deliver messages to users' junk mail files was tedious--we had to do it for each filter. Red Earth should make this process easier for the administrator. Beyond that difficulty, a standard Windows-based wizard installation asked only a few questions to set up the software.
Add Your Comment: