Network Computingwww.networkcomputing.com

We also added a twist to the typical Network Computing RFP by testing the performance of the 10,000-user mail systems proposed for MediaMakers. These tests took place in our Real-World Labs® at Syracuse University.

Unlike highly functional (and complex) messaging servers--Lotus Notes, Microsoft Exchange and Novell GroupWise--the LSMS (large-scale messaging system) is only for e-mail and relies on standards-based transport and access protocols throughout. These systems are designed for massively scalable throughput, high reliability and delegated administration.

To get an invitation to participate in the RFP, a vendor needed to have an established presence in the ISP arena as well as a corporate offering. Software.com, one of the leaders in the ISP arena, doesn't have a corporate offering, so that company wasn't invited to participate.

Based on our testing, each of these 10,000-user systems generated enough throughput to handle MediaMakers' e-mail requirement, with plenty of room for growth. Even the slowest system in the test, Novell Internet Messaging System (NIMS), processed nearly 136,000 messages during the five hours our benchmark ran. This was slow only by comparison. If we had stretched the five hours to 24, the system would have processed 653,846 messages. Keep in mind that we tested an IMAP mail system, not a less compute-intensive POP system.

Uptime All the Time?

The MediaMakers RFP required a 99.9 percent uptime guarantee. Three-nine uptime permits about 10 minutes of downtime a week, or nearly nine hours a year. In our estimation, requiring three-nine uptime doesn't automatically relegate the buyer to costly clustering/high-availability solutions. Three-nine uptime can be met by stocking hardware spares, implementing RAID-based storage and having an established, written procedure in place for swapping spare hardware for failed hardware.

Some vendors--namely, iPlanet, Novell and Rockliffe--offered cluster-based solutions that wouldn't require operator intervention if a server failed. Critical Path specified a hot-standby solution with a second mail server; it was connected via NFS (Network File System) to the storage array, ready to step in via operator intervention if the first server failed. Mirapoint and Stalker Software specified a cold-spare solution that would also require operator intervention in the event of a failure. We believe that any of these solutions would meet the three-nine requirement of the RFP.

The NetMagik RFP required a 99.99 percent uptime guarantee. Four-nine uptime permits one minute of downtime per week, or 52 minutes a year. The NetMagik uptime requirement could be met only by implementing a clustering/high-availability solution. All our vendors specified a multitiered, multiserver solution to achieve four-nine uptime as well as scalability.

And the Winner Is...

We scrutinized each bid to make sure it met the uptime and functional requirements specified in the RFP. Functionality, manageability, support policies and costs, price, and performance were central factors in determining which vendors made the shortlist as well as in determining the winner. Use these proposals as a guide to create your own shortlist: Examining each vendor's solution will give you insight into how to implement a large-scale e-mail solution of your own.

In the end, both MediaMakers' and NetMagik's bids went to Stalker Software's e-mail solution, which featured CommuniGate Pro. CommuniGate Pro wasn't the fastest mail server in the roundup (Critical Path's InScribe Messaging Server beat CommuniGate Pro's overall speed), but it put up the highest price/

performance numbers as well as the lowest error rate in the group. CommuniGate Pro worked flawlessly on the least expensive hardware during the performance testing and nailed the functionality requirements.

Rockliffe made the shortlist for MediaMakers, and Rockliffe and Critical Path made the shortlist for the NetMagik RFP. Assuming that Microsoft Windows 2000 proves its mettle during the next year, we think Rockliffe's entry will be worth a very close look even in the ISP community. Based on performance alone, Critical Path's solution should be on the shortlist--but only if the company brings its software and support costs in line with the other vendors in this RFP.

Each of these vendors would be given further consideration in a purchasing situation. You'll find summaries and evaluations of these six vendors' proposals below.