The performance of Critical Path's InScribe Messaging Server was exceptional. However, a lower-than-desired price/performance ratio coupled with an extremely high error rate kept this system off our shortlist for the MediaMakers bid. The errors were produced by slow server responses to IMAP requests. The IMAP requests timed out at an alarming rate as load increased. We believe the IMAP errors could have been reduced or eliminated with proper tuning. Because of the late arrival of Critical Path's hardware and additional hardware-related problems during setup, however, we ran out of time for additional fine-tuning. We were impressed enough with the performance we saw that we put Critical Path's bid for the NetMagik RFP on the shortlist--with one caveat: The costs for software and support need to come down considerably for this solution to be a serious contender.

Critical Path's NetMagik bid specified a two-tier infrastructure with four front-end and two back-end servers. One of the back-end servers functioned as a hot spare. All systems were connected to an NFS-based storage array. Access to the front-end systems was mediated by DNS round-robin. A front-end system contacted the back-end server to request the user's mailbox location. The front-end system then made a direct connection to the mailbox on the NFS store on behalf of the user and serviced all future requests by that client.

Remote system management was accomplished via a Java-based application. Surprisingly, installation of the remote-management application was primarily a manual process. On the plus side, the application was responsive and full-featured, giving administrators a "single-system" view of complex installations.

Network Computing's Evaluation of iPlanet's Proposal

Unfortunately, iPlanet wasn't ready for this RFP, and its responses and performance suffered as a result. We know that iPlanet has a respectable market presence; therefore, we are hesitant to draw lasting conclusions from one bad showing, so we'll let the facts speak for themselves and refrain from making harsh judgments.

The fact is that iPlanet couldn't get its act together for this RFP, even with a three-month lead time. The vendor missed the deadline to sign up for a testing slot, then complained that it couldn't make the only slot left. We accommodated it by pushing its appointment into a slot that seriously jeopardized our own deadline. Engineers were scheduled to arrive at our labs on a Monday, and iPlanet's equipment was due the Wednesday before. By the end of that Wednesday, the equipment still hadn't arrived, so we called the company and found out it was still configuring the system. We were assured it would be in by the end of the week. Friday came, and still no equipment.

Tuesday morning, the iPlanet team showed up bright and early with equipment (but no bagels) and spent the rest of the day--our only day left for testing--trying to get the system to work. It hadn't been configured completely or properly before it was shipped, and its key administrative passwords weren't known. By the end of the day, it was obvious that the configuration still wasn't completely right, but our tight schedule forced us to start the testing.

Not surprisingly, iPlanet's solution posted the lowest price/performance ratio of the systems in the review. The performance numbers were in the lower half of the systems tested, and its error rate steadily mounted as the number of threads increased.

To serve the needs of 10,000 e-mail users, iPlanet specified a $428,000 system. The software alone cost $230,000; excluding Novell's submission, this surpassed the entire cost of any other e-mail system in the review, including hardware, software and support. The hardware comprised a Sun Enterprise 450 attached to two T3 fiber-attached RAID arrays for the message store and two Enterprise 250s, one of which served as the LDAP directory server and the second as the mail front end or multiplexor. The mail configuration and directory data were stored on D1000 RAID arrays. The LDAP server was also connected to the T3 RAID array, and Veritas Software provided the mechanism for server failover. Inexplicably, the network adapters in all the systems were 100 Mbps rather than 1 Gbps. The system design would have been appropriate for a four-nine uptime requirement but was overkill for MediaMakers' three-nine requirement.

IPlanet agreed with this assessment. Even with the lower hardware costs of a three-nine solution, the bid was dragged down by the software price. We couldn't run the Java-based Management Console for the iPlanet system on our Windows 2000 workstation. It ran fine on the Sun box, and iPlanet assured us it would work on a Windows NT box. We didn't have time to set up an NT box to confirm this claim, but we'll take the company's word for it. Representatives said they didn't think anyone had ever tested the Management Console on Windows 2000.

Network Computing's Evaluation of Mirapoint's Proposal

We were impressed with Mirapoint's submission for MediaMakers' RFP. In this RFP, only Mirapoint offered a turnkey solution. Within a few minutes of unpacking the system, we were up and running. As the system booted, it requested an IP address, mask and gateway address, which we entered through a small numeric keypad on the front of the server. Once these were configured, we switched to a browser, downloaded and installed a Windows management application, connected to the server and finished the configuration. It couldn't have been easier.

During our benchmark dry runs, we could see the SMTP connections by the clients were timing out. A tunable parameter on the server increased the threads available to service SMTP requests. We gave this parameter the highest setting. This increase helped but wasn't enough to prevent SMTP time-outs under heavy loads. We believe it was this problem that led to a failure of the benchmark program at the 900-thread level. Mirapoint servers support only up to 100-Mbps Ethernet connections. A gigabit interface would have eliminated this bottleneck.

If this system had completed the benchmark, it would have been on our shortlist for the corporate solution. We withheld endorsement because the failover mechanism for hardware required operator intervention. Mirapoint says it intends to address this shortcoming. Also, its solution for the two-tier service option required all POP users to be resident on one server and the IMAP/Web users to occupy the other two servers. When a user decides to upgrade service to take advantage of IMAP or the Web, the mailbox must be moved from the POP server to one of the other servers. This operation could be scripted to cut overhead, but we'd like to see a more elegant solution.

Network Computing's Evaluation of Novell's Proposal

Novell offered a cluster-based bid for MediaMakers' RFP that was almost identical to Rockliffe's submission. Two Compaq Computer Corp. servers were connected to a fiber-based, shared RAID array. The servers ran NetWare 5 SP4, NDS eDirectory, Novell Cluster Services 1.01 and NIMS 2.5, which was released just in time for this RFP. We've tested NIMS before and have always been impressed with its performance. In fact, we recently named it a Well-Connected Award winner in the category of Enterprise Mail System (see "Web-Based E-Mail" and "Big Three Fine-Tune Offerings While Trying To Expand Their Reach").

This time, however, we found that NIMS' performance wasn't up to snuff. Novell officials pointed to our benchmark and test bed as the culprits, but the other systems performed well under identical circumstances so we don't think this gets to the heart of the matter. We can offer three possible explanations but can't say for sure why NIMS failed to meet our expectations in this RFP.

Second, when the benchmark threads completed an IMAP transaction, they dropped the connection and reauthenticated before the next transaction. Because this process happened thousands of times during the course of the benchmark, any systematic delay in responding to the authentication request would be a disaster.

The final and perhaps most plausible explanation stems from the high CPU utilization we saw during the benchmark. Because the NetWare OS, the NetWare TCP/IP stack and, subsequently, NIMS can take advantage only of a single processor, it's reasonable to blame a processor bottleneck. The Rockliffe system was nearly identical to the Novell system but had a second processor.

Even on that system, the combined CPU utilization approached 100 percent. We suspect that Novell's system would benefit from the same boost. This, of course, will depend on Novell's addition of multiprocessor support in its next NetWare release, which is code-named "Six-Pack."

In addition to our concerns about performance, the pricing scheme for Novell's bid put it near the top for the MediaMakers RFP and at the top for NetMagik, at $4.4 million. The software and support costs drove the price of Novell's bid for NetMagik into the stratosphere.

Send your comments on this article to Ron Anderson at randerson@nwc.com.