Doculabs' Assessment Method

Critical to the success of a product assessment effort is the use of a proven methodology that delves into detailed criteria but still maintains perspective on the "big picture" technical and business issues. This was certainly the case with Doculabs' Business-to-Business Electronic Commerce Solutions assessment--we used a defined methodology that centered around a set of detailed test criteria, a sample application and a two-day-per-product hands-on evaluation process in which we used each product to build the sample application.

Our first step was to devise a comprehensive set of criteria that would help differentiate the various offerings in the market. We enlisted the experience and knowledge of internal analysts, end users and the vendors themselves to help refine and solidify the evaluation criteria. We compiled more than 60 detailed items, which were grouped into 10 high-level criteria categories, as shown in the table on page 80.

We developed an initial test vehicle--a sample application for an online bookstore--that was broad enough to exercise each product's capabilities and assess its performance in each defined area. We provided the vendors with the test vehicle and asked them to develop a portion of the application before beginning the two-day laboratory assessment.

As for the test environment, most products we evaluated support Windows NT and one or more flavors of Unix, so we equipped our lab with a base configuration of Windows NT 4.0 with Service Pack 3 running over an Ethernet TCP/IP-enabled network with access to the Internet through our firewalls. For servers, we used dual Micron Pentium II machines running at 200 MHz with 128-MB RAM and 4.3-GB HDD.

There were some minor variations to the standard configuration. For example, OpenMarket brought its own server to avoid potential difficulties with the configuration of the Doculabs' firewalls. Netscape administered and developed its applications completely through a Netscape 4.5 browser and therefore did not need to use the servers we provided. Microsoft and IBM both installed their products on the servers. On the database side, Microsoft used Microsoft SQL Server 6.5 with Service Pack 4 and IBM used DB2 Universal Database 5.0.

--Gautam Desai, analyst, Doculabs (www.doculabs.com)