We ran the Kanisa, Mondosoft and dtSearch products on Windows 2000 Server (SP3) with dual Intel Pentium III processors (1 GHz), 1 GB of RAM, and gigabit network links. Panoptic provided its own Linux operating system using the same hardware platform but it ran over a 100-Mbps network link. The search engines were not tested for speed, therefore, the different network links had no impact on the test results.
The search engines were tested for performance on Network Computing's production Web site (www.networkcomputing.com). In addition, search engine features were tested on the magazine's production site, Syracuse University Web sites (www.syr.edu/*), a test server in our Syracuse University Real-World Labs® (Sunfire 280R, Solaris 9, Apache 1.3), and the author's secure intranet server.
To test search performance in the real world, we reviewed the log files of real key word searches made against our production site from January to August 2003. The key words we repeatedly searched are shown in the table below.
We studied the documents returned to determine their relevance. We tested performance by determining each of the engines ability to navigate to certain Web pages on NWC's production Web site. We used a few of the searches above and simulated searches designed to find identified Web pages.
For example, when the keywords fluke network inspector were searched with each of the engines, all of them returned the Sneak Preview of Fluke's Network Inspector. But not all the engines ranked that particular document first. One ranked it second and another ranked it third. In this case, the engines that ranked it first would receive one point. The engines that ranked it second would receive two points. And the engines that ranked it third would receive three points. The engines with lower scores win (see performance results).