Our labs and the host network at Syracuse University gave us more than enough devices and traffic to get a good feeling of how the products functioned. We installed all the packages on desktop or laptop Pentium III machines, each with 256 MB of RAM.
We limited ourselves to running the most basic versions of these applications to minimize their price. We felt that high prices and complex architectures would make it less likely the products would be used for quick diagnostics and tactical deployments. We know that in many cases these products are deployed as strategic solutions with a fair amount of success, but we didn't want to consider larger concerns such as redundancy, implementation and data access.
For comparison categories, we looked at FCAPS (fault, configuration, accounting, performance, security), with a substitution of inventory for accounting and with the addition of price. An inventory of the network is obviously part of an accounting system, but we didn't feel as though these tools were really trying to solve that larger problem. Instead, they can provide an inventory of what's on the network, vital when trying to diagnose problems like duplicate addresses.
We gave a slightly higher weight to performance and price in our report card--performance because it is generally a starting point for diagnostic procedure and price because of the tools' tactical nature. If you want to argue about this weighting, send in your arguments, and we'll point out why we disagree. Our price scores reflect only the products tested; add-ons and modules may increase cost.