Network Computingwww.networkcomputing.com

Network-simulation products address two different audiences: First, they're for service providers whose businesses depend on their abilities to provide services via WANs and maybe even the WANs themselves. Second, they're for users of WAN-oriented services--both dot-com and brick-and-mortar companies--that are looking to outsource central services, or simply to extend service to customers and suppliers. For the first audience, the simulation tool must be able to help with the details of managing the network infrastructure. For the second group, the tool needs to provide an end-to-end look at the services deployed to customers and trading partners. How the packets get routed and protocols get tuned is left up to the folks in the first group.

We tested four network-simulation packages--NetRule 2.3 from Analytical Engines, EcoPredictor 3.0 from Compuware Corp., NetCracker Professional 3.2 from NetCracker Technology Corp., and IT DecisionGuru 6.0 from Opnet Technologies (formerly MIL 3)--by running known transactions across an established but dynamic network core in our Real-World Labs® at Syracuse University. NetRule came out on top, letting us create our network model easily, quickly, accurately and inexpensively.

IT DecisionGuru, which boasted the most extensive configuration options of any of the products tested, finished second primarily because of its high price and slow performance. However, its breadth of features and its accuracy are excellent. EcoPredictor and NetCracker offered many features comparable to those of IT DecisionGuru, but both were lacking in flexibility as our models grew; the larger our models got, the more tedious these two products were to use.

The accuracy of these tools has more to do with manipulating the simulation models than with the magic of each product's math engine. Software of this type is only as good as the data it's given. It's no surprise, then, that all the vendors hedged a bit about how accurate their products' predictions would be, since they can't control what information the products are fed. Also not surprising, we found in the course of our tests that the fewer items we tried to configure and the less complicated the process, the more accurate the results were, simply because there were fewer opportunities for us to screw up.

Nevertheless, to achieve true accuracy, we had to spend time validating the results of our simulations. Each product required that we run multiple simulations to obtain accurate results: We'd set up a topology and an application, run a simulation and analyze the results. Then we'd make some changes, run another simulation and so forth, until we got results that made sense. This changing and re-evaluation was the most significant part of the simulation work.

All the products we tested let you import current network data; that is, you can dump your network into one of these packages and start simulating changes. As appealing as this is, it's not at all practical for several reasons: First, networks aren't static; getting a representative snapshot requires more than a bunch of packet traces, which remains the primary method of collecting packet and transaction information. Second, the topology from network-management stations is wrong (networks change, and port-level connectivity is still a myth). And finally, we found the matching of topology to transactions and background traffic to be tedious and error-prone--way more trouble than it's worth. It's much more reliable to focus simulation on a particular problem; this makes the exercise manageable, with results that can be verified.

Beneath the surface of all the software we tested lies the science of serious statistics: Each product uses various statistical distributions to help describe the types of traffic patterns we experienced. So expect to take some time to understand Bernoulli, Poisson, exponentials and so on. In the next generations of these products, their vendors should explain the differences in accuracy in English, and make some intelligent choices in the trade-offs between perfect accuracy and reasonable assuredness.

How We Tested
We monitored each interface, feeding our backbone for utilization, and ran an HTML-page download and a DNS from our lab across the backbone to different servers for three months. Once we had established a baseline, we took snapshots of the backbone's background utilization and overlaid these two transactions. We didn't expect to see exact results, partially because the nature of the network was always changing, but we did get a good idea of what the values were over time. We used NextPoint Networks' S3 and Lucent Technologies' VitalSuite software to map average, peak and 90 percent of peak baselines over the three months.

Although most of the tested products provided canned transactions, which seemed generally accurate, we chose to create customized transactions in order to achieve the highest accuracy possible.