|
||||||||||||||
  Lies, Damn Lies and Benchmarks May 31, 1999
The Man Behind the Curtain Since the original articles appeared, some facts have come to light. The most relevant is that the tests were conducted under contract to Microsoft and in Microsoft labs. That alone is enough to raise eyebrows. Further, it's been learned that the Linux server in question was not fully tuned. Mindcraft staff apparently were not particularly well versed in tuning Linux servers for performance. When the initial results were released, there was no indication that these tests had been done under Microsoft sponsorship, or that they had been performed at Microsoft's facilities. Interestingly, Mindcraft says that this "minor" omission of detail came about because the company was under nondisclosure with Microsoft, and secondly that it didn't want to incur the wrath of the Linux community. Well, incur wrath it has. I don't know whether Mindcraft intentionally biased its tests toward Microsoft, but it's natural to be suspicious. On Mindcraft's Web site, you'll find articles insisting that the company can't understand why the Linux community and others went ballistic upon learning that the tests were paid for by Microsoft and performed in its labs. The word clueless comes to mind. Mindcraft is now locked in a battle it can't win. The company is trying to save its name, but it's already lost on that point. The simple fact is that it ran a vendor-sponsored test on the vendor's grounds, and then failed to reveal these conditions when it publicized the results. Does Mindcraft deserve your trust? Of course not. If it was under nondisclosure, then fine, don't disclose. If it wanted to make the results public, it should have sought permission to tell the whole story and been ready to defend the tests. Mindcraft has since run a second test, and again there are disputes about whether the Linux systems were adequately tuned. So the company has proposed a third, public test. Should've thought of that a little sooner. Now it's too late. Vendors often release results of their own testing or of tests they've funded, and we all take the results with a grain of salt. We understand their motives and are appropriately skeptical. This is no different. That's not to say that vendor-funded testing is all bad. There are many good independent testers who are often hired by companies such as Microsoft. You virtually never hear of them, and that's as it should be. In order to serve the companies that pay for their services, they need to keep the results secret. If such results are meant for publication, they are clearly not impartial. Announcing them fundamentally taints the service the hired-gun testers provide. Why Mindcraft decided to publicly announce its findings is beyond me. If Microsoft wanted to solicit independent testing of its products, it should have done so for internal purposes and kept the results to itself. If Microsoft believes its products will outperform the competition, the company should participate in independent tests to show it. Go ahead, make a public challenge--the industry loves that stuff. In the end, however, you shouldn't really care all that much about how these tests come out. Performance is important, but so is stability, manageability, scalability and flexibility, and NT and Linux differ greatly in these metrics. The Mindcraft tests should be dismissed. The testing was sponsored, so we'll never really know why the testers selected the tests they did or why they released the numbers in the way they did. So don't waste any more of your time thinking about it. Send comments on this column to Art Wittmann at awittmann@nwc.com. |
Here
Here
In the tests, Mindcraft pitted NT running Microsoft IIS (Internet Information Server) against Linux running Apache and Samba. Mindcraft benchmarked both Web performance and file-service performance, and found Web service on NT to be 3.7 times faster than on Linux, and file service to be 2.5 times faster. Needless to say, this result immediately got the attention of the Linux community.
