Bruce Weiner: Lies, Damned Lies, Statistics

by Alan Cox

Linux Today

May 5, 1999

"Mindcraft did conduct a second test with support from.."

They (or rather he as it appears to be) _refused_ us access to the system. I have basically given up on Mindcraft's Microsoft funded pranks.

The careful use of the word "support" to imply we somehow validate his test is misleading. I sent him about four emails.

Only on May 4th did he offer any kind of open testing. The response I sent him was very simple

> I think I would prefer to see an open benchmark done on your
> configuration and test plan by an alternative testing body. That
> would also be better for all parties.

I think I speak for much of the community in feeling that. No such test has yet occurred. I have no faith that Mindcraft can or would deliver an accurate assesment of the general question of Linux v NT performance.

File Server Performance
-----------------------

I would also invite Mindcraft to publish the Windows NT client numbers using NTfs. That is the equivalent benchmark to using Linux where you also have file permissions and multiple timestamping on files. Perhaps Mr Weiner has conveniently forgotten that Windows 98 is supposed to be the end of the line and Windows 2000/NT5, an NT based product is the future according to the MS roadmap.

Bruce says that "Mindcraft verified the clients were set up as we documented". I do not believe anyone can verify an OS client was set up as documented by a company engineer. This comment implies that Mindcraft did not set up the client machines themselves. I would hope they followed basic test quality considerations and set the client machines up using install disks bought at a random computer shop and that no third party engineer was able to access the test site except under supervision. Who did set the client machines up ? Because Bruce refused us access we don't know.

Bruce Weiner says that Jeremy reported he duplicated the NT server numbers in their lab. Jeremy also could not reproduce the abysmal numbers Mindcraft got from their Linux box.

"The only reason to use Windows NT clients is to give Linux and Samba an advantage"

Using NT clients is a more accurate measurement. Using NTfs would also be more accurate. The file systems then are equivalent in feature sets. It is also rather easier to tune a FAT file system to put the files and blocks in perfect order for a test. I'd thus want to be sure that the files were unpacked with a standard open source tool on a newly formatted partition on each machine so we can be sure nobody used any kind of fs optimiser on either the NT or Linux box.

I could equally say

"The only reason to use Windows 98 clients its to give NT an advantage".

In fact since Microsoft supply both products in this case it is a much more reasonable statement.

So why don't they include NTfs benchmarks. I can only guess that Bruce's statement answers that in full, as well as telling a whole story

"...why the company that sponsored a comparative benchmark always came out on top. The answer is simple. When that was not the case our client exercised a clause in the contract allowing them to refuse use the right to publish the results. We've had several such cases"

Flaws In Test Analysis
----------------------

If Mr Weiner wants to get accurate OS assesments for for web performance he should run Zeus. Last time I checked Zeus was the fastest Linux web server, and IIS seems to be the fastest NT web server. Linux + Zeus also appeared to be cheaper than NT Server (including IIS). He acknowledges that Apache is not the fastest web server for Linux, yet he did not choose to use the obvious product for the testing.

He claims

"We showed that Windows NT server was faster than Linux"

He states

"None of the Unix benchmark results reported at SPEC use Apache"

Why then did he use Apache. He may be measuring Apache v IIS speed but he is not measuring Linux v NT speed, and he knows it.

He claims

"We showed that NT server was faster than Linux"

He states

"Mindcraft used a MegaRAID controller with a beta driver... PC Week server used an eXtremeRAID controller with a fully released driver. The MegaRAID driver was single threaded while the eXtremeRAID driver was multi-threaded"

By his own admission he is benchmarking an incomplete test driver and claiming that it proves NT server is faster than Linux.

I would like to see an honest truely open benchmark using the best tools for each platform and showing the areas where each one wins. I am sure there are some things NT is good for (the cynical response of BSOD aside). Only an open benchmark by a good responsible and trusted third party can provide the information users need and have a right to know when selecting the right OS product for a job.

End users have a right to honest open benchmarks they can trust when selecting a product. It is sad that the prospect of an open choice seems to scare some large vendors.

Copyright 1999