From: groud...@iplus.fr (Gerard Roudier)
Subject: Unices are created equal, but ...
Date: 1996/04/13
Message-ID: <Pine.LNX.3.91.960413224027.150A-100000@gerard>
X-Deja-AN: 147397816
sender: n...@camelot.de
content-type: TEXT/PLAIN; charset=US-ASCII
organization: Mail2News Gateway at nasim
mime-version: 1.0
newsgroups: muc.lists.freebsd.hackers
Hi all,
I was implementing some performances enhancement for "Unix A" kernel.
Seems to work fine.
I had a look for another Unix in order to compare performances.
I had luck, since "Unix B" is installed on my machine on the same
hard disk.
Unix B is installed at the beginning of the disk media and Unix A at the end.
Unix B should have better IO throughput (see below if that's ok or not ok).
Then I run the first benchmark I found to prepare the tests.
I get the following results:
P90/Plato/24MB/NCR53C810/IBMS12.
BYTE UNIX Benchmarks (Version 3.11)
System -- Unix A gerard 1.3.87 #31 Sat Apr 13 18:34:46 GMT 1996 i586
Start Benchmark Run: Sat Apr 13 21:25:08 GMT 1996
1 interactive users.
Dhrystone 2 without register variables 121950.7 lps (10 secs, 1 samples)
Dhrystone 2 using register variables 121973.7 lps (10 secs, 1 samples)
Arithmetic Test (type = arithoh) 415167.1 lps (10 secs, 1 samples)
Arithmetic Test (type = register) 12996.9 lps (10 secs, 1 samples)
Arithmetic Test (type = short) 12121.0 lps (10 secs, 1 samples)
Arithmetic Test (type = int) 12998.6 lps (10 secs, 1 samples)
Arithmetic Test (type = long) 12993.7 lps (10 secs, 1 samples)
Arithmetic Test (type = float) 15954.7 lps (10 secs, 1 samples)
Arithmetic Test (type = double) 15946.0 lps (10 secs, 1 samples)
System Call Overhead Test 65139.8 lps (10 secs, 1 samples)
Pipe Throughput Test 68105.2 lps (10 secs, 1 samples)
Pipe-based Context Switching Test 22788.7 lps (10 secs, 1 samples)
Process Creation Test 774.4 lps (10 secs, 1 samples)
Execl Throughput Test 267.6 lps (9 secs, 1 samples)
File Read (10 seconds) 209771.0 KBps (10 secs, 1 samples)
File Write (10 seconds) 18000.0 KBps (10 secs, 1 samples)
File Copy (10 seconds) 4116.0 KBps (10 secs, 1 samples)
File Read (30 seconds) 212303.0 KBps (30 secs, 1 samples)
File Write (30 seconds) 18400.0 KBps (30 secs, 1 samples)
File Copy (30 seconds) 3441.0 KBps (30 secs, 1 samples)
C Compiler Test 120.4 lpm (60 secs, 1 samples)
Shell scripts (1 concurrent) 265.0 lpm (60 secs, 1 samples)
Shell scripts (2 concurrent) 139.0 lpm (60 secs, 1 samples)
Shell scripts (4 concurrent) 71.0 lpm (60 secs, 1 samples)
Shell scripts (8 concurrent) 36.0 lpm (60 secs, 1 samples)
Dc: sqrt(2) to 99 decimal places 9098.9 lpm (60 secs, 1 samples)
Recursion Test--Tower of Hanoi 2140.5 lps (10 secs, 1 samples)
INDEX VALUES
TEST BASELINE RESULT INDEX
Arithmetic Test (type = double) 2541.7 15946.0 6.3
Dhrystone 2 without register variables 22366.3 121950.7 5.5
Execl Throughput Test 16.5 267.6 16.2
File Copy (30 seconds) 179.0 3441.0 19.2
Pipe-based Context Switching Test 1318.5 22788.7 17.3
Shell scripts (8 concurrent) 4.0 36.0 9.0
=========
SUM of 6 items 73.5
AVERAGE 12.2
BYTE UNIX Benchmarks (Version 3.11)
System -- Unix B gerard 2.0.5-RELEASE XXXXXXXXXXXX:
Fri Oct 20 00:30:52 1995 gerard:/usr/src/sys/compile/GERARD i386
Start Benchmark Run: Sat Apr 13 21:48:12 1996
1 interactive users.
Dhrystone 2 without register variables 130585.3 lps (10 secs, 1 samples)
Dhrystone 2 using register variables 130526.3 lps (10 secs, 1 samples)
Arithmetic Test (type = arithoh) 413311.1 lps (10 secs, 1 samples)
Arithmetic Test (type = register) 12753.0 lps (10 secs, 1 samples)
Arithmetic Test (type = short) 12066.6 lps (10 secs, 1 samples)
Arithmetic Test (type = int) 12950.6 lps (10 secs, 1 samples)
Arithmetic Test (type = long) 12956.4 lps (10 secs, 1 samples)
Arithmetic Test (type = float) 17777.0 lps (10 secs, 1 samples)
Arithmetic Test (type = double) 17775.0 lps (10 secs, 1 samples)
System Call Overhead Test 45727.8 lps (10 secs, 1 samples)
Pipe Throughput Test 22094.0 lps (10 secs, 1 samples)
Pipe-based Context Switching Test 6304.2 lps (10 secs, 1 samples)
Process Creation Test 240.3 lps (10 secs, 1 samples)
Execl Throughput Test 68.1 lps (10 secs, 1 samples)
File Read (10 seconds) 115117.0 KBps (10 secs, 1 samples)
File Write (10 seconds) 3600.0 KBps (10 secs, 1 samples)
File Copy (10 seconds) 3457.0 KBps (10 secs, 1 samples)
File Read (30 seconds) 115920.0 KBps (30 secs, 1 samples)
File Write (30 seconds) 3533.0 KBps (30 secs, 1 samples)
File Copy (30 seconds) 3431.0 KBps (30 secs, 1 samples)
C Compiler Test 81.8 lpm (60 secs, 1 samples)
Shell scripts (1 concurrent) 118.0 lpm (60 secs, 1 samples)
Shell scripts (2 concurrent) 60.0 lpm (60 secs, 1 samples)
Shell scripts (4 concurrent) 30.0 lpm (60 secs, 1 samples)
Shell scripts (8 concurrent) 15.0 lpm (60 secs, 1 samples)
Dc: sqrt(2) to 99 decimal places 2518.2 lpm (60 secs, 1 samples)
Recursion Test--Tower of Hanoi 2147.1 lps (10 secs, 1 samples)
INDEX VALUES
TEST BASELINE RESULT INDEX
Arithmetic Test (type = double) 2541.7 17775.0 7.0
Dhrystone 2 without register variables 22366.3 130585.3 5.8
Execl Throughput Test 16.5 68.1 4.1
File Copy (30 seconds) 179.0 3431.0 19.2
Pipe-based Context Switching Test 1318.5 6304.2 4.8
Shell scripts (8 concurrent) 4.0 15.0 3.8
=========
SUM of 6 items 44.7
AVERAGE 7.4
Even if this benchmark is a little questionnable, I invite people who say or
write that Unix B is FASTER than Unix A to stop, or to say or write
the OPPOSITE.
Regards, Gerard.
From: j...@time.cdrom.com (Jordan K. Hubbard)
Subject: Re: Unices are created equal, but ...
Date: 1996/04/14
Message-ID: <5777.829437102@time.cdrom.com>#1/1
X-Deja-AN: 147776935
sender: n...@camelot.de
references: <Pine.LNX.3.91.960413224027.150A-100000@gerard>
content-type: text/plain; charset=ISO-8859-1
organization: Mail2News Gateway at nasim
mime-version: 1.0
newsgroups: muc.lists.freebsd.hackers
Considering that you did not even pick the latest release versions of
either OS (2.0.5 is close to a year old and 2.1.0-RELEASE has been out
for 4 months), these results are not even germain and you have no
excuse, except perhaps laziness, for not running these benchmarks on
more recent releases.
I could run benchmarks all day on aged Linux kernels, but what would
that prove? My stupidity, perhaps, but nothing more.
Jordan
From: t...@dyson.iquest.net (John S. Dyson)
Subject: Re: Unices are created equal, but ...
Date: 1996/04/14
Message-ID: <199604132351.SAA04861@dyson.iquest.net>#1/1
X-Deja-AN: 147831557
sender: n...@camelot.de
references: <Pine.LNX.3.91.960413224027.150A-100000@gerard>
content-type: text/plain; charset=US-ASCII
organization: Mail2News Gateway at nasim
mime-version: 1.0
reply-to: dy...@FreeBSD.org
newsgroups: muc.lists.freebsd.hackers
>
>
> Even if this benchmark is a little questionnable, I invite people who say or
> write that Unix B is FASTER than Unix A to stop, or to say or write
> the OPPOSITE.
>
I suggest that you benchmark recent versions of both (say FreeBSD-current
vs. Linux-current). FreeBSD fork/exec perf has gone up significantly,
among other things. Please compare equivalent vintages.
Note also that the Byte/Lmbench benchmarks DO NOT measure systems under
significant VM load. Remember, simple algorithms work quickly until
you actually use them.
John
dy...@freebsd.org
From: j...@time.cdrom.com (Jordan K. Hubbard)
Subject: Re: Unices are created equal, but ...
Date: 1996/04/15
Message-ID: <16036.829530040@time.cdrom.com>#1/1
X-Deja-AN: 147670020
sender: n...@camelot.de
references: <199604142207.RAA00341@dyson.iquest.net>
content-type: text/plain; charset=ISO-8859-1
organization: Mail2News Gateway at nasim
mime-version: 1.0
newsgroups: muc.lists.freebsd.hackers
> How's about maintaining competition, and working to make the respective
> U**X clone better competitively. Competition helps keep each party
> honest!!!
Just to make a general point here, and *not* to continue the flame war,
I think John's absolutely right.
Consider, for a moment, if FreeBSD or Linux were the only free
operating system. Who would drive us? The commercial vendors?
Unlikely, since we could simply escape that comparison by saying "hey,
we're free and they're not!" With the current situation, there is
_significant_ motivation to not fall too far behind in contrast with
the other group - sort of like two brothers growing up where one can
run just a little faster than the other. Each will be constantly
motivated to try and win the next foot race by pushing himself that
much harder, the loser being motivated in turn to redouble his
efforts.
I honestly cannot imagine a world where there was only one free OS to
choose from, and I think things have actually worked out far better
than I'd have even dared hope.
Jordan
From: j...@time.cdrom.com (Jordan K. Hubbard)
Subject: Re: Unices are created equal, but ...
Date: 1996/04/15
Message-ID: <16058.829530125@time.cdrom.com>#1/1
X-Deja-AN: 147684922
sender: n...@camelot.de
references: <9604142231.AA22327@gte.esi.us.es>
content-type: text/plain; charset=ISO-8859-1
organization: Mail2News Gateway at nasim
mime-version: 1.0
newsgroups: muc.lists.freebsd.hackers
> Who cares? linux is such a better name compared with *BSD*.
But we have a nicer looking mascot.. :-)
Jordan
From: a...@cymru.net (Alan Cox)
Subject: Re: Unices are created equal, but ...
Date: 1996/04/15
Message-ID: <199604150332.EAA05825@snowcrash.cymru.net>#1/1
X-Deja-AN: 147694966
sender: n...@camelot.de
references: <199604141524.RAA03026@x14.mi.uni-koeln.de>
content-type: text/plain; charset=US-ASCII
organization: Mail2News Gateway at nasim
mime-version: 1.0
newsgroups: muc.lists.freebsd.hackers
> The (old) BYTE benchmark is not a suitable performance
> indicator at all! I've made it a port under BSD, since
> it can show misconfiguration and problem areas, and I'd
> like to know, whether you at least used that version
> (available as a "port" or "package" for easy installation).
Can we flush this whole thread down the bitbucket. lmbench is just
a very low level analysis, the byte benchmarks are near enough broken beyond
usefulness. Until people are doing formal SPEC and transaction benchmarks
we wont have good answers (and some people maintain those are garbage too
;)). As various people have observed nobody can agree what to compare, what
benchmark is right and what it means anyway.
Alan
From: torva...@cs.Helsinki.FI (Linus Torvalds)
Subject: Re: Unices are created equal, but ...
Date: 1996/04/15
Message-ID: <Pine.LNX.3.91.960415083932.30002I-100000@linux.cs.Helsinki.FI>
X-Deja-AN: 147701404
sender: n...@camelot.de
references: <199604142055.NAA05263@ref.tfs.com>
content-type: TEXT/PLAIN; charset=US-ASCII
organization: Mail2News Gateway at nasim
mime-version: 1.0
newsgroups: muc.lists.freebsd.hackers
On Sun, 14 Apr 1996, JULIAN Elischer wrote:
>
> Ok, here is my attempt at the missing information....
This is probably approaching being closer, but it's not really as
clear-cut as even this.
Personally, I _want_ linux to be faster on everything, but when it comes
to real life, I'd say that _any_ of the Free UNIXen are "fast enough".
There is always going to be some differences in different areas, but with
both groups being pretty performance-minded I don't really think we'll
ever have any really noticeable differences in any "normal" circumstances.
> I would expect the following to be true between Linix 1.3.x and FreeBSD 2.0.5.
> category 1:--Linux faster in:
> context switch, including some system calls.
> possibly some program startup.
> possibly pipe code.
> possibly FP emulation.
> possibly FP exception handling.
> Any test that does a lot of filesystem meta-data manipulation.
> e.g. file creation and deletion.
I can't really say much about it: I can't even claim to have benchmarked
any *BSD machine. All the things you mention are "good" under linux,
though, so I'd suspect that linux is at least on par with BSD in any of
the above and may be faster.
On the other hand, Dyson tells me that the FreeBSD team has been
optimizing lots of those things too, and I wouldn't be surprised if they
are more or less at the same level as linux is..
Again, I don't think the speed makes a huge difference (a 10% difference
looks huge in benchmarks, but has little meaning in real life unless
either is 10% faster at _everything_)
> category 2:--FreeBSD 2.0.5 faster in:
> Anything to do with networking
No longer true. These days it's "some things to do with networking", and
while I suspect that BSD may have the edge here still, these days it's
really the _edge_, not the whole blade ;-)
> Anything using a raw tape or disk device.
This may well be true. I'll be the first to admit that I don't really
care for "raw devices". The only time I ever use the raw device is when
fsck runs, and I couldn't care less about performance there (well, I
could, but you get the idea).
Tapes I have no idea about. You may or may not be right.
> Any benchmark that loaded the system very heavily,
> especially if it produced swapping.
The differences shouldn't be that large any more. The asynchronous
swapping code and the swap deamon in newer linux kernels help performance
under load. Again, I'm more inclined to think that one system may be
better on some hardware, while the other might be better at something
else.
> Any benchmark that tested high-speed large sustained
> IO to files.
This has actually changed, and I suspect that's why Gerard did the
benchmark in the first place. He's been working on that part of the code,
and we're doing very well.
Unlike raw device accesses, I consider filesystem access speed very
important, and I've mostly concentrated on getting good performance
through good caching. Linux traditionally hasn't been as good in things
where caches don't help (ie the cases you mention above), but that has
improved a lot lately. Maybe somebody has access to FreeBSD-current and
Linux-current on the same machine and can compare bonnie?
> category 3:--Linux and FreeBSD 2.0.5 about equivalent in:
> Anything that relies mostly on plain CPU
> with no or little OS involvement.
> (as both use the same cpu.)
I'd like to put a lot more here. In practice I don't think the
differences are so large.
HOWEVER, despite the fact that I don't think it matters in real life, I'm
all for people doing benchmarks, and crying out when one or the other is
slower in something. Not because I think it makes much of a difference for
normal users, but because it's good for development. It's a great way to
motivate people to do better (show that the competition can do better at
something, and you force us to try to improve ourselves).
I think _that_ is why benchmarks are important, not so much for testing
which one is more "worthy"..
> > If you are a FreeBSD-current user and if you have about the same
> > configuration as mine, can you run the old BYTE benchmark
> > and send to me your results?
>
> I don't think it would be useful unless we had EXACTLY the same hardware..
> I have seen small differences make up to 50% difference..
Indeed. Even on the same machine the placement of the partitions can make
a noticeable difference for disk tests, so benchmarking is not a trivial
thing.. It might still be interesting to see some kind of benchmarking
done, just for "some data" as opposed to "THE data".
If somebody wants to do benchmarking,I'd suggest using at least
- lmbench (nice microbenchmark)
- bonnie (reasonable disk performance benchmark)
- webstone (or something similar. But use "apache" as the server, not
some braindead horror like NCSA).
- ???
(the three mentioned should cover different areas, all very reasonable,
but have I missed some important area?)
Linus
From: t...@dyson.iquest.net (John S. Dyson)
Subject: Re: Unices are created equal, but ...
Date: 1996/04/15
Message-ID: <199604151339.IAA08978@dyson.iquest.net>#1/1
X-Deja-AN: 148246133
sender: n...@camelot.de
references: <Pine.LNX.3.91.960415091252.30002K-100000@linux.cs.Helsinki.FI>
content-type: text/plain; charset=US-ASCII
organization: Mail2News Gateway at nasim
mime-version: 1.0
reply-to: dy...@freebsd.org
newsgroups: muc.lists.freebsd.hackers
>
>
> On Sun, 14 Apr 1996, John S. Dyson wrote:
> >
> > How's about maintaining competition, and working to make the respective
> > U**X clone better competitively. Competition helps keep each party
> > honest!!!
>
> I'd be nervous about the benchmarks used. It's too easy to optimize for a
> specific set of circumstances, and getting blind to the "larger picture".
> But with a reasonable selection of benchmarks, this might not be a bad
> idea.
>
> Linus
I agree with your sentiments. You have stated EXACTLY the concerns that
I have been expressing about depending solely upon benchmarks like lmbench,
iozone, bonnie, etc... I have some that I use that does show *significant*
differences because of loading issues. My benchmarks are in rough shape
and are not for public use (for reasons other than the simplistic nature
of the above mentioned benchmarks -- I am not comfortable with the
prettyness of the code.)
It would be nice if we could somehow improve on the quality of the synthetic
benchmarks available in the free realm.
John
dy...@freebsd.org