Path: utzoo!attcan!uunet!nbires!ico!rcd
From: r...@ico.ISC.COM (Dick Dunn)
Newsgroups: comp.arch
Subject: RISC bashing at USENIX
Message-ID: <6888@ico.ISC.COM>
Date: 7 Jul 88 19:54:49 GMT
Organization: Interactive Systems Corp, Boulder, CO
Lines: 69
[Better start off by emphasizing that these are my own opinions, not ISC's.]
"Neal Nelson and Associates" had a booth at the USENIX vendor's exhibition
whose sole purpose seemed to be RISC-bashing. Although they purportedly
have developed a set of tests, which they call their "Business Benchmark
(R)" for helping people make realistic comparisons of machines under
realistic loads, in fact even basic descriptions of the tests cast serious
doubt on their real usefulness. Some of the tests seem overly simplistic;
others contain obvious biases toward or against certain types of hardware
and/or I/O system software.
A couple of months ago, several trade rags ran articles reporting how Neal
Nelson & Associates (NNA) had shown that CISCs would beat RISCs. It's
hard to tell just who botched what part of it--whether NNA did a bad job of
reporting it or the trade journals reported what they wanted to hear--but
the articles were abominable. At USENIX, the NNA booth had great piles of
reprints of several of these RISC-bashing trade-rag articles--and NO OTHER
reports of substantive conclusions. (Their other literature listed system
configurations they had tested and tried to explain the tests.) I saw no
attempt to present a balanced picture, nor to compare a representative set
of comparable machines.
For example, the EE Times RISC-bash article (which I happen to have at
hand) shows a Sun 3/260 beating the pants off a Model 25 RT PC and
slightly winning over a MIPS M-500 on the test EE Times chose to show
in detail. (The Sun 3 was the only CISC design represented; there were
other RISCs. I'm singling out MIPS and the RT as examples.) EE Times
chose to report in detail on only one test out of 18 in the benchmark!
The results as reported show some obvious problems:
- Why were they using the fastest Sun 3 (25 MHz) for comparison
against the slowest MIPS machine?
- If a 25 MHz CISC is only 12% faster than an 8 MHz RISC, how does
that make the CISC faster?
- Although the older RT lost badly, it's not a full RISC design...
and one ought perhaps to take into account that the Sun system
costs more than 3 times as much.
- Where are the other CISCs? It seems very much like only the best
CISC they could find was used as the basis for comparison--and
certainly the best RISCs were NOT used.
At the point that EE Times published their report, it would have been easy
(for NNA in particular) to say that it was simply a case of EE Times doing
some very selective reporting. But at USENIX, it was clear that NNA was
quite proud to show off the EE Times article...and it seems clear that NNA
has an ax to grind wrt RISCs even though it's not clear why. They're
making some pretty strong statements:
"I'm beginning to believe RISC doesn't belong everywhere, or
possibly even anywhere..."
"...we still haven't seen any areas of study that say RISC has been
implemented and shown a marked improvement..."
...but they're not backing them up. Both of these are attributed to Neal
Nelson. Incidentally, the second statement may well be true--but it's sur-
prising that someone working on benchmarks could ignore a decade or so of
work!
(It's interesting that at ASPLOS last fall, the two camps seemed to be
holding viewpoints of "RISC has won" vs "the battle isn't over"--quite a
different story from what Nelson tells.)
Most folks familiar with the RISC-CISC debate have their biases, and in a
moment of passion they may get a little carried away in supporting their
viewpoint. But what I see in this Neal Nelson stuff goes beyond a
momentary outburst--it's an unsubstantiated, unprofessional reaction to
the substantial real work (and very real, very fast machines) that have
come out of RISC development. I think it added some heat and absolutely no
light...we need thought more than we need more emotion.
--
Dick Dunn UUCP: {ncar,nbires}!ico!rcd (303)449-2870
...Are you making this up as you go along?
Path: utzoo!utgpu!water!watmath!clyde!att!rutgers!rochester!cornell!
batcomputer!itsgw!steinmetz!uunet!cbmvax!snark!eric
From: e...@snark.UUCP (Eric S. Raymond)
Newsgroups: comp.arch,comp.lang.c,comp.unix.wizards
Subject: RISC as the making of UNIX (Re: RISC bashing at USENIX)
Summary: A person with no *architectural* anti-[CR]ISC bias sez...
Message-ID: <dYTGG#465xb3=eric@snark.UUCP>
Date: 9 Jul 88 23:04:55 GMT
References: <6888@ico.isc.com>
Organization: Cosmic Karmic Recycling Central
Lines: 89
Back-References: <5...@june.cs.washington.edu>
In article <6...@ico.isc.com>, r...@ico.ISC.COM (Dick Dunn) writes:
>Most folks familiar with the RISC-CISC debate have their biases, and in a
>moment of passion they may get a little carried away in supporting their
>viewpoint. [...] think it added some heat and absolutely no
>light...we need thought more than we need more emotion.
Thank you for an excellent article. I agree, and would like to offer a
viewpoint on the debate that at the very least implies a *different* set
of religious wars ;-).
I am a software person, an ex-mathematician of somewhat theoretical bent;
I like talking and thinking about computer architectures, and I don't get
lost when the hard-core chippies start mumbling about pipelining and
register files and the glories of unmicrocoded machines etc. etc; but the
*architectural* RISC-vs-CISC and RISC-vs-RISC debates (which used to appear
fairly simple) have long since moved to levels of theological abstruseness
at which, frankly, mine eyes tend to glaze over.
Yes, I like my machines to be fast. But benchmark wars leave me utterly cold.
I mean, we're generally talking about differences that amount to small blips
on a market-driven trend curve so steep that I *expect* my available bang-
per-buck to effectively double every nine months or so. Fight your fights,
guys, because that's part of the dialectic; but they aren't *my* problem. I'll
win any way the chips fall, because free markets select for winning solutions.
Does this mean I'm neutral in the RISC/CISC wars? Not at all! The point of this
posting is to observe that there are technical, political and economic agendas
that can be involved in the dispute that have *nothing* to do with benchmark
wars and everything to do with the more abstract interests of software people.
I'm pro-RISC, rather strongly so (there; the cat's out of the bag now!). Here
are a few reasons that, as I emphasized before, have *nothing* to do with
the theology of how many mips one can induce to dance on a pinhead-sized piece
of silicon..
1. I dislike assembler. I like C. Nobody tries to make you write
assembler code for RISC machines. All RISC machines have spiffy
fast-as-blazes C compilers.
2. I dislike proprietary operating systems, vendor-controlled
standards, and we-want-to-lock-you-in software technology in
general. I like UNIX. What operating system does every RISC
vendor bring up practically before the etching fluid is dry
on this quarter's new chip? You guessed it...
3. I dislike companies that have a we-are-the-high-priests-of-hardware
-so-you'll-like-what-we-give-you attitude. I like commodity markets
in which iron-and-silicon hawkers know that they exist to provide
fast toys for software types like me to play with (just as software
types know they depend on their ability to gratify the whims of
*users*).
Now, none of these is necessarily more than a casual reason to tilt in the RISC
direction -- unless you think that the RISC revolution is *causal* in promoting
C, UNIX and vendor humility. And I do.
Consider: one major impact of RISC has been to shorten product cycles. This has
forced vendors to forgo the luxury of writing lock-me-in software technology --
how long has it been now, since one could afford to delay introduction to write
a VMS or an OS/370?
RISC has also reversed the trend towards product-line architectural uniformity
in the industry (cf SPARC, the Motorola 88000, and the rumored DEC successor to
the VAX 8650). This means that the cheap-and-sleazy way out -- porting last
decade's proprietary OS to the new silicon -- is just as big a mess, and robs
the it's-optimized-for-the-machine line of arguments of plausibility.
These trends have created a vicious (for nasty vendors) circle in which UNIX
support is the only way to get to market fast enough -- and the increasing
dominance of UNIX creates a commodity market, which in turn generates more
pressure to get the newest-fastest-spiffiest hardware out the door, which takes
us right back where we got on.
RISC is causal in all this is that, by championing high-level language
compilation in standard environments as a *dominant* development mode, it
creates an *expectation* that hardware can be treated as a commodity; pay
so many bucks, buy so many Dhrystones, thank you very much.
Forget the benchmark wars and the microcode-vs-random-logic debates and the
is-it-RISC-or-is-it-CISC lucubrations. *THIS* is why comp.lang.c and
comp.unix.wizards people do and should care about RISC vs. CISC. Because RISC
means high-level languages and software standards -- specifically C and UNIX
-- have gone from being dreams of the future to being the economic necessity
of the present. And that is good news for all of us.
--
Eric S. Raymond (the mad mastermind of TMN-Netnews)
UUCP: ..!{uunet,att,rutgers!vu-vlsi}!snark!eric Smail: e...@snark.UUCP
Post: 22 South Warren Avenue, Malvern, PA 19355 Phone: (215)-296-5718
Path: utzoo!utgpu!water!watmath!clyde!att!osu-cis!tut.cis.ohio-state.edu!
mandrill!gatech!ncar!ames!ucsd!ucsdhub!esosun!seismo!uunet!steinmetz!davidsen
From: david...@steinmetz.ge.com (William E. Davidsen Jr)
Newsgroups: comp.arch
Subject: Re: RISC bashing at USENIX
Message-ID: <11496@steinmetz.ge.com>
Date: 11 Jul 88 15:19:07 GMT
References: <6888@ico.ISC.COM>
Reply-To: david...@crdos1.UUCP (bill davidsen)
Organization: General Electric CRD, Schenectady, NY
Lines: 29
In article <6...@ico.ISC.COM> r...@ico.ISC.COM (Dick Dunn) writes:
|
| "Neal Nelson and Associates" had a booth at the USENIX vendor's exhibition
| whose sole purpose seemed to be RISC-bashing. Although they purportedly
| have developed a set of tests, which they call their "Business Benchmark
| (R)" for helping people make realistic comparisons of machines under
| realistic loads, in fact even basic descriptions of the tests cast serious
| doubt on their real usefulness. Some of the tests seem overly simplistic;
| others contain obvious biases toward or against certain types of hardware
| and/or I/O system software.
Neal Nelson presents the results of his benchmarks and the description
of each test. You are free to interpret them any way you want. That's
not a flame, just a reminder that if you choose to interpret his results
as RISC bashing, then you seem to have decided that RISC is better, and
that a benchmark which doesn't show that is either biased or useless.
We have looked at the NN benchmarks for a number of machines (I
obviously can't say which ones), and my personal reaction is that they
are reasonable and valid for business applications. If your application
is something else, why not get a benchmark suite which tests that,
rather than blasting NN?
I don't consider ANY benchmark to be the whole story on a machine,
even my own.
--
bill davidsen (w...@ge-crd.arpa)
{uunet | philabs | seismo}!steinmetz!crdos1!davidsen
"Stupidity, like virtue, is its own reward" -me
Path: utzoo!utgpu!water!watmath!clyde!bellcore!rutgers!gatech!ncar!ico!rcd
From: r...@ico.ISC.COM (Dick Dunn)
Newsgroups: comp.arch
Subject: Re: RISC bashing at USENIX
Summary: Look at how it's presented
Message-ID: <6965@ico.ISC.COM>
Date: 12 Jul 88 05:23:47 GMT
References: <6888@ico.ISC.COM> <11496@steinmetz.ge.com>
Organization: Interactive Systems Corp, Boulder, CO
Lines: 61
In response to my griping about Neal Nelson at USENIX, davidsen@
steinmetz.ge.com (William E. Davidsen Jr) writes:
> Neal Nelson presents the results of his benchmarks and the description
> of each test. You are free to interpret them any way you want. That's
> not a flame, just a reminder that if you choose to interpret his results
> as RISC bashing, then you seem to have decided that RISC is better, and
> that a benchmark which doesn't show that is either biased or useless.
Good grief! I did NOT interpret NN's results as RISC bashing...I
interpreted the *presentation* of the results as RISC bashing. The
presentation at USENIX was flamboyantly anti-RISC--meaning that there are
statements about RISCs by NN which are vehemently anti-RISC and not backed
up by fact. Yes, I am free to interpret the results any way I want--but NN
wants me to interpret them in a particular way, as strongly anti-RISC. I
don't see that they support that viewpoint at all, which is what I'm com-
plaining about.
There *might* be some useful results and good work behind it all, but I
sure-as-hell can't find it even after trying to peel back the layers of
hype and bad journalism, so I tend to doubt that there's much there.
No, I haven't decided that RISC is better. The biases in the benchmarks
(more to the point, in the reporting thereof) are evident regardless of
your own biases, if you just look at what's been said. I pointed out some
of the more obvious problems...so if you think I'm off base, why don't you
take on the *substance* of what I said? (I.e., if you're trying to say *I'm*
biased, show us how. For example, do you dispute that they compared the
fastest Sun 3 against the slowest MIPS box?)
Most of my computing is done on CISCs, and they serve well. But I had a
chance recently to run a couple of problems on a low-end RISC; they ran so
fast that I put in some debugging code to be sure something hadn't gotten
short-circuited! (It hadn't.) I'm not taking up the sword to defend RISC,
but I know the RISC guys aren't smoking rope--they're for real.
> We have looked at the NN benchmarks for a number of machines (I
> obviously can't say which ones), and my personal reaction is that they
> are reasonable and valid for business applications...
OK, so which benchmarks are the good ones? Note that the one that EE Times
gave such prominent coverage was one of the simplest--a loop with just 4
calculations (+-*/) on 16-bit integers, running 1 to 15 copies at a time.
That has about 0.5 * dsq to do with any real business program. And, as I
said in the original article, I could have attributed it to EE Times'
sloppiness (the rest of the article was an expository/stylistic/technical
mess) but for the fact that NN was showing it off. NN has 17 other
benchmarks, and they could have put together a complete presentation of the
benchmarks on comparable machines. They didn't. Why not?
> ...why not get a benchmark suite which tests [what concerns you],
> rather than blasting NN?
Done! I have tests of my own which I run when *I* want to get an idea of
how fast a processor is. The reason I'm blasting NN is that I see them
misleading people--and using a lot of PR to mislead a lot of people. It's
that aspect that bothers me--not that it's RISCs per se that they're
bashing, but that they're bashing, instead of testing and reporting
carefully.
--
Dick Dunn UUCP: {ncar,nbires}!ico!rcd (303)449-2870
...Are you making this up as you go along?
Path: utzoo!attcan!uunet!steinmetz!davidsen
From: david...@steinmetz.ge.com (William E. Davidsen Jr)
Newsgroups: comp.arch
Subject: Re: RISC bashing at USENIX
Message-ID: <11504@steinmetz.ge.com>
Date: 12 Jul 88 18:17:30 GMT
References: <6888@ico.ISC.COM> <11496@steinmetz.ge.com> <6965@ico.ISC.COM>
Reply-To: david...@crdos1.UUCP (bill davidsen)
Organization: General Electric CRD, Schenectady, NY
Lines: 30
In article <6...@ico.ISC.COM> r...@ico.ISC.COM (Dick Dunn) writes:
| > We have looked at the NN benchmarks for a number of machines (I
| > obviously can't say which ones), and my personal reaction is that they
| > are reasonable and valid for business applications...
|
| OK, so which benchmarks are the good ones? Note that the one that EE Times
| gave such prominent coverage was one of the simplest--a loop with just 4
| calculations (+-*/) on 16-bit integers, running 1 to 15 copies at a time.
The decision is yours... NN gives the result of the test and what it
measures. I don't disagree that considering (any) one benchmark as an
indicator is probably a waste, but with a selection of results you can
compare two (or more) machines in those areas which apply to your
situation.
I have a UNIX benchmark suite which I have run on a number of machines
for my personal edification. It measures some raw performance numbers
such as the speed of arithmetic for all data types, trancendental
functions, test and branch for int and float, disk access and transfer
times for large and small files, speed of bit fiddling such as Grey to
binary, etc. Then I measure speed of compile, performance under
multitasking load, speed of pipes and system calls, and a few other
things. The *one* thing I measure which consistently represents the
overall performance of the machine is the real time to run the entire
benchmark suite.
--
bill davidsen (w...@ge-crd.arpa)
{uunet | philabs | seismo}!steinmetz!crdos1!davidsen
"Stupidity, like virtue, is its own reward" -me