Tech Insider					     Technology and Trends


			      USENET Archives

Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!utgpu!water!watmath!clyde!rutgers!sri-spam!ames!lll-tis!lll-lcc!
pyramid!prls!mips!mash
From: m...@mips.UUCP
Newsgroups: comp.arch
Subject: MIPS Performance Brief, zillions of numbers, very long
Message-ID: <861@winchester.UUCP>
Date: Thu, 29-Oct-87 16:09:02 EST
Article-I.D.: winchest.861
Posted: Thu Oct 29 16:09:02 1987
Date-Received: Mon, 2-Nov-87 06:32:17 EST
Lines: 1232
Keywords: benchmarks

MIPS Performance Brief, PART 1 : CPU Benchmarks, Issue 3.0, October 1987.
This is a condensed version, with MAC Charts, some explanatory details,
and most advertisements deleted.  For a full version on paper,
with MAC charts an tables that aren't squished toether, send mail,
NOT TO ME, but to:
....{ucbvax, decvax, ihnp4}!decwrl!mips!eleanor [Eleanor Bishop]
(Please be patient: it will take a little while to get them sent.)

As usual, I tried pretty hard to get the numbers right, but let us know if we
goofed anywhere.  We'll update the next time.

The posting is 1200+ lines long, with a thousand or so benchmark
numbers & performance ratios for dozens of machines and benchmarks. 
If you're not a glutton for data, "n" now or after the main summary!
------------

1.  Introduction

New Features of This Issue
More benchmarks are normalized to the VAX-11/780 under VAX/VMS,rather than UNIX.
Livermore Loops and Digital Review Magazine benchmarks have been added, and
the Spice section uses new public domain inputs.

The Brief has been divided into two parts: user and system.  The system bench-
mark part is being greatly expanded (beyond Byte), and has been moved to a new
document "MIPS Performance Brief - Part 2".  User-level performance is mostly
driven by compiler and library tuning, whereas system performance also depends
on operating system and hardware configuration issues.  The two Briefs syn-
chronize with different release cycles: Part 2 will appear 1-2 months after
Part 1.

Benchmarking - Caveats and Comments
While no one benchmark can fully characterize overall system performance, the
results of a variety of benchmarks can give some insight into expected real
performance.  A more important benchmarking methodology is a side-by-side com-
parison of two systems running the same real application.

We don't believe in characterizing a processor with just a single number, but
we follow (what seems to be) standard industry practice of using a mips-rating
that essentially describes overall integer performance.  Thus, we label a 5-
mips machine to be one that is about 5X (i.e., anywhere from 4X to 6X!) faster
than a VAX 11/780 (UNIX 4.3BSD, unless we can get Ultrix or VAX/VMS numbers) on
integer performance, since this seems to be how most people intuitively compute
mips-ratings.  Even within the same computer family, performance ratios between
processors vary widely.  For example, [McInnis 87] characterizes a ``6 mips''
VAX 8700 as anywhere from 3X to 7X faster than the 11/780. Floating point speed
often varies more than, and scales up slower than integer speed versus the
11/780.

This paper analyzes one important aspect of overall computer system performance
- user-level CPU performance.

MIPS Computer Systems does not warrant or represent that the performance data
stated in this document will be achieved by any particular application.  (We
have to say that, sorry.)

2.  Benchmark Summary

2.1.  Choice of Benchmarks

This brief offers both public-domain and MIPS-created benchmarks.  We prefer
public domain ones, but some of the most popular ones are inadequate for accu-
rately characterizing performance.  In this section, we give an overview of the
importance we attach to the various benchmarks, whose results are summarized on
the next page.

Dhrystone [DHRY 1.1] and Stanford [STAN INT] are two popular small integer
benchmarks.  Compared with the fastest VAX 11/780 systems, the M/1000 is 13-14X
faster than the VAX on these tests, and yet, we rate the M/1000 as a 10-vax-
mips machine.

While we present Dhrystone and Stanford, we feel that the performance of large
UNIX utilities, such as grep, yacc, diff, and nroff is a better (but not per-
fect!) guide to the performance customers will receive.  These four, which make
up our [MIPS UNIX] benchmark, demonstrate that performance ratios are not sin-
gle numbers, but range here from 8.6X to 13.7X faster than the VAX.

Even these UNIX utilities tend to overstate performance relative to large
applications, such as CAD applications.  Our own vax-mips ratings are based on
a proprietary set of larger and more stressful real programs, such as our com-
piler, assembler, debugger, and various CAD programs.

For floating point, the public domain benchmarks are much better.  We're still
careful not to use a single benchmark to characterize all floating point appli-
cations.

The Livermore Fortran kernels [LLNL DP] give insight into both vector and non-
vector performance for scientific applications.  Linpack [LNPK DP and LNPK SP]
tests vector performance on a single scientific application, and stresses cache
performance.  Spice [SPCE 2G6] and Doduc [DDUC] test a different part of the
floating point application spectrum.  The codes are large and thus test both
instruction fetch bandwidth and scalar floating point.  Digital Review Maga-
zines benchmark [DIG REV] is a compendium of FORTRAN tests that measure a wide
variety of behavior, and seem to correlate well with some classes of real pro-
grams.

2.2.  Benchmark Summary Data     This section summarizes the most important
benchmark results described in more detail throughout this document. The
numbers show performance relative to the VAX 11/780, i.e., larger numbers are
better/faster.

o    A few numbers have been estimated by interpolations from closely-related
     benchmarks and/or closely-related machines.  The methods are given in
     great detail in the individual sections.

o    Several of the columns represent summaries of multiple benchmarks.  For
     example, the MIPS UNIX column represents 4 benchmarks, the SPICE 2G6
     column 3, and LLNL DP represents 24.

o    In the Integer section, MIPS UNIX is the most indicative of real perfor-
     mance.

o    For Floating Point, we especially like LLNL DP (Livermore FORTRAN ker-
     nels), but all of these are useful, non-toy benchmarks.

o    In the following table, "Pub mips" gives the manufacturer-published mips-
     ratings.  As in all tables in this document, the machines are listed in
     increasing order of performance according to the benchmarks, in this case,
     by Integer performance.

o    The summary includes only those machines for which we could get measured
     results on almost all the benchmarks and good estimates on the the results
     for the few missing data items.

                         Summary of Benchmark Results
                     (VAX 11/780 = 1.0, Bigger is Faster)
   Integer (C)            Floating Point (FORTRAN)
----------------    -------------------------------------
MIPS   DHRY  STAN   LLNL  LNPK   LNPK   SPCE  DIG    DDUC   Publ
UNIX   1.1   INT     DP    DP     SP    2G6   REV           mips  System
 1      1     1     1      1      1     1      1      1       1   VAX 11/780#
 2.1    1.9   1.8   1.9    2.9    2.5   1.6   *2     *1.3     2   Sun3/160 FPA
*4      4.1   4.7   2.8    3.3    3.4   2.4   *3      1.7     4   Sun3/260 FPA
 5.5    7.4   7.2   2.5    4.3    3.7   3.4    4.9    3.8     5   MIPS M/500

*6      5.9   6.5   5.9    6.9    5.6   5.3    6.2    5.2     6   VAX 8700
 8.0   10.8   7.3   4.5    7.9    6.4   4.1    4.4    3.5    10   Sun4/260
 9.2   11.3  11.8   8.1    7.1    7.6   6.6    7.6    7.3     8   MIPS M/800
11.3   13.5  14.1   9.7    8.6    9.2   8.0    9.3    8.8    10   MIPS M/1000

# VAX 11/780 runs 4.3BSD for MIPS UNIX, Ultrix 2.0 (vcc) for Stanford, VAX/VMS
  for all others.  Use of 4.3BSD (no global optimizer) probably inflates the
  MIPS UNIX column by about 10%.

* Although it is nontrivial to gather full set of numbers, it is important to
  avoid holes in benchmark tables, as it is too easy to be misleading.  Thus,
  we had to make reasoned guesses at these numbers.  The MIPS UNIX values for
  VAX 8700 and Sun-3/260 were taken from the Published mips-ratings, which are
  consistent (+/- 10%) with experience with these machines.  DIG REV and DDUC
  were guessed by noting that most machines do somewhat better on DIG REV than
  on SPCE, and than a Sun-3/260 is usually 1.5X faster than a Sun-3/160 on
  floating-point benchmarks.

Benchmark Descriptions:

MIPS UNIX
  MIPS UNIX benchmarks: grep, diff, yacc, nroff, same 4.2BSD C source compiled
  and run on all machines. The summary number is the geometric mean of the 4
  relative performance numbers.

DHRY 1.1
  Dhrystone 1.1, any optimization except inlining.

STAN INT
  Stanford Integer.

LLNL DP
  Lawrence Livermore Fortran Kernels, 64-bit.  The summary number is the given
  as the relative performance based on the geometric mean, i.e., the "middle"
  of the 3 means.

LNPK DP
  Linpack Double Precision, FORTRAN.

LNPK DP
  Linpack Single Precision, FORTRAN.

SPCE 2G6
  Spice 2G6, 3 public-domain circuits, for which the geometric mean is shown.

DIG REV
  Digital Review magazine, combination of 33 benchmarks.

DDOC
  Doduc Monte Carlo benchmark.

3.  Methodology

Tested Configurations

When we report measured results, rather than numbers published elsewhere, the
configurations were as shown below.  These system configurations do not neces-
sarily reflect optimal configurations, but rather the in-house systems to which
we had repeatable access.  When we've had the faster results available, we've
quoted them in place of our own system's numbers.

DEC VAX-11/780
Main Memory:        8 Mbytes
Floating Point:     Configured with FPA board.
Operating System:   4.3 BSD UNIX.

DEC VAX 8600
Main Memory:        20 Mbytes
Floating Point:     Configured without FPA board.
Operating System:   Ultrix V1.2. (4.2BSD with many 4.3BSD tunings).

Sun-3/160M
CPU:                (16.67 MHz MC68020)
Main Memory:        8 Mbytes
Floating Point:     12.5 MHz MC68881 coprocessor (compiled -f68881).
Operating System:   SunOS 3.2 (4.2BSD)

MIPS M/500
CPU:                8MHz R2000, in R2300 CPU board, 16K I-cache, 8K D-cache
Floating Point:     R2010 FPA chip (8MHz)
Main Memory:        8 Mbytes (2 R2350 memory boards)
Operating System:   UMIPS-BSD 2.1 (4.3BSD UNIX with NFS)

MIPS M/800
CPU:                12.5 MHz R2000, in R2600 CPU board, 64K I-cache, 64K D-cache
Floating Point:     R2010 FPA chip (12.5MHz)
Main Memory:        8 Mbytes (2 R2350 memory boards)
Operating System:   UMIPS-BSD 2.1

MIPS M/1000
CPU:                15 MHz R2000, in R2600 CPU board, 64K I-cache, 64K D-cache
Floating Point:     R2010 FPA chip (15 MHz)
Main Memory:        16 Mbytes (4 R2350 memory boards)
Operating System:   UMIPS-BSD 2.1
Test Conditions

All programs were compiled with -O (optimize), unless otherwise noted.

C is used for all benchmarks except Whetstone, LINPACK, Doduc, Spice 2g.6,
Hspice, and the Livermore Fortran Kernels, which use FORTRAN.  When possible,
we've obtained numbers for VAX/VMS, and use them in place of UNIX numbers.  The
MIPS compilers are version 1.21.

User time was measured for all benchmarks using the /bin/time command.

Systems were tested in normal multi-user development environment, with load
factor <0.2 (as measured by uptime command).  Note that this occasionally makes
them run longer, due to slight interference from background daemons and clock
handling, even on an otherwise empty system.  Benchmarks were run at least 3
times and averaged.  The intent is to show numbers that can be reproduced on
live systems.

Times (or rates, such as for Dhrystones, Whetstones, and LINPACK KFlops) are
shown for the VAX 11/780.  Other machines' times or rates are shown, and their
relative performance ("Rel." column) normalized to the 11/780 treated as 1.0.
VAX/VMS is used whenever possible as the base.

Compilers and Operating Systems

Unless otherwise specified, The M-series benchmark numbers use Release 1.21 of
the MIPS compilers and UMIPS-BSD 2.1.

Optimization Levels

Unless otherwise specified, all benchmarks were compiled -O, i.e., with optimi-
zation.  UMIPS compilers call this level -O2, and it includes global intra-
procedural optimization.  In a few cases, we show numbers for -O3 and -O4
optimization levels, which do inter-procedural register allocation and pro-
cedure merging.  -O3 is now generally available.

Now, let's look at the benchmarks.  Each section title includes the (CODE NAME)
that relates it back to the earlier Summary, if it is included there.

4.  Integer Benchmarks

4.1.  MIPS UNIX Benchmarks (MIPS UNIX)

The MIPS UNIX Benchmarks described below are fairly typical of nontrivial UNIX
programs.  This benchmark suite provides the opportunity to execute the same
code across several different machines, in contrast to the compilers and link-
ers for each machine, which have substantially different code.  User time is
shown; kernel time is typically 10-15% of the user time (on the 780), so these
are good indications of integer/character compute-intensive programs.  The
first 3 benchmarks were running too fast to be meaningful on our faster
machines, so we modified the input files to get larger times.  The VAX 8600 ran
consistently around 3.8X faster than the 11/780 on these tests, but we sold it,
so it's started to drop out as we've changed benchmarks.  These benchmarks con-
tain UNIX source code, and are thus not generally distributable.

For better statistical properties, we now report the Geometric Mean of the
Relative performance numbers, because it does not ignore the performance con-
tributions of the shorter benchmarks.  (In this case, the grep ratios drag the
Geometric Mean down.) Expect real performance to lie between the Geometric Mean
and the Total Relative number.

Note: the Geometric Mean of N numbers is the Nth root of the product of those
numbers.  It is necessarily used in place of the arithmetic mean when computing
the mean of performance ratios, or of benchmarks whose runtimes are quite dif-
ferent.  See [Fleming 86] for a detailed discussion.

                         MIPS UNIX Benchmarks Results
   grep       diff        yacc        nroff       Total    Geom     System
Secs  Rel.  Secs  Rel.  Secs  Rel.  Secs Rel.  Secs+  Rel. Mean

11.2  1.0  246.4  1.0  101.1  1.0   18.8 1.0   377.5  1.0  1.0   11/780 4.3BSD
 5.6  2.0  105.3  2.3   48.1  2.1    9.0 2.1   168.0  2.2  2.1   Sun-3/160M
   -   -       -   -       -   -     5.0 3.8       -  3.8  3.8   DEC VAX 8600
 2.4  4.7   35.8  6.9   19.5  5.2    3.3 5.7    61.0  6.2  5.5   MIPS M/500

   -   7       -  8.5      -   9       - 7.5       -   -   8.0   Sun-4 *
 1.6  7.0   21.6  11.4  11.2  9.0    1.9 9.9    36.3  10.4 9.2   MIPS M/800
 1.3  8.6   18.0  13.7   9.3  10.9   1.5 12.5   30.1  12.5 11.3  MIPS M/1000

+ Simple summation of the time for all benchmarks.  "Total Rel." is ratio of
the totals.

* These numbers derived as shown on next page.

Note: in order to assure "apples-to-apples" comparisons, we moved the same
copies of the (4.2BSD) sources for these to the various machines, compiled them
there, and ran them, to avoid surprises from different binary versions of com-
mands resident on these machines.

Note that the granularity here is at the edge of UNIX timing, i.e., tenths of
seconds make differences, especially on the faster machines.  The performance
ratios seen here seem typical of large UNIX commands on MIPS systems.

Estimation of Sun-4 Numbers

The 4 ratios cited in the table above were pieced together from somewhat frag-
mentary information.  Earlier Briefs used shorter benchmarks for the grep,
diff, and yacc tests, for which we were able to get numbers from a Sun-4, with
SunOS 3.2L, at 16.7MHz.  By comparing with M/500 and M/800 numbers on the same
tests, we can interpolate and at least estimate bounds on performance.

On grep, both Sun-4 and M/800 used .6 seconds user time, so we assumed the same
relative performance (7.0).

On yacc, both Sun-4 and M/800 used .4 seconds, so we assumed the same relative
performance (9.0).

On diff, 3 runs on the Sun-4 yielded .3, .4. and .4 seconds.  A MIPS M/500 is
consistently .4, an M/800 .2, and an M/1000 usually .2, but occasionally .1.
This test was clearly too small, and it is difficult to make strong assertions.
However, it does seem that the Sun-4 is faster than the M/500  (6.9X VAX) but
noticeably closer to it than to an M/800 (11.4X).  We thus estimate around
8.5X, a little lower than halfway between the two MIPS systems.

On nroff, a setup problem of ours aborted the Sun-4 run.  At a seminar at Stan-
ford this summer, the following number was given by Sun: 18.5 seconds for:
        troff -man csh.1
This is not sufficient information to allow exact duplication, but we tried
running it two different ways:
        troff -a -man csh.1 >/dev/null          troff -man ..... csh.1
The second case used numerous arguments to actually run it for a typesetter,
whereas the first emits a representation to the standard output.  The M/500
required 23.4 and 27.9 seconds user time, respectively, while the M/800 gave
user times of 11.2 and 14.1 seconds.  Assuming that the troff results are simi-
lar to those of nroff, and using the worst of the M/800 times, we get a VAX-
relative estimate of:
(9.9X for M/800) X 14.1 (M/800 secs) / 18.5 (Sun-4 secs)
which yields 7.5X for the Sun-4.

All of this is obviously a gross approximation with numerous items of missing
information.  Timing granularity is inadequate.  Results are being generalized
from small samples.  The source code may well differ for these programs.  Our
csh.1 manual pages may not be identical.  Sun compilers will improve, etc, etc.

We apologize for the low quality of this data, as Sun-4 access is not something
we have in good supply.  We'll run the newest forms of the benchmarks as soon
as possible.  However, the end results does seem quite consistent with with
other benchmarks that various people in the industry have tried.

Finally, note that this benchmark set is running versus 4.3BSD, not versus
Ultrix 2.0 with vcc.  Hence, the relative performance numbers are inflated
somewhat relative to VAX/VMS or VAX-Ultrix numbers.  From other experience,
we'd guess that subtracting 10% from most of the computed mips-ratings would
give a good estimate of the Ultrix 2.0 (vcc)-relative mips-ratings.

4.2.  Dhrystone (DHRY 1.1)

Dhrystone is a synthetic programming benchmark that measures processor and com-
piler efficiency in executing a ``typical'' benchmark. The Dhrystone results
shown below are measured in Dhrystones / second, using the 1.1 version of the
benchmark.

We include Dhrystone because it is popular.  MIPS systems do extremely well on
it.  However, comparisons of systems based on Dhrystone and especially, only on
Dhrystone, are unreliable and should be avoided.  More details are given at the
end of this section.  According to [Richardson 87], 1.1 cleans up a bug, and is
the correct version to use, even though results for a given machine are typi-
cally about 15% less for 1.1 than with 1.0.

Advice for running Dhrystone has changed over time with regard to optimization.
It used to ask that people turn off optimizers that were more than peephole
optimizers, because the benchmark contained a modest amount of "dead" code that
optimizers were eliminating.  However, it turned out that many people were sub-
mitting optimized results, often unlabeled, confusing everyone.  Currently, any
numbers can be submitted, as long as they're appropriately labeled, except that
procedure inlining (done by only a few very advanced compilers) must be
avoided.

We continue to include a range of numbers to show the difference optimization
technology makes on this particular benchmark, and to provide a range for com-
parison when others' cited Dhrystone figures are not clearly defined by optimi-
zation levels.  For example, -O3 does interprocedural register allocation, and
-O4 does procedure inlining, and we know -O4 is beyond the spirit of the bench-
mark.  Hence, we now cite the -O3 numbers.  We're not sure what the Sun-4's -O3
level does, but we do not believe that it does inlining either.

In the table below, it is interesting to compare the performance of the two
Ultrix compilers.  Also, examination of the MIPS and Sun-4 numbers shows the
performance gained by the high-powered optimizers available on these machines.

The numbers are ordered in what we think is the overall integer performance of
the processors.

              Dhrystone Benchmark Results - Optimization Effects
    No Opt             -O          -O3     -O4
NoReg    Regs    NoReg    Regs     Regs    Regs
Dhry's  Dhry's   Dhry's  Dhry's   Dhry's  Dhry's
 /Sec    /Sec     /Sec    /Sec     /Sec    /Sec    System
 1,442   1,474    1,559   1,571                    DEC VAX 11/780, 4.3BSD
 2,800   3,025    3,030   3,325                    Sun-3/160M
 4,896   5,130    5,154   5,235                    DEC VAX 8600, Ultrix 1.2
 8,800  10,200   12,300  12,300   13,000  14,200   MIPS M/500
 8,000   8,000    8,700   8,700                    DEC VAX 8550, Ultrix 2.0 cc
 9,600   9,600    9,600   9,700                    DEC VAX 8550, Ultrix 2.0 vcc
10,550  12,750   17,700  17,700   19,000           Sun-4, SunOS 3.2L
12,800  15,300   18,500  18,500   19,800  21,300   MIPS M/800
15,100  18,300   22,000  22,000   23,700  25,000   MIPS M/1000

Some other published numbers of interest include the following, all of which
are taken from [Richardson 87], unless otherwise noted.  Items marked * are
those that we know (or have good reason to believe) use optimizing compilers.
These are the "register" versions of the numbers, i.e., the highest ones
reported by people.
                          Dhrystone Benchmark Results
Dhry's
 /Sec        Rel.       System
  1571       0.9        VAX 11/780, 4.3BSD [in-house]
  1757       1.0        VAX 11/780, VAX/VMS 4.2 [Intergraph 86]*
  3325       1.9        Sun3/160, SunOS 3.2 [in-house]
  3856       2.2        Pyramid 98X, OSx 3.1, CLE 3.2.0
  4433       2.5        MASSCOMP MC-5700, 16.7MHz 68020, RTU 3.1*
  4716       2.7        Celerity 1230, 4.2BSD, v3.2

  6240       3.6        Ridge 3200, ROS 3.4
  6374       3.6        Sun3/260, 25MHz 68020, SunOS 3.2
  6423       3.7        VAX 8600, 4.3BSD
  6440       3.7        IBM 4381-2, UTS V, cc 1.11
  6896       3.9        Intergraph InterPro 32C, SYSV R3 3.0.0, Greenhills, -O*
  7109       4.0        Apollo DN4000 -O
  7142       4.1        Sun-3/200 [Sun 87] *
  7249       4.2        Convex C-1 XP 6.0, vc 1.1
  7409       4.2        VAX 8600, VAX/VMS in [Intergraph 86]*

  7655       4.4        Alliant FX/8 [Multiflow]
  8300       4.7        DG MV20000-I and MV15000-20 [Stahlman 87]
  8309       4.7        InterPro-32C,30MHz Clipper,Green Hills[Intergraph 86]*
  9436       5.4        Convergent Server PC, 20MHz 80386, GreenHills*
  9920       5.6        HP 9000/840S [HP 87]
 10416       5.9        VAX 8550, VAX/VMS 4.5, cc 2.2*
 10787       6.1        VAX 8650, VAX/VMS, [Intergraph 86]*

 11215       6.4        HP 9000/840, HP-UX, full optimization*
 12639       7.2        HP 9000/825S [HP 87]*
 13000       7.4        MIPS M/500, 8MHz R2000, -O3*
 13157       7.5        HP 825SRX [Sun 87]*
 14195       8.1        Multiflow Trace 7/200 [Multiflow]
 14820       8.4        CRAY 1S
 15007       8.5        IBM 3081, UTS SVR2.5, cc 1.5
 15576       8.9        HP 9000/850S [HP 87]
 18530       10.5       CRAY X-MP
 19000       10.8       Sun-4/200, -O3* [Sun 87]
 19800       11.3       MIPS M/800, 12.5MHz R2000, -O3*

 23700       13.5       MIPS M/1000, 15MHz R2000, -O3*
 28846       16.4       Amdahl 5860, UTS-V, cc1.22
 31250       17.8       IBM 3090/200
 43668       24.9       Amdahl 5890/300E, cc -O

Unusual Dhrystone Attributes

We've calibrated this benchmark against many more realistic ones, and we
believe that its results must be treated with care, because the detailed pro-
gram statistics are unusual in some ways.  It has an unusually low number of
instructions per function call (35-40 on our machines), where most C programs
fall in in the 50-60 range or higher.  Stated another way, Dhrystone does more
function calls than usual, which especially penalizes the DEC VAX, making this
a favored benchmark for inflating one's "VAX-mips" rating.  Any machine with a
lean function call sequence looks a little better on Dhrystone that it does on
others.

The dynamic nesting depth of function calls inside the timed part of Dhrystone
is low (3-4).  This means that most register-window RISC machines would never
even once overflow/underflow their register windows and be required to
save/restore registers.

This is not to say fast function calls or register windows are bad (they're
not!), merely that this benchmark overstates their performance effects.

Dhrystone can spend 30-40% of the time in the strcpy function, copying atypi-
cally long (30-character) strings, which happen to be alignable on word boun-
daries.  More realistic programs don't spend this much time in this sort of
code, and when they do, they handle more shorter strings: 6 characters would be
much more typical.

On our machines, Dhrystone uses 0-offset addressing for 50% of memory data
references (dynamic).  Most real programs use 0-offsets 10-15% of the time.
Of course, Dhrystone is a fairly small benchmark, and thus fits into almost any
reasonable instruction cache.

In conclusion, Dhrystone gives some indication of user-level integer perfor-
mance, but is susceptible to surprises when comparing amongst architectures
that differ strongly.  Unfortunately, the industry seems to lack a good set of
widely-available integer benchmarks that are as representative as are some of
the popular floating point ones.

4.3.  Stanford Small Integer Benchmarks (STAN INT)

The Computer Systems Laboratory at Stanford University, has collected a set of
programs to compare the performance of various systems.  These benchmarks are
popular in some circles as they are small enough to simulate, and are respon-
sive to compiler optimizations.

It is well known that small benchmarks can be misleading.  If you see claims
that machine X is up to N times a VAX on some (unspecified) benchmarks, these
benchmarks are probably the sort they're talking about.

                   Stanford Small Integer Benchmark Results
Perm Tower Queen Intmm Puzzle Quick Bubble Tree Aggr  Rel.
Secs Secs  Secs  Secs  Secs   Secs  Secs   Secs Secs* Perf+ System

2.34 2.30  .94   1.67  11.23  1.12  1.51   2.72  3.08  .84   VAX 11/780 4.3BSD
                                                 2.60   1.0 VAX 11/780@
 .72  1.07   .50   .93   5.53   .58    .97 1.05  1.42   1.8 Sun-3/160M [ours]

 .63   .63   .27   .73   2.96   .31    .44  .69   .86   3.0 VAX 8600 Ultrix1.2
 .28   .35   .17   .42   2.22   .18    .25  .35   .50   5.2 VAX 8550#
 .28   .35   .13   .15    .88   .13    .17  .50   .40   6.5 VAX 8550##
                                                  .65   4.7 Sun-3/200 [Sun 87]

 .18   .24   .15   .23   1.15   .17    .19  .34   .36   7.2 MIPS M/500
                                                  .36   7.3 Sun-4/200 [Sun 87]
 .12   .16   .11   .13    .61   .10    .12  .22   .22  11.8 MIPS M/800
 .10   .13   .10   .11    .51   .08    .10  .17   .18  14.1 MIPS M/1000

*   As weighted by the Stanford Benchmark Suite

+   Ratios of the Aggregate times

@   Estimated VAX 11/780 Ultrix 2.0 vcc -O time.  We get this by 3.08 *
    (.40o.02)/.50 = 2.60, i.e., using the VAX 8550 numbers to estimate the
    effect of optimization.  The ".02" is a guess that optimization helps the
    8550 a little more than it does the 11/780, because the former's cache is
    big enough to hold the whole program and data, whereas the latter's does
    not.  Another way to put it is that the 8550 is not cache-missing very
    much, and so optimization pays off more in removing what's left, whereas
    the 11/780 will cache-miss more, and the nature of these particular tests
    is that the optimizations won't fix cache-misses.  (None of this is very
    scientific, but it's probably within 10%!)

#   Ultrix 2.0 cc -O

##  Ultrix 2.0 vcc -O.  The quick and bubble tests actually had errors; how-
    ever, the times were in line with expectations (these two optimize well),
    so we used them.  All 8550 numbers thanks to Greg Pavlov
    (ames!harvard!hscvax!pavlov, of Amherst, NY).

The Sun numbers are from [Sun 87].  The published Sun-4 number is .356, for
SunOS 3.2L software, i.e., it is slightly faster than the M/500.

5.  Floating Point Benchmarks

5.1.  Livermore Fortran Kernels (LLNL DP)

Lawrence Livermore National Labs' workload is dominated by large scientific
calculations that are largely vectorizable.  The workload is primarily served
by expensive supercomputers.  This benchmark was designed for evaluation of
such machines, although it has been run on a wide variety of hardware, includ-
ing workstations and PCs [McMahon86].

The Livermore Fortran Kernels are 24 pieces of code abstracted from the appli-
cations at Lawrence Livermore Labs.  These kernels are embedded in a large,
carefully engineered benchmark driver.  The driver runs the kernels multiple
times on different data sets, checks for correct results, verifies timing accu-
racy, reports execution rates for all 24 kernels, and summarizes the results
with several statistics.

Unlike many other benchmarks, there is no attempt to distill the benchmark
results down to a single number.  Instead all 24 kernel rates, measured in
mflops (million floating point operations per second) are presented individu-
ally for three different vector lengths (a total of 72 results).  The minimum
and maximum rates define the performance range of the hardware.  Various
statistics of the 24 or 72 rates, such as the harmonic, geometric, and arith-
metic means give insight into general behavior.  Any one of these statistics
might suffice for comparisons of scalar machines, but multiple statistics are
necessary for comparisons involving machines with vector or parallel features.
These machines have unbalanced, bimodal performance, and a single statistic is
insufficient characterization.  McMahon asserts:

    ``When the computer performance range is very large the net Mflops rate
    of many Fortran programs and workloads will be in the sub-range between
    the equi-weighted harmonic and arithmetic means depending on the degree
    of  code  parallelism and optimization.  More accurate estimates of cpu
    workload rates depend on assigning appropriate weights for each kernel.

McMahon's analysis goes on to suggest that the harmonic mean corresponds to
approximately 40% vectorization, the geometric mean to approximately 70% vec-
torization, and the arithmetic mean to 90%o vectorization.  These three statis-
tics can be interpreted as different benchmarks that each characterize certain
applications.  For example, there is fair agreement between the kernels' har-
monic mean and Spice performance.  LINPACK, on the other hand, is better
characterized by the geometric mean.

On the next two pages are shown a summary of results from McMahon's report,
followed by the complete M/1000 results.  (Given the volume of data, we've only
done this on M/1000s.  M/800s scale directly by .833X).

The complete M/1000 data shows that MIPS performance is insensitive to vector
length.  The minimum to maximum variation is also small for this benchmark.
Both characteristics are typical of scalar machines with mature compilers.
Performance of vector and parallel machines, on the hand, may span two orders
of magnitude on this benchmark, or more, depending on the kernel and the vector
length.
                       64-Bit Livermore FORTRAN Kernels
                 MegaFlops, L = 167, Sorted by Geometric Mean
      Harm.  Geom.   Arith.           Rel.*
Min   Mean   Mean     Mean     Max    Geom.               System
 .05    .12    .12      .13      .24     .7  VAX 780 w/FPA 4.3BSD f77 [ours]
 .06    .16    .17      .18      .28    1.0  VAX 780 w/FPA VMS 4.1
 .11    .30    .33      .37      .87    1.9  SUN 3/160 w/FPA

 .20    .42    .46      .50     1.42    2.5  MIPS M/500
 .17    .43    .48      .53     1.13    2.8  SUN 3/260 w/FPA [our numbers]
 .29    .58    .64      .70     1.21    3.8  Alliant FX/1 FX 2.0.2 Scalar
 .38    .72    .77      .83     1.57    4.5  SUN 4/260 w/FPA [Hough 87]

 .39    .94   1.00     1.04     1.64    5.9  VAX 8700 w/FPA VMS 4.1
 .10    .76   1.06     1.50     5.23    6.2  Alliant FX/1 FX 2.0.2 Vector
 .33    .92   1.06     1.20     2.88    6.2  Convex C-1 F77 V2.1 Scalar
 .52   1.09   1.19     1.30     2.74    7.0  ELXSI 6420 EMBOS F77 MP=1
 .51   1.26   1.37     1.48     2.70    8.1  MIPS M/800
 .61   1.51   1.65     1.78     3.24    9.7  MIPS M/1000

1.01   1.06   1.94     3.33    12.79   11.4  Convex C-1 F77 V2.1 Vector
 .28   1.24   2.32     5.11    29.20   13.7  Alliant FX/8 FX 2.0.2 MP=8*Vec
1.51   4.93   5.86     7.00    17.43   34.5  Cray-1S CFT 1.4 scalar
1.23   4.74   6.09     7.67    21.64   35.8  FPS 264 SJE APFTN64
3.43   9.29  10.68    12.15    25.89   62.8  Cray-XMP/1 COS CFT77.12 scalar
0.97   6.47  11.94    22.20    82.05   70.2  Cray-1S CFT 1.4 vector
4.47  11.35  13.08    15.20    45.07   76.9  NEC SX-2 SXOS1.21 F77/SX24 scalar
1.47  12.33  24.84    50.18   188     146    Cray-XMP/1 COS CFT77.12 vector
4.47  19.07  43.94   140     1042     258    NEC SX-2 SXOS1.21 F77/SX24 vector

                       32-Bit Livermore FORTRAN Kernels
                 MegaFlops, L = 167, Sorted by Geometric Mean
      Harm.   Geom.    Arith.           Rel.*
Min   Mean    Mean      Mean     Max    Geom.               System
.05     .18     .20      .23      .48     .7    VAX 780 4.3BSD f77 [ours]
.10     .28     .30      .32      .58    1.0    VAX 780 w/FPA VMS 4.1
.19     .46     .50      .56     1.26    1.7    SUN 3/160 w/FPA
.30     .65     .71      .77     1.55    2.4    SUN 3/260 w/FPA [ours]
.30     .66     .74      .83     1.60    2.5    Alliant FX/1 FX 2.0.2 Scalar
.10     .60     .90     1.31     4.23    3.0    Alliant FX/1 FX 2.0.2 Vector

.40     .97    1.05     1.14     2.08    3.5    MIPS M/500
.55    1.04    1.12     1.20     2.21    3.7    SUN 4/260 w/FPA [Hough 87]
.36    1.11    1.27     1.42     3.61    4.2    Convex C-1 F77 V2.1 Scalar
.46    1.26    1.36     1.45     2.41    4.5    VAX 8700 w/FPA VMS 4.1
.68    1.31    1.46     1.61     3.19    4.9    ELXSI 6420 EMBOS F77 MP=1

.93    2.02    2.19     2.36     3.96    7.3    MIPS M/1000

.28    1.30    2.47     5.59    33.52    8.2    Alliant FX/8 FX 2.0.2 MP=8*Vec
.12    1.27    2.73     5.44    23.60    9.1    Convex C-1 F77 V2.1 Vector

* Relative Performance, as ratio of the Geometric Mean numbers.  This is a
  simplistic attempt to extract a single figure-of-merit.  We admit this goes
  against the grain of this benchmark.  The next table gives the complete
  M/1000 output, in the form used by McMahon.

            Livermore FORTRAN Kernels - Complete MIPS M/1000 Output
Vendor            MIPS    MIPS    MIPS    MIPS |   MIPS    MIPS    MIPS    MIPS
Model           M/1000  M/1000  M/1000  M/1000 | M/1000  M/1000  M/1000  M/1000
OSystem         BSD2.1  BSD2.1  BSD2.1  BSD2.1 | BSD2.1  BSD2.1  BSD2.1  BSD2.1
Compiler          1.21    1.21    1.21    1.21 |   1.21    1.21    1.21    1.21
OptLevel            O2      O2      O2      O2 |     O2      O2      O2      O2
Samples             72      24      24      24 |     72      24      24      24
WordSize            64      64      64      64 |     32      32      32      32
DO Span            167      19      90     471 |    167      19      90     471
Year              1987    1987    1987    1987 |   1987    1987    1987    1987
Kernel          ------  ------  ------  ------ | ------  ------  ------  ------
       1        2.2946  2.2946  2.3180  2.3136 | 2.9536  2.9536  2.9613  2.9727
       2        1.6427  1.6427  1.8531  1.8381 | 2.1218  2.1218  2.4691  2.4758
       3        2.0625  2.0625  2.1260  2.1021 | 2.8935  2.8935  2.9389  2.9853
       4        1.3440  1.3440  1.7954  1.9600 | 1.6836  1.6836  2.2084  2.4248
       5        1.4652  1.4652  1.4879  1.4776 | 2.0924  2.0924  2.1096  2.1374
       6        1.0453  1.0453  1.3734  1.4183 | 1.3076  1.3076  1.7920  1.8517
       7        3.1165  3.1165  3.1304  3.1281 | 3.9336  3.9336  3.9581  3.9623
       8        2.4829  2.4829  2.5725  2.5686 | 3.1625  3.1625  3.2853  3.2612
       9        3.2215  3.2215  3.2359  3.2290 | 3.8708  3.8708  3.8831  3.8632
      10        1.2293  1.2293  1.2336  1.2327 | 2.4413  2.4413  2.3419  2.3263
      11        1.1907  1.1907  1.2274  1.2320 | 1.5789  1.5789  1.6365  1.6559
      12        1.2102  1.2102  1.2404  1.2308 | 1.6004  1.6004  1.6414  1.6471
      13        0.6095  0.6095  0.6272  0.6378 | 0.9288  0.9288  0.9334  0.9428
      14        0.9712  0.9712  0.9455  0.6695 | 1.2133  1.2133  1.2175  1.0092
      15        0.9894  0.9894  0.9605  0.9585 | 1.3701  1.3701  1.3314  1.3314
      16        1.6427  1.6427  1.6159  1.6272 | 1.7904  1.7904  1.7567  1.7500
      17        2.3898  2.3898  2.2674  2.2667 | 3.7320  3.7320  3.4866  3.5210
      18        2.1462  2.1462  2.3321  2.3276 | 2.5642  2.5642  2.8431  2.8365
      19        1.8268  1.8268  1.8540  1.8536 | 2.2883  2.2883  2.3369  2.3466
      20        2.7821  2.7821  2.7800  1.9158 | 3.7104  3.7104  3.7214  3.6583
      21        1.5372  1.5372  1.6004  1.6201 | 1.9644  1.9644  2.0564  2.0855
      22        1.4507  1.4507  1.4506  1.4489 | 2.0802  2.0802  2.0646  2.0658
      23        2.1395  2.1395  2.3897  2.3729 | 2.5972  2.5972  2.9489  2.9532
      24        1.0148  1.0148  1.0458  1.0448 | 1.1995  1.1995  1.2226  1.2281
--------------  ------  ------  ------  ------ | ------  ------  ------  ------
Standard  Dev.  0.6878  0.6938  0.6902  0.6742 | 0.8702  0.8870  0.8616  0.8669
Median    Dev.  0.6869  0.6445  0.7023  0.6546 | 0.9837  0.8980  1.0239  1.0360

Maximum   Rate  3.2359* 3.2215  3.2359  3.2290 | 3.9623* 3.9336  3.9581  3.9623
Average   Rate  1.7834* 1.7419  1.8110  1.7698 | 2.3611* 2.2949  2.3811  2.3872
Geometric Mean  1.6469* 1.6053  1.6752  1.6330 | 2.1935* 2.1238  2.2175  2.2168
Median    Rate  1.6272* 1.5899  1.7056  1.7326 | 2.2084* 2.1071  2.2727  2.3365
Harmonic  Mean  1.5078* 1.4724  1.5368  1.4874 | 2.0235* 1.9590  2.0503  2.0372
Minimum   Rate  0.6095* 0.6095  0.6272  0.6378 | 0.9288* 0.9288  0.9334  0.9428

Maximum   Ratio 1.0000  0.9955  1.0000  0.9978 | 1.0000  0.9927  0.9989  1.0000
Average   Ratio 1.0000  0.9767  1.0154  0.9923 | 1.0000  0.9719  1.0084  1.0110
Geometric Ratio 1.0000  0.9747  1.0171  0.9915 | 1.0000  0.9682  1.0109  1.0106
Harmonic  Mean  1.0000  0.9765  1.0192  0.9864 | 1.0000  0.9681  1.0132  1.0067
Minimum   Rate  1.0000  1.0000  1.0290  1.0464 | 1.0000  1.0000  1.0049  1.0150

* These are the numbers brought forward into the summary section.

5.2.  LINPACK (LNPK DP and LNPK SP)

The LINPACK benchmark has become one of the most widely used single benchmarks
to predict relative performance in scientific and engineering environments.
The usual LINPACK benchmark measures the time required to solve a 100x100 sys-
tem of linear equations using the LINPACK package.  LINPACK results are meas-
ured in MFlops, millions of floating point operations per second.  All numbers
are from [Dongarra 87], unless otherwise noted.

The LINPACK package calls on a set of general-purpose utility routines called
BLAS -- Basic Linear Algebra Subroutines -- to do most of the actual computa-
tion.  A FORTRAN version of the BLAS is available, and the appropriate routines
are included in the benchmark.  However, vendors often provide hand-coded ver-
sions of the BLAS as a library package.  Thus LINPACK results are usually cited
in two forms: FORTRAN BLAS and Coded BLAS.  The FORTRAN BLAS actually come in
two forms as well, depending on whether the loops are 4X unrolled in the FOR-
TRAN source (the usual) or whether the unrolling is undone to facilitate recog-
nition of the loop as a vector instruction.  According to the ground rules of
the benchmark, either may be used when citing FORTRAN BLAS results, although it
is typical to note rolled loops with the annotation ``(Rolled BLAS).''

For our own numbers, we've corrected a few to follow Dongarra more closely than
we have in the past.  LINPACK output produces quite a few MFlops numbers, and
we've tended to use the fourth one in each group, which uses more iterations,
and thus is more immune to clock randomness.  Dongarra uses the highest MFlops
number that appears, then rounds to two digits.

Note that relative ordering even within families is not particularly con-
sistent, illustrating the extreme sensitivity of these benchmarks to memory
system design.
               100x100 LINPACK Results - FORTRAN and Coded BLAS
                  From [Dongarra 87], Unless Noted Otherwise
  DP      DP     SP      SP
Fortran  Coded Fortran  Coded                       System
   .10     .10    .11     .11  Sun-3/160, 16.7MHz (Rolled BLAS)o
   .11     .11    .13     .11  Sun-3/260,25MHz 68020o20MHz 68881 (Rolled BLAS)+
   .14     -      -       -    Apollo DN4000, 25MHz (68020 o 68881) [ENEWS 87]
   .14     -      .24     -    VAX 11/780, 4.3BSD, LLL Fortran [ours]
   .14     .17    .25     .34  VAX 11/780, VAX/VMS
   .20     -      .24     -    80386+80387, 20MHz, 64K cache, GreenHills
   .20     .23    .40     .51  VAX 11/785, VAX/VMS

   .29     .49    .45     .69  Intergraph IP-32C,30Mz Clipper[Intergraph 86]
   .30     -      -       -    IBM RT PC, optional FPA [IBM 87]
   .33     -      .57     -    OPUS 300PM, Greenhills, 30MHz Clipper
   .36     .59    .51     .72  Celerity C1230, 4.2BSD f77
   .38     -      .67     -    80386oWeitek 1167,20MHz,64K cache, GreenHills
   .39     .50    .66     .81  Ridge 3200/90
   .41     .41    .62     .62  Sun-3/160, Weitek FPA (Rolled BLAS)o
   .45     .54    .60     .74  HP9000 Model 840S [HP 87]
   .46     .46    .86     .86  Sun-3/260, Weitek FPA (Rolled BLAS)o
   .47     .81    .69    1.30  Gould PN9000
   .49     .66    .84    1.20  VAX 8600, VAX/VMS 4.5
   .49     .54    .62     .68  HP 9000/825S [HP 87]

   .57     .72    .86     .87  HP9000 Model 850S [HP 87]
   .60     .72    .93    1.2   MIPS M/500
   .61     -      .84     -    DG MV20000-I, MV15000-20 [Stahlman, 87]
   .65     .76    .80     .96  VAX 8500, VAX/VMS
   .70     .96   1.3     1.9   VAX 8650, VAX/VMS
   .78     -     1.1      -    IBM 9370-90, VS FORT 1.3.0
   .97    1.1    1.4     1.7   VAX 8550/8700/8800, VAX/VMS
  1.0     1.3    1.9     3.6   MIPS M/800
  1.1     1.1    1.6     1.6   SUN 4/260 (Rolled BLAS)+
  1.2     1.7    1.3     1.6   ELXSI 6420
  1.2     1.6*   2.3*    4.3   MIPS M/1000

  1.6     2.0    1.6     2.0   Alliant FX-1 (1 CE)
  2.1      -     2.4      -    IBM 3081K H enhanced opt=3
  3.0     3.3    4.3     4.9   CONVEX C-1/XP, Fort 2.0 (Rolled BLAS)
  6.0      -      -       -    Multiflow Trace 7/200 Fortran 1.4 (Rolled BLAS)
  7.0    11.0    7.6     9.8   Alliant FX-8, 8 CEs, FX Fortran, v2.0.1.9

 12      23     n.a.    n.a.   CRAY 1S CFT (Rolled BLAS)
 39      57     n.a.    n.a.   CRAY X-MP CFT (Rolled BLAS)
 43        -    44        -    NEC SX-2, Fortran 77/SX (Rolled BLAS)

+ The Sun FORTRAN Rolled BLAS code appears to be optimal, so we used the same
  numbers for Coded BLAS.  The 4X unrolled numbers for Sun-4 are .86 (DP) and
  1.25 (SP) [Hough 87].

* These numbers are as reported by Dongarra.  We prefer the typical results,
  which are slightly lower, viz. 1.2, 1.5, 2.2, and 4.3.

On the next page, we take a subset of these numbers, and normalize them to the
VAX/VMS 11/780.

               100x100 LINPACK Results - FORTRAN and Coded BLAS
                         VAX/VMS Relative Performance
                          For A Subset of the Systems
 Rel.    Rel.     Rel.    Rel.
  DP      DP       SP      SP
Fortran  Coded   Fortran  Coded                      System
    .8      .6       .5      .3   Sun-3/260,25MHz 68020o20MHz 68881 (Rolled)
   1.0     1.0      1.0     1.0   VAX 11/780, VAX/VMS
   1.4     -        1.0     -     80386o80387, 20MHz, 64K cache, GreenHills
   2.0     2.9      1.8     2.0   Intergraph IP-32C,30Mz Clipper[Intergraph 86]

   2.7     -        2.7     -     80386oWeitek 1167,20MHz,64K cache, GreenHills
   2.9     2.4      2.5     1.8   Sun-3/160, Weitek FPA (Rolled BLAS)
   3.3     2.7      3.4     2.5   Sun-3/260, Weitek FPA (Rolled BLAS)
   3.5     3.9      3.4     3.5   VAX 8600, VAX/VMS 4.5

   4.1     4.2      3.4     2.6   HP9000 Model 850S [HP 87]
   4.3     4.2      3.7     3.5   MIPS M/500
   6.9     6.6      5.6     5.0   VAX 8550/8700/8800, VAX/VMS
   7.1     7.6      7.6    10.6   MIPS M/800
   7.9     6.5      6.4     4.7   SUN 4/260 (Rolled BLAS)
   8.6     9.4      9.2    12.6   MIPS M/1000

  11.4    11.8      6.4     5.9   Alliant FX-1 (1 CE)
  21.4    19.4     17.2    14.4   CONVEX C-1/XP, Fort 2.0 (Rolled BLAS)
  50      65       30      28.8   Alliant FX-8, 8 CEs, FX Fortran, v2.0.1.9
 307       -      176       -     NEC SX-2, Fortran 77/SX (Rolled BLAS)

5.3.  Spice Benchmarks (SPCE 2G6)

Spice [UCB 87] is a general-purpose circuit simulator written at U.C.  Berke-
ley.  Spice and its derivatives are widely used in the semiconductor industry.
It is a valuable benchmark because it shares many characteristics with other
real-world programs that are not represented in popular small benchmarks.  It
uses both integer and floating-point computation heavily.  The floating-point
calculations are not vector oriented, as in LINPACK.  Also, the program itself
is very large and therefore tests both instruction and data cache performance.

We have chosen to benchmark Spice version 2g.6 because of its general availa-
bility.  This is one of the later and more popular Fortran versions of Spice
distributed by Berkeley.  We felt that the circuits distributed with the Berke-
ley distribution for testing and benchmarking were not sufficiently large and
modern to serve as benchmarks.  In previous version of this brief, we presented
results on circuits we felt were representative, but which contained
proprietary data.  This time, we gathered and produced appropriate benchmark
circuits that can be distributed, and have since been posted as public domain
on Usenet.  The Spice group at Berkeley found these circuits to be up-to-date
and good candidates for Spice benchmarking.  By distributing the circuits we
obtained results for many other machines.  In the table below, "Geom Mean" is
the geometric mean of the 3 "Rel." columns.

                          Spice2G6 Benchmarks Results
   digsr       bipole    comparator   Geom
  Secs Rel.   Secs Rel.   Secs Rel.   Mean System
1354.0 0.60  439.6 0.68  460.3 0.63    .6  VAX 11/780 4.3BSD, f77 V2.0
 993.5 0.81  394.3 0.76  366.9 0.80    .8  Microvax-II Ultrix 1.1, fortrel
 901.9 0.90  285.1 1.0   328.6 0.89    .9  SUN 3/160 SunOS 3.2 f77 -O -f68881
 848.0 0.95  312.6 0.96  302.9 0.96   1.0  VAX 11/780 4.3BSD, fortrel -opt
 808.1 1.0   299.1 1.0   291.7  1.0   1.0  VAX 11/780 VMS 4.4 /optimize
 744.8 1.1   221.7 1.3   266.0  1.1   1.2  SUN 3/260 SunOS 3.2 f77 -O -f68881
 506.5 1.6   170.0 1.8   189.1  1.5   1.6  SUN 3/160 SunOS 3.2 f77 -O -ffpa

 361.2 2.2   112.0 2.7   129.4  2.3   2.4  SUN 3/260 SunOS 3.2 f77 -O -ffpa
 296.5 2.7    73.4 4.1    83.0  3.5   3.4  MIPS M/500
 225.9 3.6    63.7 4.7    73.4  4.0   4.1  SUN 4/260 f77 -O3 -Qoption as -Ff0+

     -  -        -  -        -   -    5.3  VAX 8700 (estimate)
 136.5 5.9    42.6 7.0    41.4  7.0   6.6  MIPS M/800
 125.5 6.4    39.5 7.6    39.3  7.4   7.1  AMDAHL 470V7 VMSP FORTVS4.1
 114.3 7.1    35.4 8.4    34.5  8.5   8.0  MIPS M/1000

  48.0 16.8   12.5 23.9   17.5 16.7   18.9 FPS 20/64 VSPICE (2g6 derivative)

+   Sun numbers are from [Hough 87], who notes that the Sun-4 number was beta
    software, and that a few modules did not optimize.

Benchmark descriptions:
digsr    CMOS 9 bit Dynamic shift register with parallel load capability, i.e.,
         SISO (Serial Input Serial Output) and PISO (Parallel Input Serial Out-
         put), widely used in microprocessors.  Clock period is 10 ns.  Channel
         length = 2 um, Gate Oxide = 400 Angstrom.  Uses MOS LEVEL=2.
bipole   Schottky TTL edge-triggered register.  Supplied with nearly- coin-
         cident inputs (synchronizer application).
comparator
         Analog CMOS auto-zeroed comparator, composed of Input, Differential
         Amplifier and Latch.  Input signal is 10 microvolts.  Channel Length =
         3 um, Gate Oxide = 500 Angstrom.  Uses MOS LEVEL=3.  Each part is con-
         nected by the capacitance coupling, which is often used for the offset
         cancellation.  (Sometimes called Toronto, in honor of its source).

Hspice is a commercial version of Spice offered by Meta-Software, which
recently published benchmark results for a variety of machines [Meta-software
87].  (Note that the M/800 number cited there was before the UMIPS-BSD 2.1 and
f77 1.21 releases, and the numbers have improved).  The VAX 8700 Spice number
(5.3X) was estimated by using the Hspice numbers below for 8700 and M/800, and
the M/800 Spice number:
(5.5: 8700 Hspice) / (6.9: M/800 Hspice) X (6.6: M/800 Spice) yields 5.3X.

This section indicates that the performance ratios seem to hold for at least
one important commercial version as well.

                           Hspice Benchmarks Results
                                 HSPICE-8601K
               ST230
 Secs                          Rel.                          System

166.5                            .6                          VAX 11/780, 4.2BSD
92.2                            1.0                          VAX 11/780 VMS
91.5                            1.0                          Microvax-II VMS

29.2                            3.2                          ELXSI 6400
29.1                            3.2                          Alliant FX/1
25.3                            3.6                          HyperSPICE (EDGE)
16.8                            5.5                          VAX 8700 VMS
16.3                            5.7                          IBM 4381-12

13.4                            6.9                          MIPS M/800 [ours]
11.3                            8.2                          MIPS M/1000 [ours]

 3.27                          28.2                          IBM 3090
 2.71                          34.0                          CRAY-1S

5.4.  Digital Review (DIG REV)

The Digital Review magazine benchmark [DR 87] is a 3300-line FORTRAN program
that includes 33 separate tests, mostly floating-point, some integer.  The
magazine reports the times for all tests, and summarizes them with the
geometric mean seconds shown below.  All numbers below are from [DR 87], except
the M/500 and M/800 figures.

                       Digital Review Benchmarks Results
 Secs                      Rel.                      System
9.17                       1.0                       VAXstation II/GPX, VMS 4.5

2.90                       3.2                       VAXstation 3200
2.32                       4.0                       VAX 8600, VMS 4.5
2.09                       4.4                       Sun-4, SunOS 3.2L
1.86                       4.9                       MIPS M/500 [ours]

1.584                      5.8                       VAX 8650
1.480                      6.2                       Alliant FX/8, 1 CE
1.469                      6.2                       VAX 8700

1.200                      7.6                       MIPS M/800 [ours]
1.193                      7.7                       ELXSI 6420

 .990                      9.3                       MIPS M/1000*

 .487                      18.8                      Convex C-1 XP

* The actual run number was .99, which [DR 87] reported as 1.00.

5.5.  Doduc Benchmark (DDUC)

This benchmark [Doduc 86] is a 5300-line FORTRAN program that simulates aspects
of nuclear reactors, has little vectorizable code, and is thought to be
representative of Monte-Carlo simulations.  It uses mostly double precision
floating point, and is often viewed as a ``nasty'' benchmark, i.e., it breaks
things, and makes machines underperform their usual VAX-mips ratings.  Perfor-
mance is given as a number R normalized to 100 for an IBM 370/168-3 or 170 for
an IBM 3033-U, [ R = 48671/(cpu time in seconds) ], so that larger R's are
better.

In order of increasing performance, following are numbers for various machines.
All are from [Doduc 87] unless otherwise specified.

                   Double Precision Doduc Benchmark Results
DoDuc R   Relative
Factor     Perf.     System
     17      0.7     Sun3/110, 16.7MHz
     19      0.7     Intel 80386o80387, 16MHz, iRMX
     22      0.8     Sun-3/260, 25MHz 68020, 20MHz 68881
     26      1.0     VAX 11/780, VMS
     33      1.3     Fairchild Clipper, 30MHz, Green Hills

     43      1.7     Sun-3/260, 25MHz, Weitek FPA
     48      1.8     Celerity C1260
     50      1.9     CCI Power 6/32
     53      2.0     Edge 1
     64      2.5     Harris HCX-7

     85      3.3     Alliant FX/1
     88      3.4     MIPS M/500, f77 -O2, runs 553 seconds
     90      3.5     IBM 4381-2
     90      3.5     Sun-4/200 [Hough 1987], SunOS 3.2L, runs 540 seconds
     91      3.5     DEC VAX 8600, VAX/VMS
     97      3.7     ELXSI 6400
     99      3.8     DG MV/20000
    100      3.8     MIPS M/500, f77 -O3, runs 488 seconds
    101      3.9     Alliant FX/8
    113      4.3     FPSystems 164
    119      4.6     Gould 32/8750
    129      5.0     DEC VAX 8650
    136      5.2     DEC VAX 8700, VAX/VMS

    150      5.7     Amdahl 470 V8, VM/UTS
    178      6.8     MIPS M/800, f77 -O2, runs 273 secs
    181      7.0     IBM 3081-G, F4H ext, opt=2
    190      7.3     MIPS M/800, f77 -O3, runs 256 secs
    218      8.4     MIPS M/1000, f77 -O2, runs 223 secs
    229      8.8     MIPS M/1000, f77 -O3, runs 213 secs
    236      9.1     IBM 3081-K

    475     18.3     Amdahl 5860
    714     27.5     IBM 3090-200, scalar mode

   1080     41.6     Cray X/MP [for perspective:we have a lonnng way to go yet!]

5.6.  Whetstone

Whetstone is a synthetic mix of floating point and integer arithmetic, function
calls, array indexing, conditional jumps, and transcendental functions [Curnow
76].

Whetstone results are measured in KWips, thousands of Whetstone interpreter
instructions per second.  In this case, some of our numbers actually went down,
although compiled code has generally improved.  First, the accuracy of several
library routines was improved, at a slight cost in performance.  Second, on
machines this fast, relatively few clock ticks are actually counted, and UNIX
timing includes some variance.  We've been running many runs and averaging.
We've now increased the loop counts from 10 to 1000 to increase the total run-
ning time to the point where the variance is reduced.  This changed the bench-
mark slightly.  Our experiences show some general uncertainty about the numbers
reported by anybody: we've heard that various different source programs are
being used.

                          Whetstone Benchmark Results
  DP     DP     SP     SP
KWips   Rel.  Kwips   Rel.  System
   410   0.5     500  0.4   VAX 11/780, 4.3BSD, f77 [ours]
   715   0.9   1,083  0.9   VAX 11/780, LLL compiler [ours]
   830   1.0   1,250  1.0   VAX 11/780 VAX/VMS [Intergraph 86]

   960   1.2   1,040  0.8   Sun3/160, 68881 [Intergraph 86]
 1,110   1.3   1,670  1.3   VAX 11/785, VAX/VMS [Intergraph 86]
 1,230   1.5   1,250  1.0   Sun3/260, 25MHz 68020, 20MHz 68881
 1,400   1.7   1,600  1.3   IBM RT PC, optional FPA [IBM 87]

 1,730   2.1   1,860  1.5   Intel 80386o80387, 20MHz, 64K cache, GreenHills
 1,740   2.1   2,980  2.4   Intergraph InterPro-32C,30MHz Clipper[Intergraph86]
 1,744   2.1   2,170  1.7   Apollo DN4000, 25MHz 68020, 25MHz 68881 [ENEWS 87]
 1,860   2.2   2,400  1.9   Sun3/160, FPA
 2,092   2.5   3,115  2.5   HP 9000/840S [HP 87]
 2,433   2.9   3,521  2.8   HP 9000/825S [HP 87]
 2,590   3.1   4,170  3.3   Intel 80386oWeitek 1167, 20MHz, Green Hills
 2,600   3.1   3,400  2.7   Sun3/260, Weitek FPA [measured elsewhere]
 2,670   3.2   4,590  3.7   VAX 8600, VAX/VMS [Intergraph 86]
 2,907   3.5   4,202  3.4   HP 9000 Model 850S [HP 87]

 3,540   4.3   5,290  4.2   Sun-4 (reported secondhand, not confirmed)
     -   -     6,400  5.1   DG MV/15000-12
 3,950   4.8   6,670  5.3   VAX 8700, VAX/VMS, Pascal(?) [McInnis, 1987]
 4,000   4.8   6,900  5.5   VAX 8650, VAX/VMS [Intergraph 86]
 4,120   5.0   4,930  3.9   Alliant FX/8  (1 CE) [Alliant 86]
 4,200   5.1       -   -    Convex C-1 XP [Multiflow]
 4,220   5.1   5,430  4.3   MIPS M/500

 6,930   8.0   8,570  6.9   MIPS M/800
 7,960   9.6  10,280  8.2   MIPS M/1000
12,605  15         -   -    Multiflow Trace 7/200 [Multiflow]
25,000  30         -   -    IBM 3090-200 [Multiflow]
35,000  42         -   -    Cray X-MP/12

6.  Acknowledgements

Some people have noted that they seldom believe the numbers that come from cor-
porations unless accompanied by names of people who take responsibility for the
numbers.  Many people at MIPS have contributed to this document, which was ori-
ginally created by Web Augustine.  Particular contributors to this issue
include Mark Johnson (much Spice work, including creation of public-domain
Spice benchmarks), and especially Earl Killian (a great deal of work in various
areas, particularly floating-point).  Final responsibility for the numbers in
this Brief is taken by the editor, John Mashey.

We thank David Hough of Sun Microsystems, who kindly supplied numbers for some
of the Sun configurations, even fixing a few of our numbers that were
incorrectly high, and who has also offered good comments on joint efforts look-
ing for higher-quality benchmarks.

We also thank Cliff Purkiser of Intel, who posted the Intel 80386 Whetstone and
LINPACK numbers on Usenet.

We also thank Greg Pavlov, who ran hordes of Stanford and Dhrystone benchmarks
for us on a VAX 8550, Ultrix 2.0 system.

7.  References

[Alliant 86]
   Alliant Computer Systems Corp, "FX/Series Product Summary", October 1986.

[Curnow 76]
   Curnow, H. J., and Wichman, B. A., ``A Synthetic Benchmark'', Computing
   Journal, Vol. 19, No. 1, February 1976, pp. 43-49.

[Doduc 87]
   Doduc, N., FORTRAN Central Processor Time Benchmark, Framentec, June 1986,
   Version 13.  Newer numbers were received 03/17/87, and we used them where
   different.
   E-mail: seismo!mcvax!ftcsun3!ndoduc

[Dongarra 87]
   Dongarra, J., ``Performance of Various Computers Using Standard Linear Equa-
   tions in a Fortran Environment'', Argonne National Laboratory, August 10,
   1987.

[Dongarra 87b]
   Dongarra, J., Marin, J., Worlton, J., "Computer Benchmarking: paths and pit-
   falls", IEEE Spectrum, July 1987, 38-43.

[DR 87]
   "A New Twist: Vectors in Parallel", June 29, 1987, "The M/1000: VAX 8800
   Power for Price of a MicroVAX II", August 24, 1987, and "VAXstation 3200
   Benchmarks: CVAX Eclipses MicroVAX II", September 14, 1987.  Digital Review,
   One Park Ave., NY, NY 10016.

[ENEWS 87]
   Electronic News, ``Apollo Cuts Prices on Low-End Stations'', July 6, 1987,
   p. 16.

[Fleming 86]
   Fleming, P.J. and Wallace, J.J.,``How Not to Lie With Statistics: The
   Correct Way to Summarize Benchmark Results'', Communications of the ACM,
   Vol. 29, No. 3, March 1986, 218-221.

[HP 87]
   Hewlett Packard, ``HP 9000 Series 800 Performance Brief'', 5954-9903, 5/87.
   (A comprehensive 40-page characterization of 825S, 840S, 850S).

[Hough 86,1]
   Hough, D., ``Weitek 1164/5 Floating Point Accelerators'', Usenet, January
   1986.

[Hough 86,2]
   Hough, D., ``Benchmarking and the 68020 Cache'', Usenet, January 1986.

[Hough 86,3]
   Hough, D., ``Floating-Point Programmer's Guide for the Sun Workstation'',
   Sun Microsystems, September 1986. [an excellent document, including a good
   set of references on IEEE floating point, especially on micros, and good
   notes on benchmarking hazards].  Sun3/260 Spice numbers are from later mail.

[Hough 87]
   Hough, D., ``Sun-4 Floating-Point Performance'', Usenet, 08/04/87.

[IBM 87]
   IBM, ``IBM RT Personal Computer (RT PC) New Models, Features, and Software
   Overview, February 17, 1987.

[Intergraph 86]
   Intergraph Corporation, ``Benchmarks for the InterPro 32C'', December 1986.

[Meta-Software 87]
   Meta-Software, ``HSPICE Performance Benchmarks'', June 1987.  50 Curtner
   Avenue, Suite 16, Campbell, CA 95008.

[McInnis 87]
   McInnis, D., Kusik, R., Bhandarkar, D., ``VAX 8800 System Overview'', Proc.
   IEEE COMPCON, March 1987, San Francisco, 316-321.

[McMahon 86]
   ``The Livermore Fortran Kernels: A Computer Test of the Numerical Perfor-
   mance Range'', December 1986, Lawrence Livermore National Labs.

[MIPS 87]
   MIPS Computer Systems, "A Sun-4 Benchmark Analysis", and "RISC System Bench-
   mark Comparison: Sun-4 vs MIPS", July 23, 1987.

[Purkiser 87]
   Purkiser, C., ``Whetstone and LINPACK Numbers'', Usenet, March 1987.

[Richardson 87]
   Richardson, R., ``9/20/87 Dhrystone Benchmark Results'', Usenet, Sept. 1987.
   Rick publishes the source several times a year.  E-mail address:
   ...!seismo!uunet!pcrat!rick

[Serlin 87a]
   Serlin, O., ``MIPS, DHRYSTONES, AND OTHER TALES'', Reprinted with revisions
   from SUPERMICRO Newsletter, April 1986, ITOM International, P.O. Box 1450,
   Los Altos, CA 94023.
   Analyses on the perils of simplistic benchmark measures.

[Serlin 87b]
   Serlin, O., SUPERMICRO #69, July 31, 1987. pp. 1-2.
   Offers good list of attributes customers should demand of vendor benchmark-
   ing.

[Stahlman 87]
   Stahlman, M., "The Myth of Price/performance", Sanford C. Bernstein & Co,
   Inc, NY, NY, March 17, 1987.

[Sun 86]
   SUN Microsystems, ``The SUN-3 Family: A Hardware Overview'', August 1986.

[Sun 87]
   SUN Microsystems, SUN-4 Product Introduction Material, July 7, 1987.

[UCB 87]
   U. C. Berkeley, CAD/IC group, ``SPICE2G.6'', March 1987. Contact: Cindy
   Manly, EECS/ERL Industrial Liason Program, 479 Cory Hall, University of Cal-
   ifornia, Berkeley, CA 94720.

[Weicker 84]
   Weicker, R. P., ``Dhrystone: A Synthetic Systems Programming Benchmark'',
   Communications of the ACM, Vol. 27, No. 10, October 1984, pp.  1013-1030.
________
UNIX is a Registered Trademark of AT&T.  DEC, VAX, Ultrix, and VAX/VMS are
trademarks of Digital Equipment Corp.  Sun-3, Sun-4 are Trademarks on Sun
Microsystems.  Many others are trademarks of their respective companies.
-- 
-john mashey	DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP: 	{decvax,ucbvax,ihnp4}!decwrl!mips!mash  OR  m...@mips.com
DDD:  	408-991-0253 or 408-720-1700, x253
USPS: 	MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086

			        About USENET

USENET (Users’ Network) was a bulletin board shared among many computer
systems around the world. USENET was a logical network, sitting on top
of several physical networks, among them UUCP, BLICN, BERKNET, X.25, and
the ARPANET. Sites on USENET included many universities, private companies
and research organizations. See USENET Archives.

		       SCO Files Lawsuit Against IBM

March 7, 2003 - The SCO Group filed legal action against IBM in the State 
Court of Utah for trade secrets misappropriation, tortious interference, 
unfair competition and breach of contract. The complaint alleges that IBM 
made concentrated efforts to improperly destroy the economic value of 
UNIX, particularly UNIX on Intel, to benefit IBM's Linux services 
business. See SCO vs IBM.

The materials and information included in this website may only be used
for purposes such as criticism, review, private study, scholarship, or
research.

Electronic mail:			       WorldWideWeb:
   tech-insider@outlook.com			  http://tech-insider.org/