In simple terms, BTU is a new Host Communication Benchmark that provides a good
performance measure for workstations in realistic network conditions.
Our BTU Communication Benchmark does not test the maximum performance of a workstation
in a homogeneous network under ideal circumstances.
Rather, we take a vendor supplied workstation running UNIX® operating system,
install our benchmark program and connect the workstation to a blackbox
testbed which emulates a LAN connected to a WAN. The benchmark run
will submit the workstation to a carefully designed combination of
tests. The result is an indicator of what the user, at the
application level, can expect in terms of bits sent to or received from a remote host.
Our benchmark, in contrast to maximum performance tests, takes into
account concurrent activities, such as CPU and
I/O activities, which compete for resources on the test machine and
concurrent activities on the network which will interfere with the
test machine's communication.
On the one hand we intend these results
for the workstation buyer who can use the data together with the
configuration specification and the list price to make a well reasoned
decision if the workstation meets the needs when
compared to the results of other workstations. We also
believe these results can provide insights to the architects as to the
location and nature of the communication bottlenecks of a particular
workstation.
In the
paper we conclude with a discussion of our
plans for having the community accept this proposal as a communication
benchmark.
What is Our Basic Goal ?
In the BTU benchmarking process, the
vendor supplies a workstation running a UNIX
operating system, we
install our benchmark program and connect the workstation to a
testbed, consisting of blackboxes, which emulates a LAN/WAN environment.
The benchmark run
will submit the workstation to a carefully designed combination of
tests. The result is a predictor of what the user, at the
application level, can
expect in terms of bits sent to or received from a remote host. Our
benchmark takes into
account concurrent activities such as CPU and
I/O activities which compete for resources on the test machine. The
combination of these activities and
concurrent activities on the network will interfere with the
test machine's communication performance. This methodology is in contrast
to existing benchmark suites that measure just the compute performance of
a given workstation or the maximum network throughput under ideal
conditions.
The automated BTU benchmark test suite produces results
at various levels of abstractions ranging from a single number, characterizing
average performance in the style of SPEC92 [An Article by Adrian Cockcroft], to a TCP time sequence
chart for abnormal behaviors.
We intend to serve the
user of a workstation and provide data on how a particularly
configured workstation can be expected to perform in a realistic
network environment. The information should enable the users to make a
reasonable judgment when acquiring a workstation with a specific
configuration within certain cost constraints.
For this reason, we give the results of the benchmark test together with a
detailed specification of the test machine and its list price.
Our results when used along with existing benchmarks will make a powerful
combination for predicting good overall performance.
Neither price nor SPECmarks are as good a predictor of communication performance
as that measured by BTUs.
We present the results of running the modified benchmark(BTU'96) on
five different workstations (SunSparc 20/Solaris 2.3, SunSparc5/
SunOs4.1.4, SunSparc5/Solaris2.3, SGI Indy(R4000PC)/ IRIX 5.3, Intel
80486 PC/Linux 1.1.35 ) from different. The results (BTU'96) show
interesting performance differences in Operating Systems ( SunOs4.1.4,
Solaris2.3) when the same workstation (SunSparc5) is configured for different
OS. Again, the ranking of the machines show no correlation to price or
SPECmarks but reflect how useful a workstation will be when
communication is a major application.
BTU'95
We present the results (BTU'95) of applying the BTU
benchmark to four workstations (SunSparc 10/Solaris 2.3, SGI
Indy(R4000PC)/ IRIX 5.3, DEC ALPHA Model:200(4/233)/ OSF/1, Intel
80486 PC/FreeBSD ver1.1) from different vendors which shows that
neither price nor SPECmarks
are a good predictor for communication performance as measured by BTUs.
Source Code of BTU Benchmark
The BTU Client Benchmark Code for measuring the BTU rating of a
workstation to be tested has been
ported for SunOs4.1.3, SunOs4.1.4, SunOS5.3, SunOS5.4,SunOS5.5
(Solaris), SGI Indy R4000 PC with IRIX 5.3, DEC Model:200(4/233) OSF/1
V3.2c, 80486 PC with FreeBSD, Linux. The implementation of the
server("blackboxes' in the terminology of the papers)runs under SunOs
4.1.3 (most probably SunOs 4.1.4). To get
accurate results, the server program should be complied on two
SUNSparc workstations (a Sparc Station10 as an Active BlackBox and a
Sparc Station2 as a Passive BlackBox) running SunOs 4.1.3 (with source
code). The patch for the WAN emulation
has been provided in the package, but will be effective only for users
who have SunOs 4.1.3 source code available on their server. For BTU'97
we will upgrade the testbed, most likely to Ultrasparcs running
Solaris 2.5. To get a feel of the BTU benchmark a user can run the
program without the kernel source code if he wishes not to have the WAN emulation features
in the BTU benchmark.
Further information about Bits to the User (BTU): A Communication Benchmark can be obtained from maly@cs.odu.edu, Department Of Computer Science , ODU, Norfolk, Va.
Other Benchmark Resources
O ld D ominion
U niversity.