[OpenAFS-devel] performance study

[Garrett Wollman]

I did some (admittedly poorly-controlled) experiments looking at write
speed.  Keep in mind that I'm really comparing apples to oranges here,
but sometimes that's a semi-useful thing to do.

The two client machines are a new, single-processor AMD FX-53 system
running FreeBSD/amd64 5.3 and (statically-linked) Arla 0.38 with a few
patches; and a lightly-loaded two-processor Xeon 2.4 Debian system
running the Linux 2.4.28 kernel and the OpenAFS 1.2.13 client.  The
server is the same platform as the Linux client except that it's
running OpenAFS 1.3.74.

The test was the same on both systems: rewrite a 1000-Mbyte file
sequentially with zeros using:
	time dd if=/dev/zero of=tmp bs=1024k count=1000

I tracked the reported real times for both systems.  User time was
near nil, and system time naturally varies because of the difference
in architecture between OpenAFS and Arla.

The first result shows the difference in times.

x FreeBSD/Arla
+ Linux/OpenAFS
+--------------------------------------------------------------------------+
|          +            +                                  x xxxxxxxxxxxx  |
|       +  + ++  +     ++                                  x xxxxxxxxxxxx  |
|++ +   +  + +++ + +++ ++++                              x xxxxxxxxxxxxxx x|
|       |______MA______|                                       |__A__|     |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x 100       163.635       167.887       166.022     165.90256    0.83112684
+  25         149.8       155.962       153.215      153.4062      1.832173
Difference at 95.0% confidence
        -12.4964 +/- 0.482292
        -7.53235% +/- 0.290708%
        (Student's t, pooled s = 1.10045)

The Linux system is clearly faster in this case, although there is
more noise in the data due to confounding factors (that machines was
not completely idle).  This is particularly interesting because the
Linux machine has only a 50-Mbyte cache, as compared to 3 Gbyte on the
FreeBSD system.  However, this very difference can explain the
disparity, as we'll see in a moment.

The FreeBSD dd(1) utility has the convenient feature of displaying its
real time and transfer rate after it finishes writing the output file,
but (and significantly for this test) before the output file is
closed.  Given a sufficiently-large cache, Arla does not appear to
write back until the file is closed, so we can effectively subtract
out the cost of doing the local cache writes versus the cost of
writing the file over the network.  In this case:

x FreeBSD/Arla, excluding cache write time
+ Linux/OpenAFS, cache much smaller than file
+--------------------------------------------------------------------------+
|      xxxxxxxxx                                                        +  |
|    x xxxxxxxxx                                             +          +  |
|   xxxxxxxxxxxx                                       +  +  ++         +  |
|xxxxxxxxxxxxxxx xx                             ++ +   +  ++ ++ +++ ++ ++++|
|      |__A___|                                        |______MA_______|   |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x 100       138.737       142.813       140.868     140.93321    0.77364009
+  25         149.8       155.962       153.215      153.4062      1.832173
Difference at 95.0% confidence
        12.473 +/- 0.467272
        8.85028% +/- 0.331556%
        (Student's t, pooled s = 1.06618)

Make of these data what you will.

My guess is that the Linux machine wins in the first case because it
is able to (indeed must) overlap network and cache I/O, whereas the
FreeBSD machine is stuck waiting for 140 seconds after it's done for
the file to be written back.  I'm not sure what's going on in the
second case; most of the advantages (in terms of available processing
power and network) should be on the side of the Linux machine.  It may
be that Arla is actually doing some overlapped I/O, and my assumption
that it all happens after the close is invalid.  I do know that the
speed at which the FreeBSD machine is able to write to the cache is as
good as any other local file.

-GAWollman