[OpenAFS] OpenAFS speed
Hartmut Reuter
reuter@rzg.mpg.de
Wed, 25 Jun 2003 10:25:34 +0200
Stephen Joyce schrieb:
> On Tue, 24 Jun 2003, Hartmut Reuter wrote:
>
>
>>In our Linux-cluster with Gbit Ethernet
>>and ramdisk cache I see transfer rates of > 25 MB/s into AFS.
>
>
> I've never seen anyone claim transfer rates anywhere close to this with any
> brand of AFS (except for the testing that umich did ~10 years ago bypassing
> the cm and using dual paths).
>
> Did you hack the client or server or do anything special to see this speed?
We are using a client version of OpenAFS based on the CVS-version from
October last year. This because we need large file support and we also
have AIX 5.1 machines. But I think there is nothing special with respect
to perforance.
The fileservers are MR-AFS servers, of course namei, with the old lwp
threading. In this special case we do not use shared residencies, but
it's exactly the same as in OpenAFS, except the large files support.
The client machine is an IBM blade center PC, double processor, with
Redhat 8.0. The server is a blade center head node (because of the
FC-interface needed for the IDE-RAID5), also double processor, with
Redhat 7.3. Writing directly into the vicepa-partition allows for a
transfer rate of about 80 MB/s. All blade center machines have Gbit
ethernet to a common switch.
I have a special test script to test the read and write performance. The
script starts a program which gets two parameters: file size and number
of streams. It forks itself number of streams times and each copy looks
for a file "raidfile.stream-number". If the file exists it reads the
file, otherwise it creates and writes the file.
The script does 7 tests with the following numbers of streams: 1, 1, 2,
4, 4, 8, 8
That means: 1st it writes a big file and measures performance, then it
reads it. Then it writes another big file and reads at the same time the
first one. Then it writes two more big files and reads the existing two
files in parallel. Then the same with 4 new files ending up with the
parallel read of 8 files. Here we come to the limit of 4 calls per
connection
Because of the size of the files (1GB or 8GB) and the use of only 16 MB
ramdisk as cache AFS client caching cannot have any effect on the read
performance.
Output of the script for 8 GB files is the following:
==========================================================
running on machine Linux bc-c4.rzg.mpg.de 2.4.18-27-compute #3 SMP Thu
Apr 3 18:16:46 CEST 2003 i686 unknown
at Tue Jun 24 17:15:11 MEST 2003
in directory /afs/ipp/home/h/hwr/test/bc-hn2
Using files with option -g and size 8
1st test: write raidfile.0
real 5m7.033s
user 0m1.640s
sys 2m53.870s
2nd test: read raidfile.0
real 4m38.770s
user 0m2.680s
sys 0m39.280s
3rd test: write raidfile.1 and read raidfile.0
real 8m21.970s
user 0m4.370s
sys 3m46.940s
4th test: write raidfile.2 and raidfile.3 and read the others
real 16m24.759s
user 0m9.550s
sys 9m8.730s
5th test read 4 files in parallel
real 15m30.249s
user 0m10.640s
sys 7m12.800s
6th test: write raidfile.4 to raidfile.7 and read the others
real 33m34.037s
user 0m18.340s
sys 19m58.660s
7th test read 8 files in parallel
real 29m6.934s
user 0m22.340s
sys 16m6.000s
Average values:
write 1 stream 27338 KB/s
read 1 stream 30100 KB/s
write 2 streams 21809 KB/s
read 2 streams 16715 KB/s
write 4 streams 10554 KB/s
read 4 streams 8883 KB/s
write 8 streams 5795 KB/s
read 8 streams 4612 KB/s
=====================================================================
The number of streams for the average values is the number of copies of
the program running at the same time. So "write 8 streams" is actually
the average of the 4 write streams in test 6 while 4 read streams were
running at the same time.
The aggregate transfer rate comes close to 40 MB/s in the case of 4 and
8 streams parallel.
But still less than 1 CPU is used in average at the client and also at
the server side and we are far from the network bandwidth. So there is
still some unnecessary waiting.
If someone is interested in the script and program he/she can find them in
/afs/ipp-garching.mpg.de/u/hwr/public/performance-tests
-Hartmut
>
> Cheers,
> Stephen
> --
> Stephen Joyce
> Systems Administrator P A N I C
> Physics & Astronomy Department Physics & Astronomy
> University of North Carolina at Chapel Hill Network Infrastructure
> voice: (919) 962-7214 and Computing
> fax: (919) 962-0480 http://www.panic.unc.edu
>
> "UNIX *is* user friendly. It's just selective about who its friends are."
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
--
-----------------------------------------------------------------
Hartmut Reuter e-mail reuter@rzg.mpg.de
phone +49-89-3299-1328
RZG (Rechenzentrum Garching) fax +49-89-3299-1301
Computing Center of the Max-Planck-Gesellschaft (MPG) and the
Institut fuer Plasmaphysik (IPP)
-----------------------------------------------------------------