[OpenAFS] Performance
Sven Oehme
oehmes@de.ibm.com
Fri, 28 Feb 2003 07:44:13 +0100
This is a multipart message in MIME format.
--=_alternative 002501DEC1256CDB_=
Content-Type: text/plain; charset="US-ASCII"
hy ,
i get much higher rates between AFS Server and my Clients :
[root@xxx21 root]# bonnie++ -d /afs/test11a/ -s 2008
Using uid:0, gid:0.
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
xxx21.de 2008M 9584 53 13950 33 3404 10 4685 18 5843 4
903.6 7
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 251 11 2401 25 480 11 345 15 1472 26 564
10
xxx21.de,2008M,9584,53,13950,33,3404,10,4685,18,5843,4,903.6,7,16,251,11,2401,25,480,11,345,15,1472,26,564,10
i have identically results from 2 Clients running bonnie at the same time
to 1 Server . i only changed the cache to memache , without any other
changes to the clients or server .
the wired thing for me is , my Server is not going over 25% CPU
Utilization when i run this test . and also the Clients are not busy .
the clients are connected with Gigabyte Ethernet (Fiber) to the Server on
a local switch , without any other System (it is a test environment) .
The Server is Connected to an IBM Shark with 2 Gigabit Fiberchannel . when
i cat a file locally on the Server to dev/null from /vicepa i get rates
about ~70 Mb/s .
So where so i loos so many Bandwidth ??
i use Openafs Server 1.2.8 on an AIX 4.3.3 System and the Clients are
Linux Redhat 8.0 also with Openafs 1.2.8.
a copy between the Systems itself with ttcp i get network rates >100 Mb/s
so Network is also not the problem ...
Sven
Nathan Ward <nward@esphion.com>
Sent by: openafs-info-admin@openafs.org
02/28/2003 02:40 PM
To: emoy@apple.com
cc: Michael Robokoff <mrobo@ahpcrc.org>, openafs
<openafs-info@openafs.org>
Subject: Re: [OpenAFS] Performance
This is for in-house developed stuff for black box network appliances,
we don't spend any time in configure, and about 20% in install.
I'll try building our entire tree in afs with 2000000 cache size and
-chunksize 18 a bit later on and let you know.
Nathan Ward
emoy@apple.com wrote:
> I recently did some compile test comparisons, and the numbers I got
> were:
>
> Local Disk NFS AFS
> 4:23 4:48 4:31 minutes
>
> This seems reasonable, as the actual compilation itself is >90% of
> the time (including running configure and install).
>
> A 4x slowdown does seem unreasonable, unless your AFS server is using
> a 2400 baud modem for its network ;-). Or unless your cache is so
> small that the files you create are always exceeding the cache size.
> ------------------------------------------------------------------------
> --
> Edward Moy
> Apple Computer, Inc.
> emoy@apple.com
>
> (This message is from me as a reader of this list, and not a statement
> from Apple.)
>
> On Thursday, February 27, 2003, at 03:41 PM, Nathan Ward wrote:
>
>> I'll do some tests with this today.
>>
>> Yes of course it will be slower, but it is unbearably slower. I'm
>> not sure about local disk stats, but perhaps a better comparison:
>> It takes ~4 times as long on AFS over NFS, last I checked.
>>
>> emoy@apple.com wrote:
>>
>>> Sorry for any confusion, but it was a 1 GB file I used. So
>>> performance was not up to the theoretical maximum, but I was not
>>> on a quiet network, and the server was on a different subnet.
>>> That is why one has to compare the performance with NFS directly,
>>> and not against the theoretical maximum (the server in the test
>>> was doing both NFS and AFS).
>>>
>>> Other than setting the cache size to 2000000 blocks, I used
>>> "-chucksize 18" to get the 256 KB chucksize. NFS write time was
>>> about 4:20 compared to 3:30 for 256 KB chucksize and 4:50 for the
>>> default 64 KB chucksize. System CPU time for NFS was about the
>>> same as AFS (around 20 seconds).
>>>
>>> As for compiling on AFS versus local disk, of course AFS would be
>>> slower. The files have to be read from the network at least once,
>>> and new/modified files written back. However, compiling on NFS,
>>> for example, will generally be poorer, though perhaps not
>>> strikingly so, depending on how much your kernel caches NFS. And
>>> as more users access files simultaneously, the performance of NFS
>>> will degrade faster than AFS.
>>>
>>> On Thursday, February 27, 2003, at 01:24 PM, Nathan Ward wrote:
>>>
>>>> Yes, playing with tuning the cache parameters is something thats
>>>> on my run queue.
>>>> What were your afs client settings you had when you did the 2gb
>>>> file in 3.5 min?
>>>>
>>>> My calculations say that 2gb in 3.5 minutes is ~10mb/s. A 100Mbit
>>>> network can do ~12.5MB/s of packets, so allow for headers and
>>>> control data and i'd say your are limited at this point by your
>>>> network.
>>>>
>>>> You say it beats NFS, whats the CPU usages for 2GB over OpenAFS
>>>> vs NFS? Context switches?
>>>>
>>>> My developers complain that code compiles are much much slower in
>>>> AFS than on the local disk, but actual file performance when they
>>>> aren't compiling is not-so-bad. I imagine this is because of the
>>>> CPU that AFS takes to get the files, so CPU usage is important >>>
>>>> here.
>>>>
>>>> emoy@apple.com wrote:
>>>>
>>>>> As I understand it, the cache helps reads primarily, though if
>>>>> you are doing random updates of a file, those also get cached,
>>>>> until you do the finally close, which then writes all to the
>>>>> server. That assumes the file fits in the cache. If not, then
>>>>> a write can force the cache to partially empty, so the latest
>>>>> write can be cached.
>>>>>
>>>>> But there seems to be some pathological behavior, because doing
>>>>> a sequential write should ideally not be affected that much by
>>>>> size of the cache. But in a test I did with a 1 GB file, with
>>>>> regular network traffic, it took almost 5 minute to write with
>>>>> with a 2 GB cache (about 10% slower than NFS in this case), but
>>>>> took almost 11 minutes with a 0.5 GB cache. The system CPU
>>>>> time rose from 20 seconds for the 2 GB cache to 517 seconds
>>>>> with the 0.5 GB cache.
>>>>>
>>>>> As for being slower than NFS in this test case, with such a
>>>>> large cache, tuning the cache parameters does help. Setting
>>>>> the chucksize to 256KB dropped the 2 GB cache speed to about
>>>>> 3.5 minutes, beating NFS.
>>>>>
>>>>> On Thursday, February 27, 2003, at 11:51 AM, Nathan Ward wrote:
>>>>>
>>>>>> Well I'd expect that it goes slower as your cache size is
>>>>>> exceeded as it then needs to start getting that data to the
>>>>>> server. Or is the cache for read operations only?
>>>>>>
>>>>>> I notice that there are around about the same number of
>>>>>> packets/sec as context switches/sec on my client machines. I
>>>>>> wonder if switches between userland and kernel could be to
>>>>>> blame... ? Who sends packets in OpenAFS, the userspace daemon
>>>>>> or the kernel?
>>>>>>
>>>>>> Nathan
>>>>>>
>>>>>> emoy@apple.com wrote:
>>>>>>
>>>>>>> Could the slowness you see with your dd write test be related
>>>>>>> to the cache exhaustion issue that I raised recently, when
>>>>>>> writing a file larger than your cache size. Your test
>>>>>>> writes a 1 GB file, so if your cache is smaller than this,
>>>>>>> you will see poor performance once your cache size is
exceeded.
>>>>>>>
>>>>>>> On Thursday, February 27, 2003, at 11:15 AM, Nathan Ward wrote:
>>>>>>>
>>>>>>>> I see pretty bad performance to tell you the truth.
>>>>>>>> I can read and write ~60mb/s directly to my raid array, but
>>>>>>>> when using OpenAFS (locally or remotely) to the same array,
>>>>>>>> I get around 6-10MB/s, I have seen up to 25MB/s over a
>>>>>>>> peice of 1000Mbps fibre. Client and Server are both dual
>>>>>>>> P3-1ghz with 1024mb ram. I notice the context switches on
>>>>>>>> the server at this time jump to ~10000/s, and on the client
>>>>>>>> ~40000/s. I imagine this is the source of my slowdown, but
>>>>>>>> I havn't had a chance to look into it.
>>>>>>>>
>>>>>>>> I'd be interested if anyone else has the same level of
>>>>>>>> context switches going on.
>>>>>>>>
>>>>>>>> This is while doing a large sequential write operation (dd
>>>>>>>> if=/dev/zero of=/afs/alb-nz.esphion.com/public/dd.out
>>>>>>>> bs=256k count=4096).
>>>>>>>>
>>>>>>>> Michael Robokoff wrote:
>>>>>>>>
>>>>>>>>> Does anyone have any Open AFS performance information they
>>>>>>>>> can share with me. I plan on doing a couple benchmarks and
>>>>>>>>> I would like to have some idea of what to expect.
>>>>>>>>
>
>
>
_______________________________________________
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info
--=_alternative 002501DEC1256CDB_=
Content-Type: text/html; charset="US-ASCII"
<br><font size=2 face="sans-serif">hy , </font>
<br>
<br><font size=2 face="sans-serif">i get much higher rates between AFS
Server and my Clients :</font>
<br>
<br><font size=2 face="Courier New">[root@xxx21 root]# bonnie++ -d /afs/test11a/
-s 2008 </font>
<br><font size=2 face="Courier New">Using uid:0, gid:0.</font>
<br><font size=2 face="Courier New">Writing with putc()...done</font>
<br><font size=2 face="Courier New">Writing intelligently...done</font>
<br><font size=2 face="Courier New">Rewriting...done</font>
<br><font size=2 face="Courier New">Reading with getc()...done</font>
<br><font size=2 face="Courier New">Reading intelligently...done</font>
<br><font size=2 face="Courier New">start 'em...done...done...done...</font>
<br><font size=2 face="Courier New">Create files in sequential order...done.</font>
<br><font size=2 face="Courier New">Stat files in sequential order...done.</font>
<br><font size=2 face="Courier New">Delete files in sequential order...done.</font>
<br><font size=2 face="Courier New">Create files in random order...done.</font>
<br><font size=2 face="Courier New">Stat files in random order...done.</font>
<br><font size=2 face="Courier New">Delete files in random order...done.</font>
<br><font size=2 face="Courier New">Version 1.03
------Sequential Output------ --Sequential Input- --Random-</font>
<br><font size=2 face="Courier New">
-Per Chr- --Block-- -Rewrite- -Per Chr-
--Block-- --Seeks--</font>
<br><font size=2 face="Courier New">Machine Size
K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP</font>
<br><font size=2 face="Courier New">xxx21.de 2008M
9584 53 13950 33 3404 10 4685 18
5843 4 903.6 7</font>
<br><font size=2 face="Courier New">
------Sequential Create------ --------Random
Create--------</font>
<br><font size=2 face="Courier New">
-Create-- --Read--- -Delete-- -Create--
--Read--- -Delete--</font>
<br><font size=2 face="Courier New">
files /sec %CP /sec %CP /sec %CP /sec
%CP /sec %CP /sec %CP</font>
<br><font size=2 face="Courier New">
16 251 11 2401 25
480 11 345 15 1472 26 564 10</font>
<br><font size=2 face="Courier New">xxx21.de,2008M,9584,53,13950,33,3404,10,4685,18,5843,4,903.6,7,16,251,11,2401,25,480,11,345,15,1472,26,564,10</font>
<br>
<br><font size=2 face="sans-serif">i have identically results from 2 Clients
running bonnie at the same time to 1 Server . i only changed the cache
to memache , without any other changes to the clients or server .</font>
<br><font size=2 face="sans-serif">the wired thing for me is , my Server
is not going over 25% CPU Utilization when i run this test . and also the
Clients are not busy .</font>
<br>
<br><font size=2 face="sans-serif">the clients are connected with Gigabyte
Ethernet (Fiber) to the Server on a local switch , without any other
System (it is a test environment) .</font>
<br><font size=2 face="sans-serif">The Server is Connected to an IBM Shark
with 2 Gigabit Fiberchannel . when i cat a file locally on the Server to
dev/null from /vicepa i get rates about ~70 Mb/s .</font>
<br><font size=2 face="sans-serif">So where so i loos so many Bandwidth
?? </font>
<br>
<br><font size=2 face="sans-serif">i use Openafs Server 1.2.8 on an AIX
4.3.3 System and the Clients are Linux Redhat 8.0 also with Openafs 1.2.8.</font>
<br><font size=2 face="sans-serif">a copy between the Systems itself with
ttcp i get network rates >100 Mb/s so Network is also not the problem
...</font>
<br>
<br><font size=2 face="sans-serif">Sven</font>
<br>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td>
<td><font size=1 face="sans-serif"><b>Nathan Ward <nward@esphion.com></b></font>
<br><font size=1 face="sans-serif">Sent by: openafs-info-admin@openafs.org</font>
<p><font size=1 face="sans-serif">02/28/2003 02:40 PM</font>
<td><font size=1 face="Arial"> </font>
<br><font size=1 face="sans-serif"> To:
emoy@apple.com</font>
<br><font size=1 face="sans-serif"> cc:
Michael Robokoff <mrobo@ahpcrc.org>,
openafs <openafs-info@openafs.org></font>
<br><font size=1 face="sans-serif"> Subject:
Re: [OpenAFS] Performance</font></table>
<br>
<br>
<br><font size=2><tt>This is for in-house developed stuff for black box
network appliances, <br>
we don't spend any time in configure, and about 20% in install.<br>
<br>
I'll try building our entire tree in afs with 2000000 cache size and <br>
-chunksize 18 a bit later on and let you know.<br>
<br>
Nathan Ward<br>
<br>
emoy@apple.com wrote:<br>
<br>
> I recently did some compile test comparisons, and the numbers I got
<br>
> were:<br>
><br>
> Local Disk NFS AFS<br>
> 4:23 4:48 4:31 minutes<br>
><br>
> This seems reasonable, as the actual compilation itself is >90%
of <br>
> the time (including running configure and install).<br>
><br>
> A 4x slowdown does seem unreasonable, unless your AFS server is using
<br>
> a 2400 baud modem for its network ;-). Or unless your
cache is so <br>
> small that the files you create are always exceeding the cache
size.<br>
> ------------------------------------------------------------------------
<br>
> -- <br>
> Edward Moy<br>
> Apple Computer, Inc.<br>
> emoy@apple.com<br>
><br>
> (This message is from me as a reader of this list, and not a statement<br>
> from Apple.)<br>
><br>
> On Thursday, February 27, 2003, at 03:41 PM, Nathan Ward wrote:<br>
><br>
>> I'll do some tests with this today.<br>
>><br>
>> Yes of course it will be slower, but it is unbearably slower.
I'm <br>
>> not sure about local disk stats, but perhaps a better comparison:<br>
>> It takes ~4 times as long on AFS over NFS, last I checked.<br>
>><br>
>> emoy@apple.com wrote:<br>
>><br>
>>> Sorry for any confusion, but it was a 1 GB file I used. So
<br>
>>> performance was not up to the theoretical maximum, but
I was not <br>
>>> on a quiet network, and the server was on a different
subnet. <br>
>>> That is why one has to compare the performance
with NFS directly, <br>
>>> and not against the theoretical maximum (the server
in the test <br>
>>> was doing both NFS and AFS).<br>
>>><br>
>>> Other than setting the cache size to 2000000 blocks, I used
<br>
>>> "-chucksize 18" to get the 256 KB chucksize.
NFS write time was <br>
>>> about 4:20 compared to 3:30 for 256 KB chucksize and
4:50 for the <br>
>>> default 64 KB chucksize. System CPU time for NFS
was about the <br>
>>> same as AFS (around 20 seconds).<br>
>>><br>
>>> As for compiling on AFS versus local disk, of course AFS would
be <br>
>>> slower. The files have to be read from the network at
least once, <br>
>>> and new/modified files written back. However,
compiling on NFS, <br>
>>> for example, will generally be poorer, though perhaps
not <br>
>>> strikingly so, depending on how much your kernel caches
NFS. And <br>
>>> as more users access files simultaneously, the
performance of NFS <br>
>>> will degrade faster than AFS.<br>
>>><br>
>>> On Thursday, February 27, 2003, at 01:24 PM, Nathan
Ward wrote:<br>
>>><br>
>>>> Yes, playing with tuning the cache parameters is something
thats <br>
>>>> on my run queue.<br>
>>>> What were your afs client settings you had when you did
the 2gb <br>
>>>> file in 3.5 min?<br>
>>>><br>
>>>> My calculations say that 2gb in 3.5 minutes is ~10mb/s.
A 100Mbit <br>
>>>> network can do ~12.5MB/s of packets, so allow for headers
and <br>
>>>> control data and i'd say your are limited at this
point by your <br>
>>>> network.<br>
>>>><br>
>>>> You say it beats NFS, whats the CPU usages for 2GB over
OpenAFS <br>
>>>> vs NFS? Context switches?<br>
>>>><br>
>>>> My developers complain that code compiles are much much
slower in <br>
>>>> AFS than on the local disk, but actual file performance
when they <br>
>>>> aren't compiling is not-so-bad. I imagine this is
because of the <br>
>>>> CPU that AFS takes to get the files, so CPU usage
is important >>> <br>
>>>> here.<br>
>>>><br>
>>>> emoy@apple.com wrote:<br>
>>>><br>
>>>>> As I understand it, the cache helps reads primarily,
though if <br>
>>>>> you are doing random updates of a file,
those also get cached, <br>
>>>>> until you do the finally close,
which then writes all to the <br>
>>>>> server. That assumes the file fits
in the cache. If not, then <br>
>>>>> a write can force the cache to partially
empty, so the latest <br>
>>>>> write can be cached.<br>
>>>>><br>
>>>>> But there seems to be some pathological behavior,
because doing <br>
>>>>> a sequential write should ideally not
be affected that much by <br>
>>>>> size of the cache. But in a test
I did with a 1 GB file, with <br>
>>>>> regular network traffic, it took almost
5 minute to write with <br>
>>>>> with a 2 GB cache (about 10% slower
than NFS in this case), but <br>
>>>>> took almost 11 minutes with a 0.5
GB cache. The system CPU <br>
>>>>> time rose from 20 seconds for the 2 GB
cache to 517 seconds <br>
>>>>> with the 0.5 GB cache.<br>
>>>>><br>
>>>>> As for being slower than NFS in this test case, with
such a <br>
>>>>> large cache, tuning the cache parameters
does help. Setting <br>
>>>>> the chucksize to 256KB dropped the
2 GB cache speed to about <br>
>>>>> 3.5 minutes, beating NFS.<br>
>>>>><br>
>>>>> On Thursday, February 27, 2003, at 11:51 AM,
Nathan Ward wrote:<br>
>>>>><br>
>>>>>> Well I'd expect that it goes slower as your cache
size is <br>
>>>>>> exceeded as it then needs to start
getting that data to the <br>
>>>>>> server. Or is the cache for read
operations only?<br>
>>>>>><br>
>>>>>> I notice that there are around about the same
number of <br>
>>>>>> packets/sec as context switches/sec
on my client machines. I <br>
>>>>>> wonder if switches between userland and
kernel could be to <br>
>>>>>> blame... ? Who sends packets in OpenAFS,
the userspace daemon <br>
>>>>>> or the kernel?<br>
>>>>>><br>
>>>>>> Nathan<br>
>>>>>><br>
>>>>>> emoy@apple.com wrote:<br>
>>>>>><br>
>>>>>>> Could the slowness you see with your dd write
test be related <br>
>>>>>>> to the cache exhaustion issue
that I raised recently, when <br>
>>>>>>> writing a file larger than
your cache size. Your test <br>
>>>>>>> writes a 1 GB file, so if your
cache is smaller than this, <br>
>>>>>>> you will see poor performance once
your cache size is exceeded.<br>
>>>>>>><br>
>>>>>>> On Thursday, February 27, 2003, at 11:15 AM,
Nathan Ward wrote:<br>
>>>>>>><br>
>>>>>>>> I see pretty bad performance to tell you
the truth.<br>
>>>>>>>> I can read and write ~60mb/s directly
to my raid array, but <br>
>>>>>>>> when using OpenAFS
(locally or remotely) to the same array, <br>
>>>>>>>> I get around 6-10MB/s, I
have seen up to 25MB/s over a <br>
>>>>>>>> peice of 1000Mbps fibre.
Client and Server are both dual <br>
>>>>>>>> P3-1ghz with 1024mb ram. I notice
the context switches on <br>
>>>>>>>> the server at this time jump
to ~10000/s, and on the client <br>
>>>>>>>> ~40000/s. I imagine this is
the source of my slowdown, but <br>
>>>>>>>> I havn't had a chance to look
into it.<br>
>>>>>>>><br>
>>>>>>>> I'd be interested if anyone else has the
same level of <br>
>>>>>>>> context switches going on.<br>
>>>>>>>><br>
>>>>>>>> This is while doing a large sequential
write operation (dd <br>
>>>>>>>> if=/dev/zero of=/afs/alb-nz.esphion.com/public/dd.out
<br>
>>>>>>>> bs=256k count=4096).<br>
>>>>>>>><br>
>>>>>>>> Michael Robokoff wrote:<br>
>>>>>>>><br>
>>>>>>>>> Does anyone have any Open AFS performance
information they <br>
>>>>>>>>> can share with
me. I plan on doing a couple benchmarks and <br>
>>>>>>>>> I would like to have
some idea of what to expect.<br>
>>>>>>>><br>
><br>
><br>
><br>
<br>
<br>
_______________________________________________<br>
OpenAFS-info mailing list<br>
OpenAFS-info@openafs.org<br>
https://lists.openafs.org/mailman/listinfo/openafs-info<br>
</tt></font>
<br>
--=_alternative 002501DEC1256CDB_=--