[OpenAFS] Performance

Nathan Ward nward@esphion.com
Fri, 28 Feb 2003 13:40:52 +1300


This is for in-house developed stuff for black box network appliances, 
we don't spend any time in configure, and about 20% in install.

I'll try building our entire tree in afs with 2000000 cache size and 
-chunksize 18 a bit later on and let you know.

Nathan Ward

emoy@apple.com wrote:

> I recently did some compile test comparisons, and the numbers I got  
> were:
>
>     Local Disk   NFS   AFS
>         4:23    4:48  4:31  minutes
>
> This seems reasonable, as the actual compilation itself is >90% of 
> the  time (including running configure and install).
>
> A 4x slowdown does seem unreasonable, unless your AFS server is using 
> a  2400 baud modem for its network ;-).  Or unless your cache is so 
> small  that the files you create are always exceeding the cache size.
> ------------------------------------------------------------------------ 
> -- 
> Edward Moy
> Apple Computer, Inc.
> emoy@apple.com
>
> (This message is from me as a reader of this list, and not a statement
> from Apple.)
>
> On Thursday, February 27, 2003, at 03:41  PM, Nathan Ward wrote:
>
>> I'll do some tests with this today.
>>
>> Yes of course it will be slower, but it is unbearably slower. I'm 
>> not  sure about local disk stats, but perhaps a better comparison:
>> It takes ~4 times as long on AFS over NFS, last I checked.
>>
>> emoy@apple.com wrote:
>>
>>> Sorry for any confusion, but it was a 1 GB file I used.  So  
>>> performance  was not up to the theoretical maximum, but I was not 
>>> on  a quiet  network, and the server was on a different subnet.  
>>> That is  why one has  to compare the performance with NFS directly, 
>>> and not  against the  theoretical maximum (the server in the test 
>>> was doing  both NFS and AFS).
>>>
>>> Other than setting the cache size to 2000000 blocks, I used  
>>> "-chucksize  18" to get the 256 KB chucksize.  NFS write time was  
>>> about 4:20  compared to 3:30 for 256 KB chucksize and 4:50 for the  
>>> default 64 KB  chucksize.  System CPU time for NFS was about the 
>>> same  as AFS (around  20 seconds).
>>>
>>> As for compiling on AFS versus local disk, of course AFS would be   
>>> slower.  The files have to be read from the network at least once,  
>>> and  new/modified files written back.  However, compiling on NFS, 
>>> for   example, will generally be poorer, though perhaps not 
>>> strikingly so,   depending on how much your kernel caches NFS.  And 
>>> as more users  access  files simultaneously, the performance of NFS 
>>> will degrade  faster than  AFS.
>>>
>>> On Thursday, February 27, 2003, at 01:24  PM, Nathan Ward wrote:
>>>
>>>> Yes, playing with tuning the cache parameters is something thats 
>>>> on  my  run queue.
>>>> What were your afs client settings you had when you did the 2gb 
>>>> file   in 3.5 min?
>>>>
>>>> My calculations say that 2gb in 3.5 minutes is ~10mb/s. A 100Mbit   
>>>> network can do ~12.5MB/s of packets, so allow for headers and  
>>>> control  data and i'd say your are limited at this point by your  
>>>> network.
>>>>
>>>> You say it beats NFS, whats the CPU usages for 2GB over OpenAFS 
>>>> vs   NFS? Context switches?
>>>>
>>>> My developers complain that code compiles are much much slower in  
>>>> AFS  than on the local disk, but actual file performance when they  
>>>> aren't  compiling is not-so-bad. I imagine this is because of the  
>>>> CPU that AFS  takes to get the files, so CPU usage is important >>> 
>>>> here.
>>>>
>>>> emoy@apple.com wrote:
>>>>
>>>>> As I understand it, the cache helps reads primarily, though if 
>>>>> you   are  doing random updates of a file, those also get cached, 
>>>>> until  you  do the  finally close, which then writes all to the 
>>>>> server.   That  assumes the  file fits in the cache.  If not, then 
>>>>> a write  can force  the cache to  partially empty, so the latest 
>>>>> write can  be cached.
>>>>>
>>>>> But there seems to be some pathological behavior, because doing 
>>>>> a    sequential write should ideally not be affected that much by 
>>>>> size  of   the cache.  But in a test I did with a 1 GB file, with 
>>>>> regular   network  traffic, it took almost 5 minute to write with 
>>>>> with a 2  GB  cache (about  10% slower than NFS in this case), but 
>>>>> took  almost 11  minutes with a  0.5 GB cache.  The system CPU 
>>>>> time rose  from 20  seconds for the 2 GB  cache to 517 seconds 
>>>>> with the 0.5 GB  cache.
>>>>>
>>>>> As for being slower than NFS in this test case, with such a 
>>>>> large    cache, tuning the cache parameters does help.  Setting 
>>>>> the  chucksize  to  256KB dropped the 2 GB cache speed to about 
>>>>> 3.5  minutes, beating  NFS.
>>>>>
>>>>> On Thursday, February 27, 2003, at 11:51  AM, Nathan Ward wrote:
>>>>>
>>>>>> Well I'd expect that it goes slower as your cache size is 
>>>>>> exceeded   as  it then needs to start getting that data to the 
>>>>>> server. Or is   the  cache for read operations only?
>>>>>>
>>>>>> I notice that there are around about the same number of  
>>>>>> packets/sec  as  context switches/sec on my client machines. I  
>>>>>> wonder if switches   between userland and kernel could be to  
>>>>>> blame... ? Who sends  packets  in OpenAFS, the userspace daemon 
>>>>>> or  the kernel?
>>>>>>
>>>>>> Nathan
>>>>>>
>>>>>> emoy@apple.com wrote:
>>>>>>
>>>>>>> Could the slowness you see with your dd write test be related 
>>>>>>> to   the   cache exhaustion issue that I raised recently, when 
>>>>>>> writing  a  file   larger than your cache size.  Your test 
>>>>>>> writes a 1 GB  file,  so if  your  cache is smaller than this, 
>>>>>>> you will see poor   performance once  your  cache size is exceeded.
>>>>>>>
>>>>>>> On Thursday, February 27, 2003, at 11:15  AM, Nathan Ward wrote:
>>>>>>>
>>>>>>>> I see pretty bad performance to tell you the truth.
>>>>>>>> I can read and write ~60mb/s directly to my raid array, but 
>>>>>>>> when    using  OpenAFS (locally or remotely) to the same array, 
>>>>>>>> I get   around   6-10MB/s, I have seen up to 25MB/s over a 
>>>>>>>> peice of   1000Mbps fibre.   Client and Server are both dual 
>>>>>>>> P3-1ghz with   1024mb ram. I notice  the  context switches on 
>>>>>>>> the server at  this  time jump to ~10000/s,  and on  the client 
>>>>>>>> ~40000/s. I  imagine  this is the source of my  slowdown, but  
>>>>>>>> I havn't had a  chance to  look into it.
>>>>>>>>
>>>>>>>> I'd be interested if anyone else has the same level of 
>>>>>>>> context     switches going on.
>>>>>>>>
>>>>>>>> This is while doing a large sequential write operation (dd     
>>>>>>>> if=/dev/zero of=/afs/alb-nz.esphion.com/public/dd.out 
>>>>>>>> bs=256k     count=4096).
>>>>>>>>
>>>>>>>> Michael Robokoff wrote:
>>>>>>>>
>>>>>>>>> Does anyone have any Open AFS performance information they 
>>>>>>>>> can    share  with me. I plan on doing a couple benchmarks and 
>>>>>>>>> I  would   like to have  some idea of what to expect.
>>>>>>>>
>
>
>