[OpenAFS] Performance

[emoy@apple.com]

Sorry for any confusion, but it was a 1 GB file I used.  So performance  
was not up to the theoretical maximum, but I was not on a quiet  
network, and the server was on a different subnet.  That is why one has  
to compare the performance with NFS directly, and not against the  
theoretical maximum (the server in the test was doing both NFS and AFS).

Other than setting the cache size to 2000000 blocks, I used "-chucksize  
18" to get the 256 KB chucksize.  NFS write time was about 4:20  
compared to 3:30 for 256 KB chucksize and 4:50 for the default 64 KB  
chucksize.  System CPU time for NFS was about the same as AFS (around  
20 seconds).

As for compiling on AFS versus local disk, of course AFS would be  
slower.  The files have to be read from the network at least once, and  
new/modified files written back.  However, compiling on NFS, for  
example, will generally be poorer, though perhaps not strikingly so,  
depending on how much your kernel caches NFS.  And as more users access  
files simultaneously, the performance of NFS will degrade faster than  
AFS.
------------------------------------------------------------------------ 
--
Edward Moy
Apple Computer, Inc.
emoy@apple.com

(This message is from me as a reader of this list, and not a statement
from Apple.)

On Thursday, February 27, 2003, at 01:24  PM, Nathan Ward wrote:

> Yes, playing with tuning the cache parameters is something thats on my  
> run queue.
> What were your afs client settings you had when you did the 2gb file  
> in 3.5 min?
>
> My calculations say that 2gb in 3.5 minutes is ~10mb/s. A 100Mbit  
> network can do ~12.5MB/s of packets, so allow for headers and control  
> data and i'd say your are limited at this point by your network.
>
> You say it beats NFS, whats the CPU usages for 2GB over OpenAFS vs  
> NFS? Context switches?
>
> My developers complain that code compiles are much much slower in AFS  
> than on the local disk, but actual file performance when they aren't  
> compiling is not-so-bad. I imagine this is because of the CPU that AFS  
> takes to get the files, so CPU usage is important here.
>
> emoy@apple.com wrote:
>
>> As I understand it, the cache helps reads primarily, though if you  
>> are  doing random updates of a file, those also get cached, until you  
>> do the  finally close, which then writes all to the server.  That  
>> assumes the  file fits in the cache.  If not, then a write can force  
>> the cache to  partially empty, so the latest write can be cached.
>>
>> But there seems to be some pathological behavior, because doing a   
>> sequential write should ideally not be affected that much by size of   
>> the cache.  But in a test I did with a 1 GB file, with regular  
>> network  traffic, it took almost 5 minute to write with with a 2 GB  
>> cache (about  10% slower than NFS in this case), but took almost 11  
>> minutes with a  0.5 GB cache.  The system CPU time rose from 20  
>> seconds for the 2 GB  cache to 517 seconds with the 0.5 GB cache.
>>
>> As for being slower than NFS in this test case, with such a large   
>> cache, tuning the cache parameters does help.  Setting the chucksize  
>> to  256KB dropped the 2 GB cache speed to about 3.5 minutes, beating  
>> NFS.
>>
>> On Thursday, February 27, 2003, at 11:51  AM, Nathan Ward wrote:
>>
>>> Well I'd expect that it goes slower as your cache size is exceeded  
>>> as  it then needs to start getting that data to the server. Or is  
>>> the  cache for read operations only?
>>>
>>> I notice that there are around about the same number of packets/sec  
>>> as  context switches/sec on my client machines. I wonder if switches  
>>>  between userland and kernel could be to blame... ? Who sends  
>>> packets  in OpenAFS, the userspace daemon or the kernel?
>>>
>>> Nathan
>>>
>>> emoy@apple.com wrote:
>>>
>>>> Could the slowness you see with your dd write test be related to  
>>>> the   cache exhaustion issue that I raised recently, when writing a  
>>>> file   larger than your cache size.  Your test writes a 1 GB file,  
>>>> so if  your  cache is smaller than this, you will see poor  
>>>> performance once  your  cache size is exceeded.
>>>>
>>>> On Thursday, February 27, 2003, at 11:15  AM, Nathan Ward wrote:
>>>>
>>>>> I see pretty bad performance to tell you the truth.
>>>>> I can read and write ~60mb/s directly to my raid array, but when   
>>>>> using  OpenAFS (locally or remotely) to the same array, I get  
>>>>> around   6-10MB/s, I have seen up to 25MB/s over a peice of  
>>>>> 1000Mbps fibre.   Client and Server are both dual P3-1ghz with  
>>>>> 1024mb ram. I notice  the  context switches on the server at this  
>>>>> time jump to ~10000/s,  and on  the client ~40000/s. I imagine  
>>>>> this is the source of my  slowdown, but  I havn't had a chance to  
>>>>> look into it.
>>>>>
>>>>> I'd be interested if anyone else has the same level of context    
>>>>> switches going on.
>>>>>
>>>>> This is while doing a large sequential write operation (dd    
>>>>> if=/dev/zero of=/afs/alb-nz.esphion.com/public/dd.out bs=256k    
>>>>> count=4096).
>>>>>
>>>>> Michael Robokoff wrote:
>>>>>
>>>>>> Does anyone have any Open AFS performance information they can   
>>>>>> share  with me. I plan on doing a couple benchmarks and I would   
>>>>>> like to have  some idea of what to expect.
>