[OpenAFS] Performance
emoy@apple.com
emoy@apple.com
Thu, 27 Feb 2003 16:00:24 -0800
I recently did some compile test comparisons, and the numbers I got
were:
Local Disk NFS AFS
4:23 4:48 4:31 minutes
This seems reasonable, as the actual compilation itself is >90% of the
time (including running configure and install).
A 4x slowdown does seem unreasonable, unless your AFS server is using a
2400 baud modem for its network ;-). Or unless your cache is so small
that the files you create are always exceeding the cache size.
------------------------------------------------------------------------
--
Edward Moy
Apple Computer, Inc.
emoy@apple.com
(This message is from me as a reader of this list, and not a statement
from Apple.)
On Thursday, February 27, 2003, at 03:41 PM, Nathan Ward wrote:
> I'll do some tests with this today.
>
> Yes of course it will be slower, but it is unbearably slower. I'm not
> sure about local disk stats, but perhaps a better comparison:
> It takes ~4 times as long on AFS over NFS, last I checked.
>
> emoy@apple.com wrote:
>
>> Sorry for any confusion, but it was a 1 GB file I used. So
>> performance was not up to the theoretical maximum, but I was not on
>> a quiet network, and the server was on a different subnet. That is
>> why one has to compare the performance with NFS directly, and not
>> against the theoretical maximum (the server in the test was doing
>> both NFS and AFS).
>>
>> Other than setting the cache size to 2000000 blocks, I used
>> "-chucksize 18" to get the 256 KB chucksize. NFS write time was
>> about 4:20 compared to 3:30 for 256 KB chucksize and 4:50 for the
>> default 64 KB chucksize. System CPU time for NFS was about the same
>> as AFS (around 20 seconds).
>>
>> As for compiling on AFS versus local disk, of course AFS would be
>> slower. The files have to be read from the network at least once,
>> and new/modified files written back. However, compiling on NFS, for
>> example, will generally be poorer, though perhaps not strikingly so,
>> depending on how much your kernel caches NFS. And as more users
>> access files simultaneously, the performance of NFS will degrade
>> faster than AFS.
>>
>> On Thursday, February 27, 2003, at 01:24 PM, Nathan Ward wrote:
>>
>>> Yes, playing with tuning the cache parameters is something thats on
>>> my run queue.
>>> What were your afs client settings you had when you did the 2gb file
>>> in 3.5 min?
>>>
>>> My calculations say that 2gb in 3.5 minutes is ~10mb/s. A 100Mbit
>>> network can do ~12.5MB/s of packets, so allow for headers and
>>> control data and i'd say your are limited at this point by your
>>> network.
>>>
>>> You say it beats NFS, whats the CPU usages for 2GB over OpenAFS vs
>>> NFS? Context switches?
>>>
>>> My developers complain that code compiles are much much slower in
>>> AFS than on the local disk, but actual file performance when they
>>> aren't compiling is not-so-bad. I imagine this is because of the
>>> CPU that AFS takes to get the files, so CPU usage is important >>> here.
>>>
>>> emoy@apple.com wrote:
>>>
>>>> As I understand it, the cache helps reads primarily, though if you
>>>> are doing random updates of a file, those also get cached, until
>>>> you do the finally close, which then writes all to the server.
>>>> That assumes the file fits in the cache. If not, then a write
>>>> can force the cache to partially empty, so the latest write can
>>>> be cached.
>>>>
>>>> But there seems to be some pathological behavior, because doing a
>>>> sequential write should ideally not be affected that much by size
>>>> of the cache. But in a test I did with a 1 GB file, with regular
>>>> network traffic, it took almost 5 minute to write with with a 2
>>>> GB cache (about 10% slower than NFS in this case), but took
>>>> almost 11 minutes with a 0.5 GB cache. The system CPU time rose
>>>> from 20 seconds for the 2 GB cache to 517 seconds with the 0.5 GB
>>>> cache.
>>>>
>>>> As for being slower than NFS in this test case, with such a large
>>>> cache, tuning the cache parameters does help. Setting the
>>>> chucksize to 256KB dropped the 2 GB cache speed to about 3.5
>>>> minutes, beating NFS.
>>>>
>>>> On Thursday, February 27, 2003, at 11:51 AM, Nathan Ward wrote:
>>>>
>>>>> Well I'd expect that it goes slower as your cache size is exceeded
>>>>> as it then needs to start getting that data to the server. Or is
>>>>> the cache for read operations only?
>>>>>
>>>>> I notice that there are around about the same number of
>>>>> packets/sec as context switches/sec on my client machines. I
>>>>> wonder if switches between userland and kernel could be to
>>>>> blame... ? Who sends packets in OpenAFS, the userspace daemon or
>>>>> the kernel?
>>>>>
>>>>> Nathan
>>>>>
>>>>> emoy@apple.com wrote:
>>>>>
>>>>>> Could the slowness you see with your dd write test be related to
>>>>>> the cache exhaustion issue that I raised recently, when writing
>>>>>> a file larger than your cache size. Your test writes a 1 GB
>>>>>> file, so if your cache is smaller than this, you will see poor
>>>>>> performance once your cache size is exceeded.
>>>>>>
>>>>>> On Thursday, February 27, 2003, at 11:15 AM, Nathan Ward wrote:
>>>>>>
>>>>>>> I see pretty bad performance to tell you the truth.
>>>>>>> I can read and write ~60mb/s directly to my raid array, but when
>>>>>>> using OpenAFS (locally or remotely) to the same array, I get
>>>>>>> around 6-10MB/s, I have seen up to 25MB/s over a peice of
>>>>>>> 1000Mbps fibre. Client and Server are both dual P3-1ghz with
>>>>>>> 1024mb ram. I notice the context switches on the server at
>>>>>>> this time jump to ~10000/s, and on the client ~40000/s. I
>>>>>>> imagine this is the source of my slowdown, but I havn't had a
>>>>>>> chance to look into it.
>>>>>>>
>>>>>>> I'd be interested if anyone else has the same level of context
>>>>>>> switches going on.
>>>>>>>
>>>>>>> This is while doing a large sequential write operation (dd
>>>>>>> if=/dev/zero of=/afs/alb-nz.esphion.com/public/dd.out bs=256k
>>>>>>> count=4096).
>>>>>>>
>>>>>>> Michael Robokoff wrote:
>>>>>>>
>>>>>>>> Does anyone have any Open AFS performance information they can
>>>>>>>> share with me. I plan on doing a couple benchmarks and I
>>>>>>>> would like to have some idea of what to expect.