[OpenAFS] Performance
Nathan Ward
nward@esphion.com
Fri, 28 Feb 2003 13:40:52 +1300
This is for in-house developed stuff for black box network appliances,
we don't spend any time in configure, and about 20% in install.
I'll try building our entire tree in afs with 2000000 cache size and
-chunksize 18 a bit later on and let you know.
Nathan Ward
emoy@apple.com wrote:
> I recently did some compile test comparisons, and the numbers I got
> were:
>
> Local Disk NFS AFS
> 4:23 4:48 4:31 minutes
>
> This seems reasonable, as the actual compilation itself is >90% of
> the time (including running configure and install).
>
> A 4x slowdown does seem unreasonable, unless your AFS server is using
> a 2400 baud modem for its network ;-). Or unless your cache is so
> small that the files you create are always exceeding the cache size.
> ------------------------------------------------------------------------
> --
> Edward Moy
> Apple Computer, Inc.
> emoy@apple.com
>
> (This message is from me as a reader of this list, and not a statement
> from Apple.)
>
> On Thursday, February 27, 2003, at 03:41 PM, Nathan Ward wrote:
>
>> I'll do some tests with this today.
>>
>> Yes of course it will be slower, but it is unbearably slower. I'm
>> not sure about local disk stats, but perhaps a better comparison:
>> It takes ~4 times as long on AFS over NFS, last I checked.
>>
>> emoy@apple.com wrote:
>>
>>> Sorry for any confusion, but it was a 1 GB file I used. So
>>> performance was not up to the theoretical maximum, but I was not
>>> on a quiet network, and the server was on a different subnet.
>>> That is why one has to compare the performance with NFS directly,
>>> and not against the theoretical maximum (the server in the test
>>> was doing both NFS and AFS).
>>>
>>> Other than setting the cache size to 2000000 blocks, I used
>>> "-chucksize 18" to get the 256 KB chucksize. NFS write time was
>>> about 4:20 compared to 3:30 for 256 KB chucksize and 4:50 for the
>>> default 64 KB chucksize. System CPU time for NFS was about the
>>> same as AFS (around 20 seconds).
>>>
>>> As for compiling on AFS versus local disk, of course AFS would be
>>> slower. The files have to be read from the network at least once,
>>> and new/modified files written back. However, compiling on NFS,
>>> for example, will generally be poorer, though perhaps not
>>> strikingly so, depending on how much your kernel caches NFS. And
>>> as more users access files simultaneously, the performance of NFS
>>> will degrade faster than AFS.
>>>
>>> On Thursday, February 27, 2003, at 01:24 PM, Nathan Ward wrote:
>>>
>>>> Yes, playing with tuning the cache parameters is something thats
>>>> on my run queue.
>>>> What were your afs client settings you had when you did the 2gb
>>>> file in 3.5 min?
>>>>
>>>> My calculations say that 2gb in 3.5 minutes is ~10mb/s. A 100Mbit
>>>> network can do ~12.5MB/s of packets, so allow for headers and
>>>> control data and i'd say your are limited at this point by your
>>>> network.
>>>>
>>>> You say it beats NFS, whats the CPU usages for 2GB over OpenAFS
>>>> vs NFS? Context switches?
>>>>
>>>> My developers complain that code compiles are much much slower in
>>>> AFS than on the local disk, but actual file performance when they
>>>> aren't compiling is not-so-bad. I imagine this is because of the
>>>> CPU that AFS takes to get the files, so CPU usage is important >>>
>>>> here.
>>>>
>>>> emoy@apple.com wrote:
>>>>
>>>>> As I understand it, the cache helps reads primarily, though if
>>>>> you are doing random updates of a file, those also get cached,
>>>>> until you do the finally close, which then writes all to the
>>>>> server. That assumes the file fits in the cache. If not, then
>>>>> a write can force the cache to partially empty, so the latest
>>>>> write can be cached.
>>>>>
>>>>> But there seems to be some pathological behavior, because doing
>>>>> a sequential write should ideally not be affected that much by
>>>>> size of the cache. But in a test I did with a 1 GB file, with
>>>>> regular network traffic, it took almost 5 minute to write with
>>>>> with a 2 GB cache (about 10% slower than NFS in this case), but
>>>>> took almost 11 minutes with a 0.5 GB cache. The system CPU
>>>>> time rose from 20 seconds for the 2 GB cache to 517 seconds
>>>>> with the 0.5 GB cache.
>>>>>
>>>>> As for being slower than NFS in this test case, with such a
>>>>> large cache, tuning the cache parameters does help. Setting
>>>>> the chucksize to 256KB dropped the 2 GB cache speed to about
>>>>> 3.5 minutes, beating NFS.
>>>>>
>>>>> On Thursday, February 27, 2003, at 11:51 AM, Nathan Ward wrote:
>>>>>
>>>>>> Well I'd expect that it goes slower as your cache size is
>>>>>> exceeded as it then needs to start getting that data to the
>>>>>> server. Or is the cache for read operations only?
>>>>>>
>>>>>> I notice that there are around about the same number of
>>>>>> packets/sec as context switches/sec on my client machines. I
>>>>>> wonder if switches between userland and kernel could be to
>>>>>> blame... ? Who sends packets in OpenAFS, the userspace daemon
>>>>>> or the kernel?
>>>>>>
>>>>>> Nathan
>>>>>>
>>>>>> emoy@apple.com wrote:
>>>>>>
>>>>>>> Could the slowness you see with your dd write test be related
>>>>>>> to the cache exhaustion issue that I raised recently, when
>>>>>>> writing a file larger than your cache size. Your test
>>>>>>> writes a 1 GB file, so if your cache is smaller than this,
>>>>>>> you will see poor performance once your cache size is exceeded.
>>>>>>>
>>>>>>> On Thursday, February 27, 2003, at 11:15 AM, Nathan Ward wrote:
>>>>>>>
>>>>>>>> I see pretty bad performance to tell you the truth.
>>>>>>>> I can read and write ~60mb/s directly to my raid array, but
>>>>>>>> when using OpenAFS (locally or remotely) to the same array,
>>>>>>>> I get around 6-10MB/s, I have seen up to 25MB/s over a
>>>>>>>> peice of 1000Mbps fibre. Client and Server are both dual
>>>>>>>> P3-1ghz with 1024mb ram. I notice the context switches on
>>>>>>>> the server at this time jump to ~10000/s, and on the client
>>>>>>>> ~40000/s. I imagine this is the source of my slowdown, but
>>>>>>>> I havn't had a chance to look into it.
>>>>>>>>
>>>>>>>> I'd be interested if anyone else has the same level of
>>>>>>>> context switches going on.
>>>>>>>>
>>>>>>>> This is while doing a large sequential write operation (dd
>>>>>>>> if=/dev/zero of=/afs/alb-nz.esphion.com/public/dd.out
>>>>>>>> bs=256k count=4096).
>>>>>>>>
>>>>>>>> Michael Robokoff wrote:
>>>>>>>>
>>>>>>>>> Does anyone have any Open AFS performance information they
>>>>>>>>> can share with me. I plan on doing a couple benchmarks and
>>>>>>>>> I would like to have some idea of what to expect.
>>>>>>>>
>
>
>