[OpenAFS] poor out of cache behavior on writing

[Edward Moy]

On Monday, February 17, 2003, at 12:22  PM, Paul Blackburn wrote:

> Derrick J Brashear wrote:
>
>> On Mon, 17 Feb 2003, Dexter "Kim" Kimball wrote:
>>
>>
>>> It seems that cache garbage collection has some limitations when  
>>> asked to
>>> fill/purge in a short cycle.
>>>
>>> By the way, if you're going to routinely be reading/writing large  
>>> files
>>> sequentially, you may want to experiment with AFS cache chunk size.
>>>
>>> WRT AFS vs. NFS, I believe that in 1) a local area network with 2) a  
>>> lot of
>>> write activity, the caching overhead of AFS is a liability.  (Write  
>>> to local
>>> disk cache, read from local disk cache, network to fileserver, write  
>>> to
>>> fileserver disk.)
>>>
>>
>> I think there are cases where it makes sense to bypass the cache  
>> (e.g. be
>> NFS). I don't know how we might actually figure out how to do so  
>> without
>> user intervention.
>>
> I agree with Derrick, if we are looking for optimal write performance  
> we are
> going to forget distributed filesystems and look at writing to local  
> disk.
>
> You probably will get better data transfer from ftp-client =>  
> ftp-server
> than a distributed filesystem.
>
> All that said, I suggest you look at the output from "ifconfig".
> Nearly every system I have looked at "ifconfig" has far more
> Rx (Read) data traffic than Tx (Transmit) data traffic.
>
> Here is a random example:
>
> $ ifconfig tr0
> tr0       Link encap:16/4 Mbps Token Ring (New)  HWaddr  
> 00:60:94:55:11:97
>          inet addr:10.10.10.19  Bcast:10.10.10.127   
> Mask:255.255.255.128
>          UP BROADCAST RUNNING MULTICAST  MTU:4056  Metric:1
>          RX packets:260154 errors:0 dropped:0 overruns:0 frame:0
>          TX packets:80397 errors:0 dropped:0 overruns:0 carrier:0
>          collisions:0 txqueuelen:100
>          RX bytes:229960399 (219.3 Mb)  TX bytes:9374312 (8.9 Mb)
>          Interrupt:15 Base address:0x7800
>
> Just to emphasize the point: 219 MB data read and 9 mb data written.
> That's about 24 to 1 ratio of read/write data traffic.
>
> In general, for most client systems, there is more data reading than  
> data writing.
> So, distributed filesystems had better be good at handling read data.
> That's one of the things AFS is good at because if its disk (or RAM)  
> cache.
>
> When it comes to "uploading" large datafiles from machine to machine,
> I now prefer using SSH's scp which shows me a progress bar and ETA.
>
> For most all other tasks, I like AFS which gives me other capabilities
> unheard of in NFS.

Yes, it is true that fast read performance should be a higher priority.  
  But if I want to read a 1 GB file from AFS, I have to write it there  
in the first place.

Look at the times to read that same 1 GB file:

nfs              0.050u 18.050s 2:25.59 12.4%
afs 0.5GB cache  0.030u 10.300s 2:29.26 6.9%
afs 2GB cache    0.030u 11.250s 2:32.67 7.3%

Within the normal fluctuations in the network itself, AFS is very close  
to raw NFA, for first-time reading.  But also note that even when the  
cache is smaller than the file, one does not see the poor behavior  
exhibited in the write case.

It takes 2 1/2 minutes to read the file, and 4 minutes to write it when  
there is enough cache.  But as a user, when that goes up to 10 minutes,  
and uses 80% of the processor, my first reaction would be that  
something is wrong.

 From other comments I've seen, it appears that this is "expected"  
(though hopefully not desirable) behavior.  I just wanted to point out  
that there is room for improvement in this area.
------------------------------------------------------------------------ 
--
Edward Moy
Apple Computer, Inc.
emoy@apple.com

(This message is from me as a reader of this list, and not a statement
from Apple.)