[OpenAFS] openafs performace problems

[Jason Edgecombe]

Juha Jäykkä wrote:
> Hi all!
>
> What is the speed that can be expected of openafs, when writing some (perhaps 
> 10-100) largish (perhaps 10-100 MB) files or a single 4GB file to an afs 
> volume?
>
> The background for the question is a desire to move from using NFS for HPC 
> data to AFS; over the lab 100Mbps network the speeds are the same, limited to 
> abour 10MBps by the network; no problem there. But when writing to the afs 
> volume or NFS mount from the same host which is running the fileserver and 
> NFS server processes, things get nasty: I can write ~80MBps to the NFS share 
> (it being effectively local disc now), but AFS speed stays at 10MBps. I 
> cannot seem to figure out why.
>
> I've tried the different cache manager startup options, fs 
> storebehind -allfiles 2000000 and various speed-measuring tools: bonnie, 
> iozone and dd if=/dev/zero of=file_on_afs count=something bs=something. 
> Nothing seems to give more than 15 MBps from fileserver host to its own 
> volume. There is no significant network traffic meanwhile, i.e. contacting 
> the volume location servers etc is not the issue.
>
> How can I speed things up? This is rather annoying (and will prevent replacing 
> NFS with OpenAFS), since the volumes in question get huge bursts of incoming 
> data (over a 1Gbps link), which should be written to disc rather fast; 
> preferably at speeds where either the 1 Gbps link or the disc speed is the 
> limiting factor. Now, the limit is somewhere else and I cannot tell, where.
>
> Another question, which is just a curiosity: if I try to set storebehind 
> larger than about 2 gigabytes, fs says:
>
> # fs st -allfiles 4000000 -v
> fs: 4000000 must be 0 or a positive number.
>
> Obviously, 4000000 is positive and the system is 64-bit so, there should be no 
> problems with 32bit integers either - unless openafs has.
>
> Cheers,
> Juha
>
>   
Have you tinkered with the chunksize setting for afsd? Chunksize 
specifies the size of the block of data that is sent to or received from 
the server. When experimenting, be aware that chunksize is a power of 
two, so if you say "-chunksize 20" on the command-line, that means 2^20 
bytes or 1megabyte.

Be careful with the storebehind parameter, it allows for write calls to 
finish before the data is written to the server. If the write to the 
server fails, you have no way of knowing.

Jason