[OpenAFS] Server disk operations speed

Markus Koeberl openafs-info@openafs.org
Mon, 8 Apr 2013 10:00:18 +0200


On Sunday 07 April 2013 20:55:41 jukka.tuominen@finndesign.fi wrote:

> So, I thought that the network communication was the bottle neck, but just
> in case I tested whether it's the server itself that was somehow
> exceptionally slow. I copy/pasted the earlier mentioned test directory
> (see the full message below) first in the normal unix environment
> /home/user/...(source and target), which turned out to be extremely fast
> (HW supported virtualization on a SSD disk). So, no problem there.

Check the I/O scheduler settings on your virtual machine and on the server. 
The default cfq is not the fastest for this usecase. Especially if you are 
using a battery backed hw raid you may test both set to noop.

You are testing the worst case for your setup:
Disk I/O and network I/O is very expensive for virtualization. Also if you are 
using paravirtualization the overhead is noticeable.

Copy at /home means you only have the disk I/O overhead for writing (guess all 
300MB are already in cache and there is no need for synchronous write 
operations).

> Then I got access to an afs account on the same machine (kinit; aklog),
> and copied the same test directory in /afs/cell/user/...(source and
> target) which turned out to be extremely slow.

For afs:
Also if the data are already in cache the client has to check if the files or 
permissions did not change meanwhile on the server. so you get network I/O for 
each file and probably disk I/O for the client disk cache.
The client copies each file which will lead to disk I/O on the client disk 
cache, network I/O for client and server and disk I/O for the server. The disk 
I/O most probably includes lots of synchronous operations...

Using a afs client on the fileserver is most probably slower than on another 
client in your network. 
Also try to use memcache on the client for the test!
 
> I did try to optimize the fileserver parameters, but I could only gain
> like 30-40% improvement. I believe there is some bigger issue to be solved
> first. I wonder if anybody else has had a similar issue, or if you could
> think of any possible causes for this sort of symptons?

For us AFS performs mach faster on small files than our cluster filesystem 
MooseFS which we are using in parallel. AFS is something like 20-30 times 
faster for small files... (tested with 40905 Files of total 444.47 MB)

We also use AFS fileserver with ssd for our small $HOME volumes but without 
virtualization. For other fileservers (mainly hosting volumes with bigger 
files) we use xen with paravirtualization which is noticeable slower.

-- 
Markus Koeberl
Graz University of Technology
Signal Processing and Speech Communication Laboratory
E-mail: markus.koeberl@tugraz.at