[OpenAFS] Re: Server disk operations speed

[Andrew Deason]

On Sun, 7 Apr 2013 17:59:48 +0300 (EEST)
jukka.tuominen@finndesign.fi wrote:

> I had a directory containing about 2000 files, 300 MB in total.
> 
> I duplicated the directory A) inside afs space
>                            B) outside afs, but on the same server

So, you're using rsync or 'cp -r' or something to copy to /afs/ ? A few
things you should probably just be aware of in general about this:

 - Latency probably matters more than bandwidth. If you're copying a
   tree of lots of little files, it will probably be slow; as is the
   case with many filesystems, our performance isn't always great for
   lots of little files. This is because for every file we write, we
   must create the file, then write to it and close it, and possibly
   update metadata on the file. Each of these operations must happen
   synchronously, so every file written involves a delay of at least
   3xRTT I think, since we need to communicate with the server 3 times
   (and possibly more, depending on the specific process involved). The
   given numbers suggest an average of 150K per file, which I wouldn't
   call "tiny" but still not large.

 - If you're copying to /afs/ using a client on the same machine as the
   fileserver in question, the file(s) you're writing hit the disk
   twice. Once when you write to the client cache, and once when you go
   to the fileserver. If the partition your cache is on is the same as
   the partition the fileserver is using for that volume, that could
   result in slower performance. Also keep in mind any performance
   bottlenecks may be due to the client instead of the fileserver.

So, I would wonder if you get the same speed if you write a single large
file instead of all of those little files (say, just tar up the files
and copy the tarball into afs).

It may also help see what's going on if you can get some kind of idea of
what the cpu or disk utilization looks like during the transfer. Just
see if/when the cpu or disk are getting saturated, or look at the
network traffic if you want to.

Also, this is a baseless guess, but try looking at if there are udp
packet drops; look in the section marked -udpsize in
<http://openafs.org/pages/newsletter/newsletter-2013-03-volume004-issue05.html>.

-- 
Andrew Deason
adeason@sinenomine.net