[OpenAFS] Re: Server disk operations speed
Sun, 7 Apr 2013 18:09:07 -0500
On Sun, 7 Apr 2013 17:59:48 +0300 (EEST)
> I had a directory containing about 2000 files, 300 MB in total.
> I duplicated the directory A) inside afs space
> B) outside afs, but on the same server
So, you're using rsync or 'cp -r' or something to copy to /afs/ ? A few
things you should probably just be aware of in general about this:
- Latency probably matters more than bandwidth. If you're copying a
tree of lots of little files, it will probably be slow; as is the
case with many filesystems, our performance isn't always great for
lots of little files. This is because for every file we write, we
must create the file, then write to it and close it, and possibly
update metadata on the file. Each of these operations must happen
synchronously, so every file written involves a delay of at least
3xRTT I think, since we need to communicate with the server 3 times
(and possibly more, depending on the specific process involved). The
given numbers suggest an average of 150K per file, which I wouldn't
call "tiny" but still not large.
- If you're copying to /afs/ using a client on the same machine as the
fileserver in question, the file(s) you're writing hit the disk
twice. Once when you write to the client cache, and once when you go
to the fileserver. If the partition your cache is on is the same as
the partition the fileserver is using for that volume, that could
result in slower performance. Also keep in mind any performance
bottlenecks may be due to the client instead of the fileserver.
So, I would wonder if you get the same speed if you write a single large
file instead of all of those little files (say, just tar up the files
and copy the tarball into afs).
It may also help see what's going on if you can get some kind of idea of
what the cpu or disk utilization looks like during the transfer. Just
see if/when the cpu or disk are getting saturated, or look at the
network traffic if you want to.
Also, this is a baseless guess, but try looking at if there are udp
packet drops; look in the section marked -udpsize in