[OpenAFS] improving cache partition performance

[Andy Cobaugh]

Just to add some more datapoints to the discussion...

Our webservers are HP DL360G5's, 14GB RAM. Pair of 36GB 15K 2.5" SAS 
drives in RAID-1. /usr/vice/cache is ext2 with noatime,nodiratime. These 
machines run dovecot IMAP, apache with lots of php applications, RT, and 
vsftpd serving anon and private ftp accounts.

Serving content that isn't in the cache yet, we can get about 70-80MB/s 
depending on which fileserver it's coming from, and after it's cached, the 
gigabit network becomes the bottleneck. The cache partition is ~34GB in 
size, and we're running with these options:

-dynroot -dynroot-sparse -fakestat -afsdb -nosettime -daemons 20
-stat 48000 -volumes 2048 -chunksize 19 -rxpck 2048

With those cache manager settings, cache partition utilization is sitting 
at about 92%. I can get even better numbers with memcache, and indeed most 
of our other machines are running with 2GB of memcache, I like seeing read 
performance in GB/s, and when most of your machines have 32GB or more (we 
have 3 with 256GB), a couple GB here and there won't have a noticeable 
impact.

Jason: do you know in particular what kind of workload is causing issues 
for you? You mentioned your wait times are on the order of seconds, are 
you sure that's caused by the underlying disk? At the very least, I would 
try mounting your cache partition as ext2, as has already been suggested. 
Turning off atime and diratime shouldn't hurt, and if your disks are 
having issues with seaks, this should help some.

Also, you really want to run 1.6.0pre7, or 1.6.0 when it shows up. Nothing 
wrong with 1.4, but if you're trying to get the most performance out of 
afs on modern hardware, switching to 1.6 gives you some real cheap gains. 
There are huge performance improvements on Linux going from 1.4 to 1.6, 
and all of my new installations are 1.6.0pre7 for that reason. Especially 
with disk-based caches, as Simon mentioned. 1.6.0pre7 gets write 
performance for disk caches almost on par with memcache, though read 
performance is still lacking, as memory will almost always be faster than 
disk, but disk will always be 'cheaper' than memory.

Worth a try at least, and pre7 has been very stable in our environment.

--andy