[OpenAFS] improving cache partition performance

[Jason Edgecombe]

On 08/29/2011 07:10 PM, Andy Cobaugh wrote:
>
> Just to add some more datapoints to the discussion...
>
> Our webservers are HP DL360G5's, 14GB RAM. Pair of 36GB 15K 2.5" SAS 
> drives in RAID-1. /usr/vice/cache is ext2 with noatime,nodiratime. 
> These machines run dovecot IMAP, apache with lots of php applications, 
> RT, and vsftpd serving anon and private ftp accounts.
>
> Serving content that isn't in the cache yet, we can get about 
> 70-80MB/s depending on which fileserver it's coming from, and after 
> it's cached, the gigabit network becomes the bottleneck. The cache 
> partition is ~34GB in size, and we're running with these options:
>
> -dynroot -dynroot-sparse -fakestat -afsdb -nosettime -daemons 20
> -stat 48000 -volumes 2048 -chunksize 19 -rxpck 2048
>
> With those cache manager settings, cache partition utilization is 
> sitting at about 92%. I can get even better numbers with memcache, and 
> indeed most of our other machines are running with 2GB of memcache, I 
> like seeing read performance in GB/s, and when most of your machines 
> have 32GB or more (we have 3 with 256GB), a couple GB here and there 
> won't have a noticeable impact.
>
> Jason: do you know in particular what kind of workload is causing 
> issues for you? You mentioned your wait times are on the order of 
> seconds, are you sure that's caused by the underlying disk? At the 
> very least, I would try mounting your cache partition as ext2, as has 
> already been suggested. Turning off atime and diratime shouldn't hurt, 
> and if your disks are having issues with seaks, this should help some.
>
> Also, you really want to run 1.6.0pre7, or 1.6.0 when it shows up. 
> Nothing wrong with 1.4, but if you're trying to get the most 
> performance out of afs on modern hardware, switching to 1.6 gives you 
> some real cheap gains. There are huge performance improvements on 
> Linux going from 1.4 to 1.6, and all of my new installations are 
> 1.6.0pre7 for that reason. Especially with disk-based caches, as Simon 
> mentioned. 1.6.0pre7 gets write performance for disk caches almost on 
> par with memcache, though read performance is still lacking, as memory 
> will almost always be faster than disk, but disk will always be 
> 'cheaper' than memory.
>
> Worth a try at least, and pre7 has been very stable in our environment.
I was told that noatime is bad for an AFS cache partition because AFS 
uses the atime to know when the cache entry was last accessed.

Is that the case?

Jason