[OpenAFS] improving cache partition performance
Jason Edgecombe
jason@rampaginggeek.com
Mon, 29 Aug 2011 19:39:14 -0400
On 08/29/2011 07:10 PM, Andy Cobaugh wrote:
>
> Just to add some more datapoints to the discussion...
>
> Our webservers are HP DL360G5's, 14GB RAM. Pair of 36GB 15K 2.5" SAS
> drives in RAID-1. /usr/vice/cache is ext2 with noatime,nodiratime.
> These machines run dovecot IMAP, apache with lots of php applications,
> RT, and vsftpd serving anon and private ftp accounts.
>
> Serving content that isn't in the cache yet, we can get about
> 70-80MB/s depending on which fileserver it's coming from, and after
> it's cached, the gigabit network becomes the bottleneck. The cache
> partition is ~34GB in size, and we're running with these options:
>
> -dynroot -dynroot-sparse -fakestat -afsdb -nosettime -daemons 20
> -stat 48000 -volumes 2048 -chunksize 19 -rxpck 2048
>
> With those cache manager settings, cache partition utilization is
> sitting at about 92%. I can get even better numbers with memcache, and
> indeed most of our other machines are running with 2GB of memcache, I
> like seeing read performance in GB/s, and when most of your machines
> have 32GB or more (we have 3 with 256GB), a couple GB here and there
> won't have a noticeable impact.
>
> Jason: do you know in particular what kind of workload is causing
> issues for you? You mentioned your wait times are on the order of
> seconds, are you sure that's caused by the underlying disk? At the
> very least, I would try mounting your cache partition as ext2, as has
> already been suggested. Turning off atime and diratime shouldn't hurt,
> and if your disks are having issues with seaks, this should help some.
>
> Also, you really want to run 1.6.0pre7, or 1.6.0 when it shows up.
> Nothing wrong with 1.4, but if you're trying to get the most
> performance out of afs on modern hardware, switching to 1.6 gives you
> some real cheap gains. There are huge performance improvements on
> Linux going from 1.4 to 1.6, and all of my new installations are
> 1.6.0pre7 for that reason. Especially with disk-based caches, as Simon
> mentioned. 1.6.0pre7 gets write performance for disk caches almost on
> par with memcache, though read performance is still lacking, as memory
> will almost always be faster than disk, but disk will always be
> 'cheaper' than memory.
>
> Worth a try at least, and pre7 has been very stable in our environment.
I was told that noatime is bad for an AFS cache partition because AFS
uses the atime to know when the cache entry was last accessed.
Is that the case?
Jason