[OpenAFS] /usr/vice/cache size recommendations

[Kim Kimball]

Your mileage may vary.  However, I believe that a 1GB cache shouldn't be
problematic ... I routinely ran 3 GB caches under HPUX 9/10/11 without a
problem, in production, for years.

The success of such an approach (e.g. the 3GB cache) may depend critically
on the composition of the files being cached.

In my case, about 1.5 GB was cached in about fifteen files -- and we forced
a refresh of the client caches when the corresponding volumes were vos
release'd.  We also "pinned" the files in the cache by catting them to
/dev/null periodically -- because we didn't want them purged and refetched
over slow networks.  Ever.

The rest of the cache (about 1.2 GB -- as ~ 10% is reserved/overhead for
cacheindex) consisted of files that were 40K or less.

We had very good response time characteristics.  But the number of cached
files being tracked was skewed by the large number of Vfiles occupied by the
fifteen files, pinned in cache.  So in some ways the performance
characteristics were consistent with a 1.2GB cache.

On the other had I also experimented with a 12 GB cache with the same
fifteen files and the balance in 40K files.

 The garbage collection (FILO) of the V files caused the afsd's to
effectively refuse all other activity during a cache purge.  This was with
AFS 3.3 and a while ago, and I did not vary from default the number of
afsd's I was running, which might have helped with the garbage collection
problem.  In any case, the AFS client appeared to be locked up, until
purging completed -- at which point normal activity resumed, until the next
cache purge, at which point the AFS client appeared to be locked up again.

Note that filesize and the number of cache entries ("V-files") allocated can
also result in performance conundrums.

One example of this for me was an application that read lots of small files
in a loop.  The files were 1-2K.  The AFS client assumes that each Vfile
(cache file) will contain about 10K by default, and allocates/creates  a
number of Vfiles based on that 10K/Vfile assumption.

With the tiny files this assumption was violated -- the result was that the
cache was purged far more often than available disk space indicated --
because the cache was being purged to make V files available for new items,
not because the cache space was consumed.

In fact, I never even got close to using the available configured cache disk
space:  I was using five or ten times the number of Vfiles than the AFS
client assumes with its 10K metric, since each V file can contain chunks of
exactly one file.  (The chunksize is a MAX size of a Vfile, and defaults to
64K -- but a Vfile will not contain 64K of data unless all of that data is
from exactly one file.)  And each used Vfile contained 1-2K, not the 10K
average that is assumed.

I noticed the purging frequency using xstat (afsmonitor is easier for some).
Creating more Vfiles made a significant difference in the purging frequency,
and performance improved.

Tweak it, test it, run it.

Kim

-------------------------------------------------------
Dexter "Kim" Kimball
CCRE, Inc.
14421 N. County Rd. 25E (Swanson Ranch Road)
PO Box 209
Masonville, CO 80541

        kim@ccre.com
-------------------------------------------------------


> To: openafs-info@openafs.org
> Date: Tue, 09 Apr 2002 10:19:25 -0700
> From: Lee Damon <nomad@ssli-mail.ee.washington.edu>
> Subject: [OpenAFS] /usr/vice/cache size recommendations
>
> In the {old,expired,questionably_useful} quick start guide (the one
> that still says IBM all over it), they talk about making /usr/vice/cache
> all of maybe 50MB.  I realize this is really old data, and suspect
> that a much larger cache is reasonable. However, I remember a conversation
> I had long ago in which it was pointed out that too large a cache is
> counter-productive.
>
> So, what's the current thinking of a reasonable size for /usr/vice/cache,
> especially with today's disk size?  Is 512MB too large? 1GB?
>
> thanks,
> nomad
>  -----------                       - Lee "nomad" Damon -          \