[OpenAFS-devel] Large Caches: Implementation Discussion

[Nathan Neulinger]

Derek Atkins wrote:
> 
> Derek Atkins <warlord@MIT.EDU> writes:
> 
> > One thing that I haven't had a chance to think about (until now)...
> > I'm not sure how the CacheItems file keeps track of what is contained
> > in each cache file.  However, I have a feeling that if someone changes
> > the number of files per subdirectory, the entries in the CacheItems
> > file may become invalid.  The reason is that the individual files will
> > in essence be re-numbered when new files are added to each directory.
> > For example, D1/V0 is cachefile number "1 * files_per_subdir + 0",
> > which will change based on files_per_subdir.
> 
> This has been solved.  Part of the solution is that when we upgrade
> from the old-style cache to this new-style cache, users will NOT lose
> their existing cached data.  The cachefiles will just get moved into
> their appropriate subdirs.

That also takes care of the problem with changing number of files?

> I have tested my patch, and it takes under 5 minutes to build a
> 3-million-file cache hierarchy (for "3GB" of data) using the default
> settings of 2048 files per directory.  This implies 147
> subdirectories.

1k chunks? Have you seen better cache performance with the smaller
chunks? Or did you just change the algorithm for number of total files
as part of this large cache patch?

> Anyways, I think I've got all the issues worked out, now.  Does anyone
> else have any input before I submit my patches?

Not really, other than that you might as well wait, since Derrick is
going to be gone for a few days still (I think next thursday, but he
might have meant today). I'd send it to the list so some more people can
test it... 

-- Nathan

------------------------------------------------------------
Nathan Neulinger                       EMail:  nneul@umr.edu
University of Missouri - Rolla         Phone: (573) 341-4841
CIS - Systems Programming                Fax: (573) 341-4216