[OpenAFS-devel] PATCH: limit afsd auto-tuning to 500000 files

[Troy Benjegerdes]

On Wed, Aug 24, 2005 at 07:38:57AM -0400, Todd M. Lewis wrote:
> 
> 
> Jeffrey Hutzelman wrote:
> >
> >We won't know unless we can collect and analyze some data.  Until we do 
> >that, repeatedly making changes to the autotuning algorithm isn't going 
> >to make things better; it's just going to make it unpredictable.
> >
> >
> >So, let's hear from people who are actually using large caches [...]
> >
> >[...] I'd be very interested to see how either of these numbers 
> >varies with cache size, working set size, average file size, or "site 
> >size", whatever that means....
> 
> I wholeheartedly agree. However, I don't think you're going to get much 
> response to this request for data.  Most folks don't even know whether 
> they qualify as "people who are actually using large caches". 
> Furthermore, everybody who comes up with some numbers to pass on to you 
> are going to invent their own methods of gathering that data based on 
> their individual misunderstanding of the questions.

I'm going to argue that the only data really worth getting is after 2
weeks of uptime, does anyone have a huge (> 20g) cache more than 50% 
usage of total blocks available, or total files available.

The 500K files is very arbitrary, but I picked it to keep the amount of
time it takes to create all the cache files and do the cache scan to
some amount most users will probably not complain about. While the
sqrt() approach mentioned earlier is a conceptually cleaner approach,
I think it's more complicated to for users to figure out.

In reality, I think we're going to find that our original 32k average
file sizes are not that far off, but individual use cases may be way off
the average. What I'd really like to know is if there is anyone out
there that actually has more than 1 million files in their cache.

There are also a bunch of linear 'for i=0; i < cacheFiles' loops
scattered about in the kernel module. I'd say that this, along with
vmmalloc usage is a good reason to have some static upper limit on the
auto-tuned number of files.