[OpenAFS] cache performance

[Phil.Moore@morganstanley.com]

>>>>> "Todd" == Todd M Lewis <utoddl@email.unc.edu> writes:

Phil> This is very interesting indeed, but we're way too diverse to
Phil> impose a execution mechanism on our environment, especially with
Phil> the specter of 30,000 Windows boxes all just waiting to finally
Phil> have a stable distributed filesystem out of which to run
Phil> applications.

Todd> I don't quite follow you.  The overhead of having an application send a 
Todd> UDP datagram on startup is not a particularly onerous imposition. And it 
Todd> doesn't require any reconfiguration of the client.

I'm not concerned about the performance overhead, I'm concerned about
the management overhead....

What I'm after is a near-complete (I'll take 90-95%) audit of the
usage of my AFS volumes.  Your approach would not scale in an
environment as large as ours, simply because of the diversity of apps
we have, and the global nature of our client base.

The AFS infrastructure is *THE* place to deploy production apps, and
we have 1000's of distinct applications (managed by lots of distinct
groups), running on lots of different platforms, all of which would
have to be wrapped in some fashion.  That's a gargantuan task to begin
with.

Then we have to worry about managing the collection of data.  We can't
have the entire planet logging to one place.  We have to distribute
that (London logging somewhere in Lond, NY in NY, etc.) and then
collect the data and analyze it centrally.

Phil> The strategic focus *MUST* be on getting richer statistics out
Phil> of the fileservers, so we can perform this analysis centrally.

Todd> Maybe I wasn't clear, but the logging happens centrally,
Todd> wherever you choose to run the runloggerd daemon, so analysis is
Todd> central as well.  I agree some interesting info could be gleaned
Todd> from the fileservers, but you can put runlogger into just the
Todd> apps/pkgs you are interested in and get very focused logs to
Todd> analyze. (Or do what we do and log everything you can get your
Todd> fingers into.)

The log information may be collected centrally, but it is generated on
each and every client, by each and every app.  That's my point.

But again, I'm not knocking your solution as a viable alternative,
merely saying it won't scale to *our* scale, which is far larger than
most other AFS shops.  Therefore, I still encourage small to medium
sized shops to look at this approach; it has value.

If you're a huge enterprise like us, a client-based solution is
simply too expensive (I mean *everything* scales, if you've got enough
time and money).

Therefore, we're going to focus our energy on server-based auditing
that will meet these needs.  Hopefully, if we do it right, you'll just
have to upgrade to the latest OpenAFS server code, and get this new
feature for free.

Everything we fund for OpenAFS is contributed back to the code base;
we don't do Morgan Stanley specials (been there, done that, paid
millions to undo it).