[OpenAFS] Re: OpenAFS-info digest, Vol 1 #479 - 13 msgs

[Kim Kimball]

This isn't as simple as it might sound.

The size of the fileservers is one issue.

The nature of the connections is another issue.

Some connections are fairly quiet.  I built a document distribution system
for United Airlines -- worldwide distribution of millions of small text and
TIFF files -- over 40,000 users, 24x7, 365/yr -- for maintenance
documentation that is required at every maintenance facility and every
_gate_ world wide.  We did this with servers at five domestic locations and
three international locations -- twelve or so servers in all -- and found
that we could have accommodated the entire load on three or so servers --
EXCEPT that we needed fast response time over relatively slow WAN -- hence
the multiple locations were due to WAN constraints and not server
constraints.

These relatively quiet connections are in an environment that uses AFS for
document distribution, a replication-heavy environment.

Some connections are distinctly NOT quiet.  A chip manufacturer I consulted
for briefly uses AFS for document sharing, in a parallel distributed
processing (PDP, but not to be confused with an old DEC) environment.  The
AFS volumes are shared across multiple processes on multiple
machines/processors (primarily Linux) -- sometimes several hundred AFS
clients are sharing the same AFS volumes and there are thousands of accesses
in short periods of time.  The chip simulations are CPU intensive -- but
need to share data files -- and in this environment the relatively few
servers are being pounded heavily by hundreds of simultaneous clients/users
_per AFS readwrite volume_ -- implying of course hundreds of users per
fileserver instance.

These very busy connections showed throughput levels (transaction levels
against individual AFS volumes and their fileservers) that completely
revised my previous guesses of how AFS would handle instense loads.  I am
still quite surprised that AFS performed as well as it did in this PDP
environment.

To make a long story short, the number of simultaneous connections _and_ the
nature of connections will determine how heavily you can load your AFS
fileservers.

I suggest monitoring loads using volinfo to determine a) what individual
transaction rates are against specific volumes and b) what the aggregate
transaction rates are against the fileserver housing the volumes.

Somewhere I have scripts I've written to monitor volume transaction rates --
in order to determine a) how to move volumes around to best balance loads
across fileservers b) when it's prudent to add another fileserver, c) when a
replicated volume might benefit from further replication (or conversely), d)
when to "grow" the cell by adding another fileserver.

There are other tools -- xstat, afsmonitor, etc -- at least with
Transarc/IBM AFS -- that help determine data transfer, and response times --
two other components that might lead me to grow the cell.  Or shrink it.

-------------------------------------------------------
Dexter "Kim" Kimball
CCRE, Inc.
14421 N. County Rd. 25E (Swanson Ranch Road)
PO Box 209
Masonville, CO 80541

        kim@ccre.com
-------------------------------------------------------

================
Message: 7
To: "T. Matthew Cocker" <matt@cs.auckland.ac.nz>
Cc: openafs-info@openafs.org
Subject: Re: [OpenAFS] maximum connections to afs
From: Derek Atkins <warlord@MIT.EDU>
Date: 03 Feb 2002 21:42:41 -0500

How large are your file servers?  There may be a theoretical limit,
but I don't know what it is.  For all practical purposes I don't think
there is one.  But then again I don't know how large a user-base you've
got.

-derek

"T. Matthew Cocker" <matt@cs.auckland.ac.nz> writes:

> Hi
>
> We finally pushing OpenAFS into full productions after running a small
> testing cell for 6 months and we are trying to decide how to spread
> our users around the fileservers. Is there maximum recommended number
> of simultaneous connections to anyone FS?
>
> Cheers
>
> Matt