[OpenAFS] All limitations of OpenAFS

[Steven Jenkins]

On Tue, Jan 26, 2010 at 10:59 AM, Lars Schimmer
<l.schimmer@cgv.tugraz.at> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Harald Barth wrote:
>>> - -max number of db servers in one cell?
>>
>> I have seen cells with up to 7 db servers. Most cells use 3. A few use five.
>
> I know that 3 db servers are fine to work with.
> But I do not know the exact size and setup of the cell - maybe it could
> be handy to setup a db server on every fs, to.

I would not suggest setting up a dbserver on every fileserver.  There
is only a limited gain to be had, and you are greatly increasing
complexity and timing in a failure situation.

3-5 dbservers should be adequate for almost all cells.

> And as the cell is planned in the 20-200 fileservers area, I just asked.

There is a hard limit of 254 fileservers per cell.  I have not
personally seen a cell with more than about 40-50 fileservers, but
there is no technical reason you could not have more.

> I would love to have 3-5 db servers distributed and fileservers
> seperate, but money and other limitations could force me...
>
>> I don't understand why you would want more than say 3. If you have
>> unreliable network connections (say between Asia and South America)
>> you should rather consider making more cells than one spanning the
>> whole area with the bad network bottleneck in the middle. Here "good"
>> is not bandwidth but good uptime, low packet drop and reasonable
>> roundtrip times (multiple satelite hops are bad).
>>
>>> For a new EU project
>>

I agree with Hartmut with respect to creating multiple cells rather
than creating a single, very large cell.  There are several schools of
thought with respect to architecting a very large namespace as you are
describing.  Could you explain the expectations your users would have
with respect to a WAN problem impacting your cell?

>...
> Our setup would look like:
> 1 cell EU wide, with 5-200+ local sites, each has at least one fileserver.
> Each site has groups AKA departments.
> Each group has special volumes on "its local" fileserver and on other
> fileservers RO/RW rights to special directories/volumes.
>
> (setup is guided by the rule "data HAS to be kept local and only local
> groups and external persons with special rights should be able to read
> it. Local groups should be able to make only very small subpart of data
> available to one or more other (external) groups).
>

It would be very helpful to understand why you desire a single cell.
I suspect your main concern is to keep a single username/userid,
group/groupid namespace.  If that's your desire, one strategy you
could use is:

- have one cell per site
- use a single Kerberos domain across all cells
- replicate your pts database across all cells
- set up a single namespace

Managing replication across 200 cells would be very interesting, however.

Another strategy would be to maintain multiple Kerberos domains and
PTS groups.  If you choose to use that strategy, you will need to
create and maintain careful naming conventions to manage the mappings
between users and locations across each cell.  This would look more
like the 'public' AFS space.

Also, as you mention, managing volume names will be a challenge.  Many
large sites have tools in place to manage namespace management (i.e.,
mappings between paths and volumes) along with administrative
delegation (e.g., allowing certain users to do certain operations like
vos release's but  not other operations).

Steven