[OpenAFS] Re: afs vs nfs

[lamont@scriptkiddie.org]

On Thu, 24 Nov 2005, Jeffrey Altman wrote:
> lamont@scriptkiddie.org wrote:
>> On Wed, 23 Nov 2005, Jeffrey Altman wrote:
>>> Cells are demarcations of authorization boundaries.   If your Bangalore
>>> and New York offices are both controlled by the same authorities then
>>> they should not be separate cells.
>>>
>>> What you want to happen in this case is for your volumes to be migrated
>>> from the New York servers to the Bangalore servers within the same cell.
>>
>> Then how would you setup a company that had a single administrative
>> and authorization domain and a dozen geographically diverse offices?
>> Separating the database servers geographically isn't a very good
>> option, and running everything over the WAN isn't a very good option
>> either. Particularly with regards to bangalore, your transit will tend
>> to suck and it would be good not to have to go over the WAN to your
>> database servers.
> I would use the AFS volume management capabilities to keep the user's
> volumes close to where they reside.   When the employee moves from New
> York to Bangalore they could register the move via a web form which
> would schedule the movement of their volumes to servers in Bangalore.
>
> The cost of distributing the volume database servers geographically is
> cheaper than the cost of running all the clients across the WAN,
> therefore, I would distribute the database servers.
>
> If you want to have a global root then you are going to have to have
> some form of global database for the clients to access.   The global
> root as it is now can be considered the sum of all AFSDB records in
> DNS.  That is the top-level mapping from "cell name" to volume database
> server pool.
>
> The location independence of AFS is entirely a result of the Volume
> Database.   It sounds to me that what you really want is a more
> efficient set of protocols for distributing change sets among the
> database servers over a WAN.  Perhaps even over a segmented WAN in which
> there is only periodic access.   This is the most important set of
> functions for the next version of Microsoft's Active Directory, an
> ability to perform synchronization between the database servers using
> alternate protocols including delayed e-mail when access to branch
> offices is sporadic.   You really don't want to get rid of the
> /afs/cellname/ scheme.

With a few dozen geographically separated datacenters, the latency and 
overall transaction rate for the database servers would fall dramatically. 
Also, if there's a network partition and you lose quorum then you lose 
write transactions to the database entirely.

This doesn't work if you've got a requirement for 100s of TPM to the 
database servers for local volume creation/modification and the datacenter 
needs to be able to run autonomously and writes need to be highly 
available.

You lose a lot trying to do this, and I don't understand how the mapping 
of administrative domain onto cell buys you anything really concrete.

Definitely as it exists today ubik doesn't like graphically separated 
database servers, and if you administer multiple datacenters on multiple 
coasts/continents you will be setting up multiple cells.  The Morgan 
Stanley model, and total location-independence, makes much more sense than 
the /afs/<cellname> convention for that kind of organization.  It may be 
possible to design a database server which allows for the use cases I 
mention but which preserves the /afs/<cellname> convention and a single 
cell for a geographically large organization, but such a database server 
is entirely vaporware.