[OpenAFS] Re: Redundant Internet links

[Andrew Deason]

On Sat, 18 Dec 2010 03:41:34 +0100
Jaap Winius <jwinius@umrk.nl> wrote:

> > To be clear, the fileserver does not become readonly; what becomes
> > readonly are the databases that contain volume location information
> > and authenticated user metadata. So, that means you can read and
> > write to files to any fileserver you can reach, but you cannot
> > create, remove, or release volumes, create, remove, or alter
> > users/groups, or anything else that requires modifying those
> > databases.
> 
> Very interesting. So, I take it a different, local database is used to
> keep track of the changes made to individual files in local R/W
> volumes, and this database stays R/W even if the server it's on gets
> cut off?

I wouldn't call it much of a "database". There is a database mapping
volume names to locations (among other data), and then there are the
volumes on the fileservers themselves; they're stored very much like
regular files. When you access files inside a volume on a fileserver,
the only communiation that takes place is between the client and the
fileserver (usually), so if the fileserver was "cut off" from the other
sites, it wouldn't even know.

> > ... You contact the vlserver at site A, and
> > it will tell you that the volume is on a fileserver at site B, and it
> > will also tell you all known IP addresses for the fileserver at site B.
> 
> Sounds like you're referring to the IP addresses for the servers that  
> the clients are given. In that case I understand. I can do that with  
> AFSDB RRs.
> 
> What I meant, though, are the IP addresses that the servers have to  
> contact each other. On Debian, these are in  
> /etc/openafs/server/CellServDB. I'd like to use multiple IP addresses  
> for each host in there too, but that would adversely affect the voting  
> algorithm.

Those addresses are for the database servers (which control the VLDB and
PTDB); the CellServDB and AFSDB/SRV RRs are only related to accessing
the VLDB/PTDB (and budb, kadb, etc if you're using them).

The IP addresses for fileservers are not kept in the VLDB nor in DNS.
They are kept in the VLDB, and they are normally maintained
automatically; you don't need to configure them. When a fileserver
starts up, it detects what IP addresses it has, and tells the VLDB "I am
fileserver X, and I have IPs foo, bar, and quux". Then, later, when a
client asks the VLDB "where is volume V?", it gets the answer "it is on
fileserver X, which has IPs foo, bar, and quux". You can change which
addresses a fileserver advertises with the NetInfo and NetRestrict
configuration files (see the documentation on how to use them).

So, my point is that you can keep only three addresses in the CellServDB
or in DNS, and you only need to be able to access _one_ of those IP
addresses in order to read or write file data (in most situations). 

> On the other hand, what if I were to set up virtual hosts on which to  
> run the file servers separately? In that case, each database server  
> would still run on the bare metal OS and those CellServDB files would  
> still contain only three IP addresses. Lower level routing would still  
> take care of connectivity if one of the main links went down. The  
> files servers, however, could each have a CellServDB file with five  
> addresses: a local private range address and four public addresses for  
> the two remote file servers (which would be reached through  
> port-forwarding).

That's possible, yeah (or you can even compile the fileserver binaries
to use a different path to the CellServDB, though that would get
confusing). But fileservers only need RW access to the database servers
once, at startup. I'm fairly sure all other dbserver operations they
perform are readonly, and thus only require reaching one dbserver site
anyway.

-- 
Andrew Deason
adeason@sinenomine.net