[OpenAFS] Re: Redundant Internet links

[Jaap Winius]

Quoting Andrew Deason <adeason@sinenomine.net>:

> Is the "internal" address only routable from that site?

No. Although each of the three sites will be connected to a small  
internal network with private IP addresses, the broadband modems will  
be in bridged mode and the server will also have two public IP  
addresses. Each server will also act as a router between the internal  
network and the external links (but not between the latter). I figure  
that it will be best if each OpenAFS server is only known by its  
public IP address(es), even to the clients on its internal network.  
This way, each OpenAFS client at each location in the cell will always  
contact the three OpenAFS servers using the same set of public IP  
addresses.

> If you're just taking the "real" sites and adding additional IPs,
> though, I think the clients should be fine. Aside from the
> aforementioned sync site determination, it's effectively just a list of
> sites to try to contact.

Okay, so I could give all of the clients a list of six IP addresses  
for the three servers. I plan to use DNS with AFSDB RRs for this,  
although that doesn't provide a method for giving priority to the  
faster links. However, I recently noticed RFC5864 (good work, Russ!):  
what's the minimum OpenAFS version necessary for anyone wanting to use  
SRV RRs for OpenAFS instead?

> dbservers are different, though; the specification in CellServDB affects
> the voting algorithm, and is not as simple as a list of sites to try to
> reach. I don't think I'd recommend putting multiple IPs per site in
> there, but I don't really know what happens if you do that. Each site is
> aware of the local addresses of the other sites, and I'm not sure what a
> site does if it sees that another site claims ownership of more than one
> CellServDB server entry... But I don't imagine it ending well.

Yeah, I suspected that would be a problem... despite the fact that the  
names following the "#" behind the IP addresses would be the same. So,  
I'll just use one IP address per server in the CellServDB files of the  
DB/file server machines and let the routing system take care of any  
communications problems.

> As to whether this is advisable in general... it's not immediately clear
> to me what failure case you're trying to cover here. If you have three
> geographically-diverse sites, losing access to one of those db sites
> will not significantly affect accessing AFS beyond an initial delay when
> the site goes away. If you lose two sites, then you lose the ability to
> make vldb or ptdb changes, but you can still access data (assuming the
> relevant fileservers are still accessible).

The redundant links will significantly reduce the risk for each site  
that, if it temporarily loses its connection to the Internet, the  
local OpenAFS file server also becomes read-only until the link is  
restored. But, besides that, the business already grinds to a halt  
when Internet access is lost due to their dependence on a proprietary  
online database management solution. So, the redundant links will also  
address an existing problem.

> ... it doesn't seem worth the effort of running with such a non-typical
> configuration. While it may be possible to make it work, providing the
> redundancy at a lower level seems easier.

Indeed. It was always the plan to implement that low-level redundancy anyway.

> What would seem to have more immediate benefit is to have your
> fileservers register (at least) two IP addresses in the vldb; one for
> each redundant link. That way, if one link to the site containing that
> fileserver goes down, clients should still be able to access the data on
> that fileserver.

Would it then not be necessary to have the database and file servers  
installed on separate hosts? Otherwise they would share the same  
CellServDB files (even with a single physical machines, that's  
possible using virtual hosts, although I would not be able to afford  
two public IPv4 addresses for both hosts). Or is that not what you mean?

> Of course, this should already happen automatically if
> the two IPs for the fileserver are addresses known by its local
> interfaces (that is, they show up in 'ifconfig' or whatever).

I plan to use the "iproute" package in Debian.

Cheers,

Jaap