[OpenAFS] Multi-homed server and NAT-ed client issues

[Jeffrey Hutzelman]

On Wed, 2013-07-17 at 17:43 +0300, Ciprian Dorin Craciun wrote:
>     Hello all!  I've encountered quite a blocking issue in my OpenAFS
> setup...  I hope someone is able to help me... :)
> 
> 
>     The setup is as follows:
>     * multi-homed server with, say S-IP-1 (i.e. x.x.x.5) and S-IP-2
> (i.e. x.x.x.7), multiple IP addresses, all from the public range;

Things get much easier if you just use the actual names and addresses,
instead of making up placeholders.  Frequently, doing that sort of thing
hides critical information that may point to the source of the problem.
For example, in this case, Linux's choice of source IP address on an
outgoing UDP packet sent from an unbound socket (or one bound to
INADDR_ANY) will depend on the interface it chooses, which will depend
on the route taken, which depends on the server's actual addresses and
the network topology, particularly with respect to the client (or in
this case, to the public address of the NAT the client is behind).

You also haven't said what version of OpenAFS you're using, so I'll
assume it's some relatively recent 1.6.x.


>     * the second IP, S-IP-2 (i.e. x.x.x.7), is the one listed in
> `NetInfo` and DNS record (and correctly listed when queried via `vos
> listaddrs`);
>     * the first IP, S-IP-1 (i.e. x.x.x.5), is listed in
> `NetRestricted` (and doesn't appear in `vos listaddrs`);

So, the machine the fileserver runs on is multi-homed, but you're only
interested in actually using one of those interfaces to provide AFS
service?  In that case, you use the -rxbind option, which tells the
servers to bind to a specific address instead of INADDR_ANY.  That
option needs to be passed to each server process for which you want that
behavior.


>     Thus my question is how can I resolve such an issue?

Besides -rxbind, there are a couple of other options, depending on which
components you control.  For example, if the NAT is your home router and
you only have one or two AFS clients behind it, you can assign those
clients static addresses on your inside network, and then configure your
router to remap the client-side addresses on both inbound and outbound
traffic, mapping each inside host's port 7001 to a different outside
port.  For example, my router (running OpenWRT) installs the following
rules:

### Static ports for AFS
for i in `seq 50 249` ; do
  iptables -t nat -A prerouting_wan  -p udp --dport $((7000+$i)) \
    -j DNAT --to 192.168.202.${i}:7001
  iptables -t nat -A postrouting_rule -o $WAN -p udp -s 192.168.202.$i
    --sport 7001 -j MASQUERADE --to-ports $((7000+$i))
done
iptables -A forwarding_wan -p udp --dport 7001 -j ACCEPT

(in OpenWRT's default configuration, the 'forwarding_wan' and
'prerouting_wan' chains get called from the FORWARD and nat PREROUTING
chains, respectively, for traffic originating from the internet.  The
'postrouting_rule' chain gets called from the nat POSTROUTING chain for
all traffic).

So, when 192.168.202.142 sends traffic to a fileserver from port 7001,
it comes from the routers port 7142.  And inbound traffic to that port
gets sent back to 192.168.202.142 port 7001, regardless of where on the
Internet it came from or whether the router knows about the connection.
As you can see, I do this for a range of 200 addresses, which are the
ones my DHCP server hands out -- anyone who visits my house gets working
AFS, without keepalives, and even when talking to a multihomed server.






> 
>     I must say I've tried to `iptables -j SNAT ...` outgoing packets
> to the right S-IP-2, however this doesn't work because SNAT also
> changes the source port.  I've also tried to `-j NETMAP` these
> packets, but it doesn't work because NETMAP in the `OUTPUT` or
> `POSTROUTING` tables actually touch the destination...  Thus if
> someone knows of an `iptables`...

Well, you can give SNAT a specific port to use.  Or, you can play games
with routing tables to give AFS traffic a routing table that doesn't
include the second interface.  But that's functionally equivalent to
using -rxbind but a lot more work.

-- Jeff