[OpenAFS] Openafs vs Red Hat's Netkey

[Peter Grandi]

>>> I run a network of machines running Scientific Linux 6 (a
>>> Red Hat Enterprise clone).  We have both AFS and NFS file
>>> servers.  In an effort to add some security to NFS, we are
>>> using IPSEC.

IPSEC may or may not be a good idea, but in an ideal world you
would be using NFSv4 with a kernel newer than 2.6.35 (or with
the relevant patches backported as in RHEL/SL). Because that
means that you then have a fully Kerberized NFS with both authN
and data encryption if you use 'sec=krb5p'. This is rather
documented in a rather simplified way here:

  https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/s3-nfs-security-hosts-nfsv4.html

I have written a far more comprehensive checklist here:

  http://www.sabi.co.uk/Notes/linuxFS.html#fsHintsNFS

However IPSEC is a good idea with OpenAFS as even with the
'rxkad' patches it uses 56-bit DES for data encryption (if
requested!).

You may also want to investigate whether getting one of the
commercially supported AFS implementations (e.g. YFS) is
something you can afford (or your policies allow), as it can
be very worthwhile in terms of performance and security.

>>> I have discovered that IPSEC, specifically Red Hat's NETKEY
>>> protocol stack, sends OpenAFS performance through the floor.
>>> [ ... ] If you switch the client to the KLIPS stack (by using
>>> the kernel module that comes with the Openswan source),
>>> things run fine. It does not seem to matter which stack is on
>>> the server.  [ ... ]

> By "miserable" I mean transfers well under 1 MB/sec on a 100
> Mbps link.  Normal is 6-7 MB/sec.

Even 6-7MB/s is a bit low, should be around 10MB/s.

>> A guess would be that something is causing packets to get
>> dropped somewhere along the line. Do you have any idea if
>> you're using jumbograms?

> I don't see any config setting in the files that activates
> jumbograms, and the default is not to do them, so I don't think
> I'm using them.

"don't think" is not as useful as looking at frame/packet traces
very intently.

[ ... ]

> Now it gets weird.  Iperf shows the same performance with or
> without IPSEC.  But if I run iperf under IPSEC, openafs
> performance jumps back up to normal and stays there for several
> minutes.

That sounds like a network problem triggered by something OPENKEY
does and that KLIPS does not. Something related to buffering and
packet loss in the switches or less likely in the endstations.

Probably looking at frame/packet traces at both ends will help a
lot.

> Does this give anyone any ideas?

Guessing wildly, this seems like a switch firmware issue or a
switch configuration issue. It could be related to MTU if you are
using tunnel mode IPSec instead of transport mode. Lookout for
packet fragmentation if using IPv4.