[OpenAFS] Linux OpenAFS, VPN, NAT (saga, advice, patch)

Wed, 19 Jun 2002 17:10:52 -0700 (PDT)

--0-1780791711-1024531852=:2940
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

As my previous posts may have indicated, I've been
battling trying to get a slightly out-of-the-ordinary
setup to work.  I appear to have succeeded, so I
thought I would post what I've learned (and had to
modify).

Here's my setup:
I have three identical setups at geographically
distributed locations.  Each includes a gateway
machine and a file server.  The gateway is running an
iptables-NAT and FreeS/WAN VPN on Linux 2.4.18
(SourceMage - heavily customized).  The file server is
running OpenAFS 1.2.4 on Linux 2.4.18 (also a modified
SourceMage).  The gateway is multihomed
public/private, the file server only has a private IP.

My goal:
To run a single cell on all three servers, accessible
by clients (mostly running Win2k or WinXP) both inside
and outside each site's private network.

The problems:
1. The servers sync extremely slowly or not at all.
2. AFS only passes private IP addresses to clients.

The solutions:
1. It turns out the server sync failure (mainly the
VLServer) results from a well known problem with MTU
sizes and packet fragmentation over FreeS/WAN (see
their documentation for more on this).  Their
recommended solution -- reducing the MTU of the IPSec
interface on the gateway -- appears to fix the
problem, but actually causes a different one that's
much harder to diagnose (their documentation hints
that might happen).  The solution I arrived at: reduce
the MTU on the file server machine's ethernet adapter
(I set mine to 1400).  My reasoning is to long to
include here -- suffice to say it works.
2. After going nowhere with various schemes to 'trick'
the server into believing it was multihomed (with the
public IP address of the gateway), I stumbled across
documentation of a feature included in a patch of
Transarc AFS.  It allowed you to put 'fake' IP
addresses in a NetInfo file.  I succesfully
implemented this functionality in OpenAFS, but it
turns out it was a red herring (for me, at least). 
The AFS Windows client obtains the IP addresses of the
file servers for a given cell by essentially running
the command "vos examine root.afs", and looking at the
replication sites.  (I haven't looked at how
non-Windows clients obtain this info.)  So, my
solution was to modify the AFS Windows client --
adding a routine that, after obtaining the IP
addresses from the server, checks for a file called
afsdipmap.ini in the Windows root directory.  This
file has one or more lines in the following format:
<private IP> <public IP>
For each private IP address obatined from the server
that appears in the file, the corresponding public IP
address is substituted.  
Adding this functionality requires changing the file
afsd/cm_volume.c, which results in a new
afsd_service.exe.  I've attached a zip of my modified
cm_volume.c file, in case anyone's interested (note: I
was not going for style/continuity with my changes...
:-) ).  It's taken from a OpenAFS 1.2.3 source tree.

Thanks to everyone who helped with my previous posts
(particularly the tenacious Derek Atkins).

Matt

__________________________________________________
Do You Yahoo!?
Yahoo! - Official partner of 2002 FIFA World Cup
http://fifaworldcup.yahoo.com
--0-1780791711-1024531852=:2940
Content-Type: application/x-zip-compressed; name="cm_volume.zip"
Content-Transfer-Encoding: base64
Content-Description: cm_volume.zip
Content-Disposition: attachment; filename="cm_volume.zip"

UEsDBBQAAAAIAEWQtSzedCqIPg8AAFkyAAALAAAAY21fdm9sdW1lLmPVGmlv
2zj2swPkP7BZTGo7ztU9PqwnWaSJU3jXOeD0GCANDFmibU0kUSvKdTLT7G/f
d5C6LCduBwV2/cGWSL6Dj++m99ubG6ItTlX8mPjTWSreHBwcdEQ/SmUSOamv
IicQb+faj6TW4sJxZ/gE65NYJTQvnMgTKp3JRO8RrpMgEEPEpcVQapl8kR5P
0Nf7ma+FVpN04SRSzBwtxlJGIpGBdLT0xDzyZCIAnQAOQi3UhF76by/E9Xwc
+C5hGfiujLTcE+JcJcKTqeMHuiO0lLR60D/tXd70xMQPpPAjRqfi3UB+kQFQ
nyeuJDyen0g3VcmjADQqCmBzwknFLE3jv+/vLxaLPRXLyJnoPZVM971gP2DC
hwd7szQMEMf+5sbmxp/8yA3mnhQ/w9r92EmccG92XB3XqadpGCcmsNGJOPvn
u+vr4rqFH3lqoSvQMKqVe/+mMhyN/3zAQzLQsjijH4EcQMjUzEeePxH7bfGK
KBLbhdVp4kfTCvLQCQLlVgaV9isjycN+8pDtyoxuwXa9vdkWDgLIKFkAqvtR
Ktxw9EUF81AO4L27uZG9w1wbXoDmR3rXcReBvyjfQ6B+5Kc80cSh1ubG75sb
DZ2CCroCKajIRRz4A4AN2GwTh6/gvbmNo62WQJAGMYLofCfwf5PDT8hJc7vE
WEdsuaGYBmoM2s/DAuG2WoBbVD5lrsWRuPwwGCAPDWSgF3kFHgrgT5sbTyQz
UANUu4uTX0aXHy5G/evRzfuh+MtBeYqHR4PeJZgowvlRKqYyxYnQiXXTnTmJ
4IPUt3e3ZZi7jsD1ofNwOQ/710Z8oA5Dmc6TSJOFRPNwDLYHFte/Fo7nJWjy
iLsrdg/FBCxEJgl8o+40iNwC7Of2zV//dkciBwJpICN8DlQ0Fa7ypJ0w4/jo
g4gO7MuvxRfXPjAdM4OU2iI+B1uOnXRWGYqckIic9wc9M0SK08hmAQ+pox/D
Xvb8yN8iFMAczLyT6Se2uDPrCpq4K/AloBxqQi+tllUphgLGxNevtD9xXF4I
jgwFCgJDEJYxiFKkCXgoOBoxdtx7HTh6BicygdX/ngNdj2WK0gOe4BDhgRFa
wiRpnN89vBOvjsTrz59ft3Cl66SG4a3Pn0k/G8tYxI59zaSCY8RjJliAISVq
t9jym7Awoy8Ky1jDRQtVPN8uOpeCfqB1xqCM6aSZgQKPP+mf9FZHMMc5M0ZY
NAB8TNDrluAS3tokkTIfr3C3Pmek+I5X0HOJ0ePej2M8o8XMT6XQseNKBoEB
QN8E+SBzYHUuMwHnDWfRuzoXxrkQM698TbBNt2XHG9Yw/bvbX3d27qysW6Ty
Bu5X3EDF1i18w5rE0nbgM06kc8+InvCbvjAiCIP3GBS2jhPk4/Xng9eGiZ0d
fsiM0jDmI2PWd7B0LVFkxnWiSIHRPrgSNDlbx9whL0/2mFxEhOLa3i5ztZIp
wxNimLiB0ub085NnSVROnE/ZvPtd42n3Odf5EHsOHK9jHDvkEKeg7+D4Zirw
tBl9rcnhC4g2Qbx7HD609kywZ9cWjhiNiUnA/9yl6OZKSH/a+B138H0OGRDG
NvzlEbB5HICfuANcliIgUmPnXIgurooiCo/wi1HRTpEz7TIKyrOSoZzgwlQn
xXXZNM8VpwzfmCmgIYwwV8KXE3gpLKM9p5PAmerC6HzE4zKMK+sN2i+BN5ZR
CtkVPvXwqYu256nodSpAIuLy4+DsrXiUaRcj0ONrSAlRlVwVhpBZkgbhLodX
rprDr1HLUnShQNS/voA4dVuOoUtBsGvQQaDD5fY17FLAAMYCxU4Bg56YJCrk
DJL4MEDAQzHqMuFOmXArQ+hCVgsRdQ6JSeCJMUQaVPOSU8HDAqSsZ8nihk5K
x8YwqsOwEgF2jyP5kJKFwOlez1Oeb/IcnzbZCHtMHG5ZO6qlq+rpqh9Md3xf
Szcf/uN0c62E84jUQiyw7PgibV7HujpPOMMDB4b1zEwGXhfU4F6K4fUp1A6o
CIHwUzo6i9BTzLTJJYCzUzDRt48XhvsmuYHd4y+B3U5HGD9grR8/Ypssuy63
tDlHC9Q9Sv1oTipPieVATQ+bmNeAFU4B49bpCYS+j4MRpDRkam8fLyG0XgnK
gTDqEjkjeByspWj2soynSUzuHrsOubYCmo7YzuybpS7MWYNETqCAfPxNMnR5
+xSsy9+01yJbuWjBzj4O4DB66PBJKIyGja0qMErSjE6hx5E0iFkuRPtYQQln
4hOVA1djKCCjTwmE/UolsOpQsg3vkVcU2yCx89HwU++X/s37m9YyUGbJ/TPU
/gycSfW92wNyTw0u5J6HPvg2pq5eZEqtYurwrobU8yyq72Dx7cnpv15icny/
isk3a0jOQHNWQ2c+5I7DuocO5U/TPzroCv/nnIXI2DWM7uzYBAvdPuRGlGKw
ZxIcBE26xnG0tBVedo7jkAIRj5TdpLmQbs5HZ1eX7z/c9Cq+oJEF7D3tR6OJ
E/rBI2A/OR/1L3vvzRoTo2F8lqooaFZpX1LtB8TZizaw2muGuN/wZxsuRZhv
ktnLsB5BMgJREXMIExJv37TDu5bJFYv06xbuHFq6TzVbcuiBfjAc5OkGrYzZ
PZxDbDVBYTsD74jTi9FNb/ixNxxhgdjKRfsKIGuUzX5yxJdy8SJe68kbAEk+
32QApAw+drCw6cW6sHC00Q9ICB7FBCdSnosTOdFWTbiUADbA5SJKSJWZSmHo
iKmVPGD1g7HCgQInSZu4p1dcI9WpuP2snsFiFqIjFrQQCiEqokMNfJ02dWvP
Mo5pNoqiLXqOOxMYL3BpUZSQpzZboAkghxBsg9oPFghkIDjbw9afjiE/fz+T
Gh33RCYygppsgbF4LIWXqDiWXseCgsiQvcVMcsvPRHiQPTh9zDGxP0iPFFcs
GKRogQ8nQ41CWDyZRy52NvfMgvw8KgY5/PTxasAqjtKBoualuJCZj8mCqkJB
pezmKzjtqckOcc2qQ1qRMmb2tQquboNX627w6ts3SF091EueHaAabVfzzo7I
cjn4mDqAasJV23iydvdJFgoN2g6GGdiQePWPV6U8rvr5I0ddCGVrymIVF3Ua
kCfGzwDaz4pcuuhoQUxnYEQmMbL2pSgdYv9Udq0taw2VLLx2I0/sA9kfgJ0N
wThV6P8GOfUVeQ0eN7N9bvRP/ERDhWa8peZUHSc0prF+DEjuOzgQYSJvYBPC
DGEv9qFmV9jED6k0ncmQXEKoAEvussZy6kcRtrwNgkxXxnSTgesSuasSexex
mKmAPR3Aus5cSwNoMsrU3lPQEvAhc7zLgED7yO7f7svEAF8bcE8tInBwQJ7W
esp4IMVkjZy6uFXw4ejMcJh4NBhwXgu8SwlkNE1n4tDsic4JNdkWz8fi0Gok
nF52GDa4LRWCpCe2fWN6KaboppO1jRCoE7gL8vaxf9Y0LRDqV3AThNaZZO2s
pieySQ2MYlckvyXAoTaUz0AgNo2R5YZJQfWoGQBjWLHcUE/p9vDgoJjEFnsH
xdEXqwBMiagbxJZcvKggS+PvI1MXgdHakpZKEg7X1oz5dXu7Jl1tNueR9qdg
fa1MaDkgFQBfv9bAiefh1Aq47PMsOGbPreeyJduBtANPtgGCPbVSLJ6gLk9A
Ib1qNU3lAfa/LM9ycrrk7b8ldWcObBcwpaJ+BUU6qIISXMxT+WBtInyoYG1Y
wN1j6/ohH4QA8eGidz44eTca9m567zNjy0rYUsvQWEepICZm8hy1Wsja9ogh
eiT+U0O24N6zPnCRj+USqGazVfqEIDNEE45oXdkxWCE9FeVln+AwyHMtfA21
O6iB53vocyc+K4NWWNnoeZCSxlBrsHhc9j6hZN94pzDf6mRKW988KPgoBLXC
r2Ba7s0IcdARmf8poF7PHxKtVb1hdFVtZvuS2iibeKFp1yIvWdc4Hwbe8tYx
USRl+AMus+gQ7Rhbjp9dknJU4nu/XIuIg7W7Jz/AfdL9Vxg3i0Is9bdKxlP1
UfZT0NU11HYKxQN1hjP15HKy4EfMLs0tmrke5NY+2ZZFHMoQShESSgf1bPXK
4rbzmq8yTUJblm5hnVu9qbZWXEGEsitsgG8OC0Ju7RyW2MNjiB+bBWhrj5e2
zZitrVy/V3wPXbznuhri7FaNKGxsEHghhrqqUydJ8d8ik2U3XwZl5/n1qM5l
I6GCDtAV2u/5e6NMPLuasvNsNpSw65maAxsLldzDuWJSHSlKTF0ap6x2qtIU
0tg4TTgvxR5B1ox+lOXouE7gWz+CfUsA+674VRM+SvRWxy5OHZbg7RHUBKE6
ET2z/RrXbf9lcq4SV5Z3GY4mvoduE37WuNHLHW4pC+6ucMKNcjrayEycq65T
eKWsGomzAyDxksfhRpPZTtcmXFo6iTvLKgimuKxJ/xu5Lm/L8LhOkrsC4MXs
th7u/yutZcpzvr02RFCpQsQ2VCp9JyN2Sc+7gZIhrvaD32P2y1a/ptG/YLq8
0et5/hcwJmiu9TmDxONw4jhRkCaiiGwHgaK1ve0nqbmV+/Isabux93ZVU4UN
m/KoWM7mxl5Cp/MWTe5qlzS8oHcNC7Gi05f9laNO8Z/Ho9bCQ4bwHJ5S34nx
YF85diLfbW6NHc+KFwzRj0RVoJCfj6hRPhrh06B/CU/lkGSfjHcuyLDkoXMd
WDojcxrruLdCT7xslLu7+K+UPJa+YJJW/wzT3AxyvF1q5vDSjmkAJdSHhgyg
I4xrppULxJx7kU38g0qk0uqfTUCawyveOPdYVmwdVvtet5qa1zmBPAMg7wnJ
dFVjyH5B6kVls6df0JXiigWveNGaK4dNLJeO+XQm3Xvery7+4bMuhq7wgpbG
jwx3ZeWpbUt/z13j80f3nYnt86w9ky+Vln/bTfk6dvZUOaq1ouAnaTNsJ9DY
q50kUuNdkzvDv6BRaRArH6+W0IwKfwBrtwVmVlrgtRSjxusnJ8U2ru9hI7Z0
XYfd40RSJ5iuqfyEcJjOrHBcF+bwT4OmtQxlfTQFHhInusemMKMBe0Y+cvXG
RdjK/pg5M14Ies2fNt9Poprhf5Q4De+WTaDdyE3ARkG7pcXMhzQwpP/0U7UB
HLsqfsT9MSnKH36MdQAQ2YfJIQ4F1WhZBxuEnfXJIyswbljXWNR69/Uvte82
N+wVXnZ9emR4Nok03elUfSA5QTwP3krp7G5WdM7xziq7Ad5vszoxt7xvugIu
bZsve4kGF0vumh36qnhqe3nrHPJaMfFp/bj4X1BLAQIUABQAAAAIAEWQtSze
dCqIPg8AAFkyAAALAAAAAAAAAAEAIAC2gQAAAABjbV92b2x1bWUuY1BLBQYA
AAAAAQABADkAAABnDwAAAAA=

--0-1780791711-1024531852=:2940--