[OpenAFS-devel] RX retransmit timeout value being overestimated? (Poor performance over WAN)
Peter Wells
peter.wells@lutraconsulting.co.uk
Fri, 27 Apr 2012 19:11:06 +0100
This is a multipart message in MIME format.
------=_NextPart_000_00DB_01CD24A9.7F562F40
Content-Type: multipart/alternative;
boundary="----=_NextPart_001_00DC_01CD24A9.7F565650"
------=_NextPart_001_00DC_01CD24A9.7F565650
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit
Hi all,
I'm using OpenAFS over an OpenVPN WAN and am experiencing some strange
issues and wanted to bounce some ideas off the developer community before I
started looking through the source-code.
Below is a graph of data transfer between a Linux fileserver and Linux
client. It's a 512KB chunk written out from /dev/random. The capture was
taken using tcpdump on the fileserver machine.
I'm trying to work out why the fileserver keeps pausing like this.
The last packet before the pauses is an ACK from the client with a mixture
of 32 +ve and -ve acknowledgements. then silence between the server and
client for 1.2 seconds.
. then the fileserver transmits all of the NACK data packets before the rate
picks up again.
As the rate picks up, the client will NACK a data packet, and then
subsequent ACK packets grow in length (in terms of the number of ACKS) until
they reach 32, at which time there is another long pause.
rxdebug -rxstats says:
Average rtt is 0.104, with 17838 samples
Minimum rtt is 0.000, maximum is 2.147
That's a pretty large maximum rtt and I was wondering if this was somehow
skewing the calculation of the retransmit timeout value, somehow causing the
fileserver to snooze before suddenly realising it should be retransmitting
packets.
Any thoughts you have will be much appreciated. The AFS versions are as
follows in case it helps:
1.4.12 (on the server)
1.4.10 (on the client)
Kind regards,
Pete
Peter Wells
Lutra Consulting
23 Chestnut Close, Burgess Hill, RH15 8HN
<http://www.lutraconsulting.co.uk> www.lutraconsulting.co.uk
Tel: +44 (0)1444 848012
Mob: +44 (0)7704 781683
SIP: 1507403@sipgate.co.uk
------=_NextPart_001_00DC_01CD24A9.7F565650
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta =
http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii"><meta name=3DGenerator content=3D"Microsoft Word 14 =
(filtered medium)"><!--[if !mso]><style>v\:* =
{behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DEN-GB link=3Dblue =
vlink=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal>Hi =
all,<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p =
class=3DMsoNormal>I’m using OpenAFS over an OpenVPN WAN and am =
experiencing some strange issues and wanted to bounce some ideas off the =
developer community before I started looking through the =
source-code. <o:p></o:p></p><p =
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>Below is a =
graph of data transfer between a Linux fileserver and Linux =
client. It’s a 512KB chunk written out from =
/dev/random. The capture was taken using tcpdump on the fileserver =
machine. <o:p></o:p></p><p =
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal><span =
style=3D'mso-fareast-language:EN-GB'><img width=3D592 height=3D125 =
id=3D"Picture_x0020_1" =
src=3D"cid:image001.png@01CD24A8.53EAE0E0"></span><o:p></o:p></p><p =
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>I’m =
trying to work out why the fileserver keeps pausing like this. =
<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p =
class=3DMsoNormal>The last packet before the pauses is an ACK from the =
client with a mixture of 32 +ve and –ve acknowledgements… =
then silence between the server and client for 1.2 =
seconds…<o:p></o:p></p><p =
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>… then =
the fileserver transmits all of the NACK data packets before the rate =
picks up again. <o:p></o:p></p><p =
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>As the rate =
picks up, the client will NACK a data packet, and then subsequent ACK =
packets grow in length (in terms of the number of ACKS) until they reach =
32, at which time there is another long pause. <o:p></o:p></p><p =
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>rxdebug =
-rxstats says:<o:p></o:p></p><p =
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal> =
Average rtt is 0.104, with 17838 samples<o:p></o:p></p><p =
class=3DMsoNormal> Minimum rtt is 0.000, maximum is =
2.147<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p =
class=3DMsoNormal>That’s a pretty large maximum rtt and I was =
wondering if this was somehow skewing the calculation of the retransmit =
timeout value, somehow causing the fileserver to snooze before suddenly =
realising it should be retransmitting packets. <o:p></o:p></p><p =
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>Any thoughts =
you have will be much appreciated. The AFS versions are as follows =
in case it helps:<o:p></o:p></p><p =
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>1.4.12 (on =
the server)<o:p></o:p></p><p class=3DMsoNormal>1.4.10 (on the =
client)<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p =
class=3DMsoNormal><o:p> </o:p></p><p =
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal><span =
style=3D'mso-fareast-language:EN-GB'>Kind =
regards,<br><br>Pete<br><br><br><br><b>Peter Wells</b><br><br><b>Lutra =
Consulting</b><br><span style=3D'color:gray'>23 Chestnut Close, Burgess =
Hill, RH15 8HN</span><br><a href=3D"http://www.lutraconsulting.co.uk" =
target=3D"_blank" title=3D"This external link will open in a new =
window"><span =
style=3D'color:blue'>www.lutraconsulting.co.uk</span></a><br>Tel: =
+44 (0)1444 848012<br>Mob: +44 (0)7704 781683<br>SIP: =
1507403@sipgate.co.uk<o:p></o:p></span></p><p =
class=3DMsoNormal><o:p> </o:p></p></div></body></html>
------=_NextPart_001_00DC_01CD24A9.7F565650--
------=_NextPart_000_00DB_01CD24A9.7F562F40
Content-Type: image/png;
name="image001.png"
Content-Transfer-Encoding: base64
Content-ID: <image001.png@01CD24A8.53EAE0E0>
iVBORw0KGgoAAAANSUhEUgAAAlAAAAB9CAYAAABtT12EAAAABHNCSVQICAgIfAhkiAAAEfFJREFU
eF7tnQty2yoYRvGdrsbycip3ObaX06jLsbwdX7CjxJFARhJvjmY60yAEP+fj8YGUdneXl+CCAAQg
AAEIQAACmRLY7XYitJ35L1NWhA0BCEAAAhCAAASiEcBARUNPxRCAAAQgAAEI5EoAA5WrcsQNAQhA
AAIQgEA0AhioaOipGAIQgAAEIACBXAlgoHJVjrghAAEIQAACEIhGAAMVDT0VQwACEIAABCCQKwEM
VK7KETcEIAABCEAAAtEIYKCioadiCEAAAhCAAARyJYCBylU54oYABCAAAQhAIBoBDFQ09OVXrP5l
WC4IQAACEIBAugR6cTnshFqv1J9j9xlpfxGHz7THvcNF9KNGYKDSVZXIIAABCEAAAhDwSaD7J8Tf
++O/gbl/tKI7HsXgoYRoxYdKV3+uJ9GM4sBA+RSGsiEAAQhAAAIQSJdAexKnwRm1rbRMN3EbHzUZ
os/CQPEqyKBeBslol4FIhFgEAcZaETLSiJgEemmemlb8/jpq6sTx8RrvIC4aU7WTR1P3mPHa1B3j
f1m2iYs8ZgKDZmhnZsQdCLgkwFhzSZOyciOwZAOhtz3qW6iD6E93Id/k/bj6y0Ecznv5Ou9DnlB9
X8kbqAGKvsG5SVxPvBioerSmpWkQwECloQNRxCGwtf93R/kBufh4fAc1vdRJ1EU01+v36z6ZKYtX
eNPGkAIBCEAAAhCAAAS2E1AnTGbzZC4/CwOlTp+WHM+Zm8sdCEAAAuUReD3xLa91tAgCHgl0R3Ho
WnHVnjw96338Zt6Pb6Oe6Um/wns9ktt6POcRP0VrCKCdBgpJEPBEgFfmnsBSbDYE1nmE53dP59EH
4u3HVTTyVOorvTmLq+afMcBAZdM98goUA5WXXkSbNwEMVN76Ef12AusM1LZ6s3iFt62JPA0BCEAA
AhCAAATcEkjWQMVwk27RUhoEIAABCEAAAqUSSNZAlQq8hnZhfmtQmTZCAAIQqJsABqpu/YO0nt+i
DIKZSiAAAQhAICCBJA0UJxgBewBVQQACxRBgs1KMlDQkAwJJGigdNyYGHRXSIACB2gm42HCqMrgg
AIFlBLIxUMuaRW4IQAACEIAABCDgj0ByBsrFbsofLkqGAAQgUBYBTp/K0pPWhCOQnIEK13RqCkmA
V7AhaevrYqHUcyEVAj4IbBlvW5710RbK1BP4pU8mFQLrCHCCuI6b76eYkH0TpnwITAkwH06Z+EzR
zXNq8+7rSspA0dl8yUy5EBBiOAX0OaHAOU8C9A0/ur2evNuOu2EdZD1crokt4+Ul65/gFZ6eC6kQ
KIYAE3ExUlo1ZMnrcvqGFdJNmZQeSzTZVBkPByWQjIGyGch0wqB9w1iZ7pjUmJkb3gks0YMx5F2O
oBXYzJtBA6IyIwGbsYeeRnxJ3kjGQCVJh6CcErCZQJxWWFFhJhPFhFxRJ6CpyRAwjTvmwGQkchII
BsoJxvoKMS3Y9ZGI2+JhomZijqsDtUPANQHGtGui7stLwkCZ3Lr75lLiVgJotZVguOdNWjExh9Mg
5ZpM/SPlmEuIzTT+0CM/dZMwUPlhI2IdAZsJwDR56MojbZ7AmDds53lxFwIQgIBLAhgolzQrKovF
Ok2xB13G5irNaInKFwHGpy+ybspFHzccY5cS3UAx0cfuAuHrZ/LYznxu3Ci+7y40eEeI+xAIR8A0
nhmn4TRYU1N0A7U0aDrUUmLkr5GAjYmqkUtJbTYtujZt3PKsTfnkeU+Atew9o9RzRDVQDOLUu8d8
fFsmgC3PzkfFXVsCaGBLqo589Ic4OrMOxuHuotaoBspFAygjHAEGejjWczWhwxydfO4pHbnKI7Bk
fHJSnLf+GKi89cs6ena8WctH8BCAgAMC70wU86QDyJ6KwEB5AltLsQzusEov2d2GjYzaciFAH8pF
KeJMnQAGKnWFAsa35ZXC2kkZAxZQYKoKQsBmHNnk2RosY2srQZ6HwDyBX/O3/d1du+D6i4iSIQAB
CIQjMBicd69wXEU0mLZQ9bmKm3IgYEtAtzHx2d+jGShbIOQLQ2AwtCZja0pX0bHTDauRy9pCL+Iu
Y0+5rLnxEjpujFNo4u7rY5zaMfVplnQRZPkKjwVbJyVpORPQ7Zxybg+x+yWwxKCp+TL0wuK39ZQO
gTQIZGmg0kBHFK8EtkzQGGL6UikEbIyNTR5XPLaMS1cxUA4ESiWAgSpV2ZXtwsysBOfgMU6hHECk
iE0EGP+b8ImQ5nhbpDztgkDW30DZLjjswua7CoN+no/vuwP/uf6MRr5VcFv+YESYe9xypTQIpEQg
ioFysRjYTkxzi1JKQqQciwu9Um4fsUEgFgGMVizy+dVLX0lPM17hpadJ9Ig4xo8jAdzjcA9Zq4vN
iIsyQraZuiBQKgEMVKnK0i4IQAACEHBOgLcazpFmW2DxBopd/XzfZDc7zyeFu741Yoz4UdkHV999
wQ8JSoVAmQSKN1Blyua/VT4mf/9R51cDC2J+mpkiRksTmXLSB405hSpH0y0twUBtocezzghg2J4o
4eCsS1EQBCAAAa8Eghsodmle9XReOHo5R0qBlRIwjSVb02x6vlKcyTUbfZKTxHtAwQ2U9xZRgTWB
dwNeTexccQm80yhudNT+joCtOXpXTsj7OcYcko+qKxajWPWG5ptLfRioXJQizmoIMElWIzUNzYgA
m5mMxAoUapR/SDNQ26jGAQFOoRxANBTBhGwAQ/KEAH1lgoQECEwIqHEyvnyuYRioMW1+hgAEILCQ
wDuD8+7+wurIDgEIaAj4NEua6kQVr/B4JTKVngl9yiSllKHPhpoQGCP+1N+qIWPVnzZbSmbMbKFX
xrNeDJTuGE3hYiIoo9P4agUTki+ylJs6Afp+2gqxdqWtT6zonBuooaOZTFSshlIvBHIjsPXkIrf2
lhzvWi1ZuEvuFbQtdwJODRSDPY/ugE7xdUKD+BoQAQS2EhhODkOOZ04rt6rm7nlnBmrcgRDZnUiU
BAEI1EdgPKfWR4AWQyBtAs4MVNrNJLqBAJMyfQECEICAPQHmTHtWteV0YqBMHYxTqNq6E+2FQH0E
TPPfFhI+ylwSD3O3Pa2137fZ10DOVAlsNlC2A902ny9QTAj8FqSvvkW5EHBBgDnKBUXKgEA4ApsN
1Dv3zaQQTswSaqqhv8TeTJj6SQ3sTW1PKT3V/pESo9RiebcOphYv8bghsNlAuQmDUiAAAQhAAAJp
EUjVzLLZSaOfBDFQuPP4Yqc6EcQnQwQQSIsA82VaehANBEwEghgoVTmTgkkC0iEAAQg8CTBP0hMg
kA+BYAYqHyTlRcrpUzqaokU6WhDJPIHaXxMxVuf7B3dFHf+ZMEJDAAIQ8EGARdYHVcqEQB4EfuUR
ppsohx0Vx+RueFIKBCAAAQhAIBUCakMzvnyu91UZqDHYGn5mh1yDyrQRAhCAAAR8miUdXb6B0lEh
LSqBUr+9yMHMlso+aofOuPJa+wNjNeNOGzB0DFRA2FQFAQhAAAIQgEAZBDBQZeiobUUOuyht4CRC
IAMCjK8MRCJECHgkgIHyCJeiITAQYLGlL0AgDwKM1Tx0SiFKDFQKKhDDhECt315MQERIgP176DUt
svSH9/0hVg60iUX+WS8GKi5/b7XXNMF7g0jBEIAABCAAAQOB6gwUjt3QExJMRqt4osDezJ7NiZkN
d8ITYKyGZz7UWJ2BioeammslwIJbq/LltLuWRZqxWk6fDdESDFQIyoHrKGkSqGXiDtxFrKqDvRUm
MkEgOgHGahwJMFBxuFMrBCCQIYGSNicZ4idkCAQk0IvLYSfUmN/tjqLT1FylgRrcugLDlT4Bdlfx
NIJ9PPbUDIElBBirS2i9z9tf/ojz/kMortfzTRyPUwtVpYFS6BSUEo0UO+T3AyNkDvQISdtvXbVr
WfoCXbu+fkdPbqX34l/Xi7ZtH4E3v1vRdN3kFKr6/0xYTQrq4jQq7Q5e+uSdMn3Yp6xO+NiYK8Mz
t62xhrE6rNm2TNblu4m+b0Sz/3xa/mUv7dOtlz833yVWb6AGFGFEWSclTz0JoFG8ngD7eOxTqpl+
kJIa+lhq1sjW3NsxkqbpxSzpaFsZKNugdBWQBgEIQAACEIAABHwTsDNGtlHcJidO4yetDJQuqBDv
i6ljLJf5Z1iZ2YzvwGpMxPwzrMxsxndgNSZi/hlWZjbjO7AaEwnx8140r6dPvTRT8iVeOzqRqvYj
8hASUAcEIAABCEAAArkRaIT8blycL8/fvOv/daKXH5Q/Pyn/bovVCVRuTSdeCEAAAhCAAAQgsJZA
c/orzoeD/AUzWUJzFtfr2D7JXz6Tr+eev4a2sBaOFe2BwQpW9gTsc9KvYGVPwD4n/QpW9gTsc4bo
V/bRuMnJKzw3HCkFAhCAAAQgAIGKCCRtoFYejiUnH+1ISxL0QA8fBOhXPqiuLxM91rPz8WQperyy
SdpA+RCRMiEAAQhAAAIQgMBWAhiorQR5HgIQgAAEIACB6ghUb6BKPFbMuRejR1rqoQd6pEUgrWgY
H2npETqa6g1UaODUBwEIQAACEIBA/gRW/zMG+TedFkAAAhCAAAQgAIF1BDiBWseNpyAAAQhAAAIQ
qJgABqpi8Wk6BCAAAQhAAALrCGCg1nHjKQhAAAIQgAAEKiaAgapYfJoOAQhAAAIQgMA6Ahioddx4
CgIQgAAEIACBigloDFQvLoed/B+I1Z+j6LRwbPJoHyRxMYFX1jtx1AqCHouxbn2gO8rxcRCXXlcQ
euioeEt7aPE5Zx0uYioJenhjPym4E8dBC8bHhE6wBO38ZDMObPIEa0XyFU0MVH/5I877D6H+gbDr
+SaOmhXbJk/yLc8lwO6fEH/vDz3uH63ojlNTix6hxZSTTHcTjaFa9DCA8ZHcX8TheBPn6+cYuZ4m
uqCHD/D6MtX8dDtfP+ervTj/mRpa9NCzc5P6aYDkRrsdFWjD3SaPmzjLKGVkoHrxr+tF2z7RN79b
0XTd6BTKJk8ZcJJoRXsSp2Gllrq04iZuP7bY6BFcp+4iuqYVe23F6KHF4imxu5yFOP/9HiOTetBj
gsRbQi9ut0a0vz8nrH0zMbNCng++X2O8BVhBwY04qc2E3Gz/vGy42+SpAOGCJo4M1E30vez0w8og
/7KfLNhzeWxeNy2IjqyjMSDNk1y4h/npeRM9wnYT2ccvQpxOevsk5HgxjyHGh1ut5Oaukwu2uHy9
wpsemKOHW+ZzpcnFW46L80GdkstXeYez2J/GJ4LoMUfQ37057kOtc3mYu3Ta/JomStNkejfxldmQ
R+7MH6//rmP3O62FlKUEZAf+oyaku2ZXhx5Laa7O/+jjrbjLArSfoz0KRo/VfFc82EkLdb9/SEHk
t1DqFbf8+88ZCD1WYF33iDwxPzcH+R2UGh2tmByEMD7WcXXylGEc/CjbkIe1XavA5BsotYP++YpI
95whjzqylZPYQf9lra4g0iwJdMfDw5zqJyT0sMS4MZs6fZLf25zebRDQYyPoRY+38lODx/VYvNWp
1Phx9BgT8fOzOqU4iF5u8p7fbApppKbfbBrXGNYPP7J8lWoYBz9qNeRBG602IwO1F83r6VMvYU52
0zN5mpO4yo+d/4o/jyP16XG6NgYS3xDoL3JHJ+TJntY9occbfA5vqyPuXr6iGH5DVf19/Jt46OEQ
+IqiXj5BeDyNHisgrnxEjQ/5neawv3gY2vGCjB4r4W58bIb7V8kzeVjb9fzlTuHHdT03d9F+PNJe
/y6PyO/yk/K7/G2XH+k/83wXpdIblZlrG4GPVr6zO9+nJNFjG9itT3/zf5aEHluJrnle7im+5iu5
wZBvVls5U6kLPdbw3PaMYi7un8uHWiju8gs19NgGdeXT4/lpbt1mrKyE/DhqHV3X+7l5fOIxWrhf
BTHkeUxgn89+DZxx+fxsT+CF8xfXYYJCD3uOPnKOJyj08EH5fZmvY+S5wXs+gx7v2XnI8TBNwxrw
YqZeDK1cyvVrDOuHA0E0a8bXBtzAHW1Wc9+pJ/VnU6RCAAIQgAAEIAABCOgIaD4i12UjDQIQgAAE
IAABCEBgIICBoi9AAAIQgAAEIACBhQQwUAuBkR0CEIAABCAAAQhgoOgDEIAABCAAAQhAYCEBDNRC
YGSHAAQgAAEIQAACGCj6AAQgAAEIQAACEFhI4H961AkZWYPRRAAAAABJRU5ErkJggg==
------=_NextPart_000_00DB_01CD24A9.7F562F40--