[OpenAFS] idle dead timeout issue?

[Jeffrey Altman]

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig7005DC452F315BFDA43A4C44
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 4/4/2012 2:00 PM, Jack Neely wrote:
> On Tue, Apr 03, 2012 at 11:21:10AM -0400, Jeffrey Altman wrote:
>> On 4/3/2012 11:13 AM, Jack Neely wrote:
>>> Folks,
>>>
>>> What's the status of the idle dead timeout issue?  We are continuing =
to
>>> have issues with 1.6.1pre2.  I've seen a lot of git activity and am
>>> wondering if the idle dead issue has been resolved at this point.
>>>
>>> Thanks,
>>> Jack
>>
>> What are the issues and why do you believe they are idle dead related?=

>>
>>
>=20
> Because Russ told me so.  ;-)
>=20
>     https://lists.openafs.org/pipermail/openafs-info/2012-January/03752=
4.html
>=20
> See the bottom of the email.

Indeed read the bottom of the mail.  Russ identifies two issues:

 1. idle dead timeouts

 2. vnode lock contention tying up all of the server threads

1.6.1pre2 contained a fix for the idle dead issues.   It does not
contain a fix for the vnode lock contention issues in the file server.
Nor does it contain a fix for client side "rx busy" packet processing
that was added in 1.6.1pre4.

> I've grabbed 1.6.1, read the release notes and saw some notes that
> probably apply to this situation.  I'm still unclear if the OpenAFS
> folks believe this issue is solved or just better.  In any case there's=

> nothing like tossing it on one of the web servers and giving her a spin=
=2E

That depends by what is meant by "this issue".

The client side idle dead processing issue is fixed to the extent that
we are going to fix it.   A request to a file server for a
non-replicated object will use a timeout of 20 minutes.  If your file
servers are not responding in that time period (unless your data is
offline in an hierarchical storage management system) then you bigger
problems.

If the request is against a replicated object (read/only) then a shorter
timeout is used but a retry will not be sent to the same file server
that failed to respond in a timely manner.

If "this issue" refers to the limited thread pool size in OpenAFS file
servers and the fact that all of the threads can become blocked on a
single vnode, then the answer is 'no'.  This is not fixed.

If "this issue" refers to the fact that a client can receive the
response to a call and issue the next call on the same call channel
before the file server has finished cleaning up the prior call which
results in an RX BUSY packet being sent to the client, the answer is
'no'.  There is a race in the file server that has existed since the
beginning of time.   Simon and I have researched several solutions but
none of them are appealing because they all introduce additional
contention with the file server's rx listener thread which slows down
the file server's ability to process incoming packets.

If "this issue" refers to the fact that the UNIX client was failing to
unconditionally retry requests when an RX BUSY packet is received, the
answer is 'yes'.

If "this issue" refers to something else, please specify what you
intend to mean.

The fact is that I cannot root cause your performance problems by the
data provided in this e-mail thread.   No one can.  People can make a
guess as to what the issues might be and that is all.

> Performance appears better compared to our other web servers, slightly.=

> However, we are still getting periods of time where AFS takes multiple
> seconds to 30 seconds to respond.  Then suddenly, all hanging AFS
> transactions return at the same time.
>=20
> See the graph.  The Y-axis is the number of httpd processes, the X-axis=

> is the number of seconds past 13:00 today.  (Data gathered from the htt=
p
> logs of how long each request took.)
>=20
>     http://www4.ncsu.edu/~jjneely/web-apr4-1325.pdf

So you have more httpd processes issuing requests to /afs then the
maximum number of file server threads.  If those threads block (perhaps
waiting for callbacks on a client to break or on other threads waiting
for a callback to complete), everything will halt until the active
threads can finish their task.

Jeffrey Altman



--------------enig7005DC452F315BFDA43A4C44
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)

iQEcBAEBAgAGBQJPfJm8AAoJENxm1CNJffh48QUIANQHoucQcuNDCeB3iI5q5yu5
bW3vLT409BoKqbKbQsv/shqD5x4ZrFbIwolmKo7ks2mDGup+DVrOVg3qnKNzYBCU
+TsQqui5ZJAG/O3dYaBwcPidQoTLhxxrUqUU7x2OUGjkhLMUbaIAy6uvnryLmiBf
lE7REluM+RFHHw67zfQx9pR0oCm07LxK2q1DG+9JvomgN4oz7tRY4Q7kJ59VPF5l
xhBQ0/M4okEDPZbZ+0b52YGitfE3xEOPkMvtiTHqDB33Gdik4ocmtHsnV/b+5skc
A3Pv0X1FC7kU6FOuoODhNtJrrfVJ0+E8raLJr/dT/7AlwAl3QMi9Z2M3qbivp58=
=LTI7
-----END PGP SIGNATURE-----

--------------enig7005DC452F315BFDA43A4C44--