[OpenAFS] Re: Afs User volume servers in VM's

[Jeffrey Altman]

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig41E65BFE7F6B4CEABAFF00BC
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 10/26/2011 3:09 PM, Renata Maria Dart wrote:
>> How many threads on the file server?
>=20
> '/usr/afs/bin/fileserver -L -p 128 -udpsize 262144 -sendsize 262144 -vc=
 4000 -s 40000 -l 12000 -cb 1500000'

You should upgrade to 1.6 so you can increase -p to 255.  This
will help but will not address the core problem.

>> How many outstanding RPCs?
>=20
> Up to ~350 calls waiting when it is at its worst.

I suspect that part of the problem is that there are clients that are
not responding to
>>
>> Do these processes communicate with each other to synchronize their wo=
rk
>> and as such rely on the AFS cache coherency semantics?  If not, could
>> these jobs be modified to open the files in a different manner that di=
d
>> not enforce those semantics?

> [...]

You answered a question but not the question I asked.

The issue in this case is that the job in question may have more RPCs in
flight against a given vnode than there are worker threads in the file
server AND callback breaks are taking so long that there is no progress
being made.  The reality is that only one worker thread is going to make
progress on any FID.  What would make a great deal of sense is that if
GetVolumePackageWithCall() were going to block because the associated
{Volume,Vnode} were locked by another thread that the blocking call
would be queued for the worker thread that is currently holding the
locks to process.  Thereby, releasing the blocking worker thread back
into the pool.

Implementing such a change is not impossible but would need to be done
quite carefully.

In any case, I asked the question I did because at the present time the
file server cannot complete a call and free the associated worker thread
until the callback processing is complete.  This is necessary to ensure
that the data modification performed by the call is visible to all of
the interested clients prior to its completion.  By doing so any
out-of-band messages sent by one application to another working on the
same data set will be guaranteed to be serialized.  If there is no
requirement for serialized messaging between the clients it *might* be
possible for us to add support for O_NONBLOCK that would instruct the
file server to not wait for callback processing to complete but instead
queue the callback processing for a background task and complete the
call immediately.  Jobs that made use of such an option when it is safe
to do so would not run into the problems that you are experiencing.

I should note that at this point this is just an idea and has not be
vetted in anyway what so ever.

Jeffrey Altman



--------------enig41E65BFE7F6B4CEABAFF00BC
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)

iQEcBAEBAgAGBQJOqHr1AAoJENxm1CNJffh4ArUH/2PhJTqW+v+1wJsp7bR426ig
//m4EOO9dOfA3V4IMkPgezhbQdzRK2546lg07i3W++9HPj0DaE8/kFCivLhRXV9U
OulIOb+wsrPkugzMmEZliDfc+TtS+/2q/GISomLiclI0C2NG+PdFP6xjAtEY7HiG
xunzsCpWpwym91KWPOIqOC24KVKTnovyxVLTYT/Zx0+IOd+fnllDycy1W1zJ1dSS
l748uToKLsj2emMHvlnD2Tv2wbbsH7mt/PqBKH4GnRKnyC2/GabtYbDJLKKF0wSC
O4WRIfuXXMsRAI7dq/HoQeEcZB2TCUBuBg0pBOACt/Pl2VoELcZZAn9rsf/V5kY=
=ElWY
-----END PGP SIGNATURE-----

--------------enig41E65BFE7F6B4CEABAFF00BC--