[OpenAFS-devel] pthreading the bosserver
Derrick Brashear
shadow@gmail.com
Thu, 8 Aug 2013 16:27:44 -0400
--001a11c309aa96417604e3757d27
Content-Type: text/plain; charset=ISO-8859-1
On Thu, Aug 8, 2013 at 3:40 PM, Benjamin Kaduk <kaduk@mit.edu> wrote:
> Hi all,
>
> It looks like the last hurdle before a core rxgk spec gets through
> afs3-stds is the description of the GSS negotiation loop. The right answer
> there is well-known, though describing it is easy, so it should just be a
> matter of time before it's done.
>
> With just the core rxgk spec done, we can still start adding rxgk to the
> non-AFS3 protocol pieces in our tree, and I think the bosserver is the most
> promising target. This allows us to get some code in the tree for the
> security class and get some working experience, and reduces the burden for
> developing additional code out-of-tree.
>
> However, the bosserver is currently using LWP for parallelism, and GSSAPI
> libraries which are compatible with LWP are hard to come by; the obvious
> solution is to convert the bosserver to pthreads. Chas Williams came up
> with an implementation about 8 months ago, which is currently languishing
> in gerrit (it just got a newly rebased version, the tip commit is gerrit
> number 8794). I had a mostly complete independent implementation before I
> learned about his work, unfortunately; mine is in gerrit with the tip
> change 10130. Mine is still incomplete in that it needs some build
> massaging on a few Unix platforms, and needs a bit of Windows attention. As
> tempting as it may be, I seem to recall that we do not have critical mass
> to drop server support for windows on master, so that would need to be
> fixed for my code to move forward.
>
> There are a few general differences between the approaches in the two
> patchsets, and I was hoping we could have an architectural discussion on
> this list.
>
> First off: do we need to keep an LWP version of the bosserver around as
> well as a pthreaded one? I don't think so, and I believe Simon agrees, but
> it would be good to get consensus.
>
>
It's just going to bitrot, so i think it's unwise to leave it.
> Second, how strong of an integrity guarantee do we need for the bos
> config? My understanding is that configuration changes (adding or removing
> or en/disabling bnodes) are rare events, and it is highly unlikely that
> multiple administrator connnections changing things will be made
> concurrently. If this is true, then we can rely on time-domain "locking"
> for synchronization and eliminate some aspects of code-level locking. For
> example, a per-bnode lock acquired before writing any bnode state would not
> be needed, and a single global lock would be sufficient.
>
> A global lock is probably sufficient, but this isn't performance-critical
code.
> Relatedly, is it okay to assume that shutdown/restart/etc. will not be
> issued concurrently with config changes? A "fully correct" implementation
> would seem to need to only shutdown/restart the bnodes which were
> configured when the command was issued, and ignore any new nodes created
> since then. Because the implementation of shutdown/restart must drop
> locks, making this guarantee seems to require additional sychronization
> effort, whether via a temporary queue to store the bnodes being acted upon,
> or a higher-level lock.
>
> Then there's the question of signal handling. The discussion on
> gerrit/6947 raises some potentially large spectres, in particular
> LinuxThreads compatibility. Chas has dedicated pthreads for each child
> process to listen for SIGCHLD, plus a global thread for SIGTERM/SIGQUIT
> (hmm, SIGKILL is added to the set, too, but can't actually be blocked or
> masked); my version just has a single signal handler routine that uses the
> sigpipe trick to wake up the bproc thread and check the children. This
> sigpipe trick doesn't work directly on Windows; I'll need to look more
> carefully at how to workaround. I seem to recall that we hand-roll signal
> compatibility bits for Windows anyway, so it would stay in-tree. I haven't
> been able to convince myself that the additional complexity of the extra
> watcher threads is necessary, but if someone else could convince me, that
> would be good.
>
> Off the top of my head, those are the main structural differences between
> the two existing implementations, I'd be interested to hear everyone's
> opinions on the questions. I'm not tied to my code, but I did continue
> with it after learning about Chas's work because I did have some questions
> about these architectural questionss. If the consensus is that his stuff
> is fine, we should go with that -- my main goal is just to get a pthreaded
> bosserver in the tree so that we can build off of it with rxgk.
>
>
> Thanks,
>
> Ben
> ______________________________**_________________
> OpenAFS-devel mailing list
> OpenAFS-devel@openafs.org
> https://lists.openafs.org/**mailman/listinfo/openafs-devel<https://lists.openafs.org/mailman/listinfo/openafs-devel>
>
>
--
Derrick
--001a11c309aa96417604e3757d27
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><br><div class=3D"gmail_extra"><br><br><div class=3D"gmail=
_quote">On Thu, Aug 8, 2013 at 3:40 PM, Benjamin Kaduk <span dir=3D"ltr">&l=
t;<a href=3D"mailto:kaduk@mit.edu" target=3D"_blank">kaduk@mit.edu</a>><=
/span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hi all,<br>
<br>
It looks like the last hurdle before a core rxgk spec gets through afs3-std=
s is the description of the GSS negotiation loop. =A0The right answer there=
is well-known, though describing it is easy, so it should just be a matter=
of time before it's done.<br>
<br>
With just the core rxgk spec done, we can still start adding rxgk to the no=
n-AFS3 protocol pieces in our tree, and I think the bosserver is the most p=
romising target. =A0This allows us to get some code in the tree for the sec=
urity class and get some working experience, and reduces the burden for dev=
eloping additional code out-of-tree.<br>
<br>
However, the bosserver is currently using LWP for parallelism, and GSSAPI l=
ibraries which are compatible with LWP are hard to come by; the obvious sol=
ution is to convert the bosserver to pthreads. =A0Chas Williams came up wit=
h an implementation about 8 months ago, which is currently languishing in g=
errit (it just got a newly rebased version, the tip commit is gerrit number=
8794). =A0I had a mostly complete independent implementation before I lear=
ned about his work, unfortunately; mine is in gerrit with the tip change 10=
130. =A0Mine is still incomplete in that it needs some build massaging on a=
few Unix platforms, and needs a bit of Windows attention. As tempting as i=
t may be, I seem to recall that we do not have critical mass to drop server=
support for windows on master, so that would need to be fixed for my code =
to move forward.<br>
<br>
There are a few general differences between the approaches in the two patch=
sets, and I was hoping we could have an architectural discussion on this li=
st.<br>
<br>
First off: do we need to keep an LWP version of the bosserver around as wel=
l as a pthreaded one? =A0I don't think so, and I believe Simon agrees, =
but it would be good to get consensus.<br>
<br></blockquote><div><br></div><div>It's just going to bitrot, so i th=
ink it's unwise to leave it.<br>=A0<br></div><blockquote class=3D"gmail=
_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:=
1ex">
Second, how strong of an integrity guarantee do we need for the bos config?=
=A0My understanding is that configuration changes (adding or removing or e=
n/disabling bnodes) are rare events, and it is highly unlikely that multipl=
e administrator connnections changing things will be made concurrently. =A0=
If this is true, then we can rely on time-domain "locking" for sy=
nchronization and eliminate some aspects of code-level locking. =A0For exam=
ple, a per-bnode lock acquired before writing any bnode state would not be =
needed, and a single global lock would be sufficient.<br>
<br></blockquote><div>A global lock is probably sufficient, but this isn=
9;t performance-critical code.<br>=A0<br></div><blockquote class=3D"gmail_q=
uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e=
x">
Relatedly, is it okay to assume that shutdown/restart/etc. will not be issu=
ed concurrently with config changes? =A0A "fully correct" impleme=
ntation would seem to need to only shutdown/restart the bnodes which were c=
onfigured when the command was issued, and ignore any new nodes created sin=
ce then. =A0Because the implementation of shutdown/restart must drop locks,=
making this guarantee seems to require additional sychronization effort, w=
hether via a temporary queue to store the bnodes being acted upon, or a hig=
her-level lock.<br>
<br>
Then there's the question of signal handling. =A0The discussion on gerr=
it/6947 raises some potentially large spectres, in particular LinuxThreads =
compatibility. =A0Chas has dedicated pthreads for each child process to lis=
ten for SIGCHLD, plus a global thread for SIGTERM/SIGQUIT (hmm, SIGKILL is =
added to the set, too, but can't actually be blocked or masked); my ver=
sion just has a single signal handler routine that uses the sigpipe trick t=
o wake up the bproc thread and check the children. =A0This sigpipe trick do=
esn't work directly on Windows; I'll need to look more carefully at=
how to workaround. =A0I seem to recall that we hand-roll signal compatibil=
ity bits for Windows anyway, so it would stay in-tree. =A0I haven't bee=
n able to convince myself that the additional complexity of the extra watch=
er threads is necessary, but if someone else could convince me, that would =
be good.<br>
<br>
Off the top of my head, those are the main structural differences between t=
he two existing implementations, I'd be interested to hear everyone'=
;s opinions on the questions. =A0I'm not tied to my code, but I did con=
tinue with it after learning about Chas's work because I did have some =
questions about these architectural questionss. =A0If the consensus is that=
his stuff is fine, we should go with that -- my main goal is just to get a=
pthreaded bosserver in the tree so that we can build off of it with rxgk.<=
br>
<br>
<br>
Thanks,<br>
<br>
Ben<br>
______________________________<u></u>_________________<br>
OpenAFS-devel mailing list<br>
<a href=3D"mailto:OpenAFS-devel@openafs.org" target=3D"_blank">OpenAFS-deve=
l@openafs.org</a><br>
<a href=3D"https://lists.openafs.org/mailman/listinfo/openafs-devel" target=
=3D"_blank">https://lists.openafs.org/<u></u>mailman/listinfo/openafs-devel=
</a><br>
<br>
</blockquote></div><br><br clear=3D"all"><br>-- <br>Derrick
</div></div>
--001a11c309aa96417604e3757d27--