[OpenAFS-devel] Re: pthreading the bosserver
Derrick Brashear
shadow@gmail.com
Thu, 8 Aug 2013 16:50:01 -0400
--e89a8ff2485542c67104e375cdf7
Content-Type: text/plain; charset=ISO-8859-1
On Thu, Aug 8, 2013 at 4:30 PM, Andrew Deason <adeason@sinenomine.net>wrote:
> On Thu, 8 Aug 2013 15:40:08 -0400 (EDT)
> Benjamin Kaduk <kaduk@MIT.EDU> wrote:
>
> > However, the bosserver is currently using LWP for parallelism, and
> > GSSAPI libraries which are compatible with LWP are hard to come by;
> > the obvious solution is to convert the bosserver to pthreads.
>
> Just mentioning... you can have a pthread process with lwp emulation
> that implements all of the lwp primitives in terms of pthreads. This is
> like having just a giant anti-preemption lock around everything, and I
> thought this already existed somewhere. I don't think that helps with
> signalling, but it can help with locking issues.
>
> Code to do this is in Arla; I donated it a jillion years ago, and it's
probably not drop-in for OpenAFS at this point.
> Not that that's a good general pthread-ifying solution, but since
> bosserver doesn't need to run very fast, consistency seems more
> important than actual parallelism.
>
> > First off: do we need to keep an LWP version of the bosserver around
> > as well as a pthreaded one? I don't think so, and I believe Simon
> > agrees, but it would be good to get consensus.
>
> It does not need to stay very long; lwp fileserver has already been
> removed. If you're asking if you can get rid of the LWP bosserver at the
> same time as introducing a pthreaded bosserver, I think that depends on
> how sure you are that it functions correctly. I would vote for a tbozo
> directory, but if the changes are not complex and you're verify
> confident, it may not be necessary. But I think it's easier to implement
> a tbozo, and then remove bozo (and move tbozo into it) when it's just as
> good.
>
> > Second, how strong of an integrity guarantee do we need for the bos
> > config? My understanding is that configuration changes (adding or
> > removing or en/disabling bnodes) are rare events, and it is highly
> > unlikely that multiple administrator connnections changing things will
> > be made concurrently.
>
> We can assume they are infrequent, but we must assume that they will
> happen. That is, there needs to be locking, but it doesn't need to be
> very granular. That is, it can be slow, but it cannot cause something to
> break or behave weirdly.
>
> > If this is true, then we can rely on time-domain "locking" for
> > synchronization and eliminate some aspects of code-level locking. For
> > example, a per-bnode lock acquired before writing any bnode state
> > would not be needed, and a single global lock would be sufficient.
>
> I don't really see how one of these is offering integrity but the other
> isn't, but... A single lock is fine, if I understand this correctly.
> You've never been able to do certain bozo things in parallel, but I
> haven't heard complaining about it. In any case, rxgk is more important
> than improving that.
>
> > Relatedly, is it okay to assume that shutdown/restart/etc. will not be
> > issued concurrently with config changes? A "fully correct"
> > implementation would seem to need to only shutdown/restart the bnodes
> > which were configured when the command was issued, and ignore any new
> > nodes created since then. Because the implementation of
> > shutdown/restart must drop locks, making this guarantee seems to
> > require additional sychronization effort, whether via a temporary
> > queue to store the bnodes being acted upon, or a higher-level lock.
>
> Are you talking about a 'bos create' racing with a 'bos restart -all'? I
> would think you'd block out all modifications during a restart. While
> the ordering may not matter for 'bos restart -all', it may matter for
> 'bos restart -bosserver', just so it doesn't leave behind a running
> process and then re-exec itself or something.
>
> > I haven't been able to convince myself that the additional complexity
> > of the extra watcher threads is necessary, but if someone else could
> > convince me, that would be good.
>
> My opinion is that we should explicitly drop LINUX24 support on servers
> (or at least tbozo, if we eventually provide both tbozo and bozo). I
> have never heard of demand for LINUX24 servers, and it's easy to migrate
> off of them. The thing I have heard demand for and is not easy to
> migrate off of is LINUX24 clients, which we could still keep.
>
> I mean, regardless of what solution we end up with, how much testing is
> anyone really going to do for bozo on LINUX24? We're just going to end
> up with something that theoretically works but we're not very confident
> has solved various possible race conditions or whatnot. If we want to
> keep LINUX24 for this, we should at least put a big warning on it that
> mentions something involving the relevant issues.
>
>
That would be my theory. At this point a lot of stuff is inadequately
tested on 2.4.
> That doesn't deal with any signalling specifics, but keep in mind our
> current bozo signal handling is not always great, and does not
> necessarily need to be fixed at the same time. I've always seen bozo
> misidentify core dumps, which I thought was due to this, but I've never
> really cared.
>
> --
> Andrew Deason
> adeason@sinenomine.net
>
> _______________________________________________
> OpenAFS-devel mailing list
> OpenAFS-devel@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-devel
>
>
--
Derrick
--e89a8ff2485542c67104e375cdf7
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quo=
te">On Thu, Aug 8, 2013 at 4:30 PM, Andrew Deason <span dir=3D"ltr"><<a =
href=3D"mailto:adeason@sinenomine.net" target=3D"_blank">adeason@sinenomine=
.net</a>></span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div class=3D"im">On Thu, 8 Aug 2013 15:40:0=
8 -0400 (EDT)<br>
Benjamin Kaduk <<a href=3D"mailto:kaduk@MIT.EDU">kaduk@MIT.EDU</a>> w=
rote:<br>
<br>
> However, the bosserver is currently using LWP for parallelism, and<br>
> GSSAPI libraries which are compatible with LWP are hard to come by;<br=
>
> the obvious solution is to convert the bosserver to pthreads.<br>
<br>
</div>Just mentioning... you can have a pthread process with lwp emulation<=
br>
that implements all of the lwp primitives in terms of pthreads. This is<br>
like having just a giant anti-preemption lock around everything, and I<br>
thought this already existed somewhere. I don't think that helps with<b=
r>
signalling, but it can help with locking issues.<br>
<br></blockquote><div>Code to do this is in Arla; I donated it a jillion ye=
ars ago, and it's probably not drop-in for OpenAFS at this point.<br>=
=A0<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;b=
order-left:1px #ccc solid;padding-left:1ex">
Not that that's a good general pthread-ifying solution, but since<br>
bosserver doesn't need to run very fast, consistency seems more<br>
important than actual parallelism.<br>
<div class=3D"im"><br>
> First off: do we need to keep an LWP version of the bosserver around<b=
r>
> as well as a pthreaded one? =A0I don't think so, and I believe Sim=
on<br>
> agrees, but it would be good to get consensus.<br>
<br>
</div>It does not need to stay very long; lwp fileserver has already been<b=
r>
removed. If you're asking if you can get rid of the LWP bosserver at th=
e<br>
same time as introducing a pthreaded bosserver, I think that depends on<br>
how sure you are that it functions correctly. I would vote for a tbozo<br>
directory, but if the changes are not complex and you're verify<br>
confident, it may not be necessary. But I think it's easier to implemen=
t<br>
a tbozo, and then remove bozo (and move tbozo into it) when it's just a=
s<br>
good.<br>
<div class=3D"im"><br>
> Second, how strong of an integrity guarantee do we need for the bos<br=
>
> config? =A0My understanding is that configuration changes (adding or<b=
r>
> removing or en/disabling bnodes) are rare events, and it is highly<br>
> unlikely that multiple administrator connnections changing things will=
<br>
> be made concurrently.<br>
<br>
</div>We can assume they are infrequent, but we must assume that they will<=
br>
happen. That is, there needs to be locking, but it doesn't need to be<b=
r>
very granular. That is, it can be slow, but it cannot cause something to<br=
>
break or behave weirdly.<br>
<div class=3D"im"><br>
> If this is true, then we can rely on time-domain "locking" f=
or<br>
> synchronization and eliminate some aspects of code-level locking. =A0F=
or<br>
> example, a per-bnode lock acquired before writing any bnode state<br>
> would not be needed, and a single global lock would be sufficient.<br>
<br>
</div>I don't really see how one of these is offering integrity but the=
other<br>
isn't, but... A single lock is fine, if I understand this correctly.<br=
>
You've never been able to do certain bozo things in parallel, but I<br>
haven't heard complaining about it. In any case, rxgk is more important=
<br>
than improving that.<br>
<div class=3D"im"><br>
> Relatedly, is it okay to assume that shutdown/restart/etc. will not be=
<br>
> issued concurrently with config changes? =A0A "fully correct"=
;<br>
> implementation would seem to need to only shutdown/restart the bnodes<=
br>
> which were configured when the command was issued, and ignore any new<=
br>
> nodes created since then. =A0Because the implementation of<br>
> shutdown/restart must drop locks, making this guarantee seems to<br>
> require additional sychronization effort, whether via a temporary<br>
> queue to store the bnodes being acted upon, or a higher-level lock.<br=
>
<br>
</div>Are you talking about a 'bos create' racing with a 'bos r=
estart -all'? I<br>
would think you'd block out all modifications during a restart. While<b=
r>
the ordering may not matter for 'bos restart -all', it may matter f=
or<br>
'bos restart -bosserver', just so it doesn't leave behind a run=
ning<br>
process and then re-exec itself or something.<br>
<div class=3D"im"><br>
> I haven't been able to convince myself that the additional complex=
ity<br>
> of the extra watcher threads is necessary, but if someone else could<b=
r>
> convince me, that would be good.<br>
<br>
</div>My opinion is that we should explicitly drop LINUX24 support on serve=
rs<br>
(or at least tbozo, if we eventually provide both tbozo and bozo). =A0I<br>
have never heard of demand for LINUX24 servers, and it's easy to migrat=
e<br>
off of them. The thing I have heard demand for and is not easy to<br>
migrate off of is LINUX24 clients, which we could still keep.<br>
<br>
I mean, regardless of what solution we end up with, how much testing is<br>
anyone really going to do for bozo on LINUX24? We're just going to end<=
br>
up with something that theoretically works but we're not very confident=
<br>
has solved various possible race conditions or whatnot. If we want to<br>
keep LINUX24 for this, we should at least put a big warning on it that<br>
mentions something involving the relevant issues.<br>
<br></blockquote><div><br></div><div>That would be my theory. At this point=
a lot of stuff is inadequately tested on 2.4.<br>=A0<br></div><blockquote =
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid=
;padding-left:1ex">
That doesn't deal with any signalling specifics, but keep in mind our<b=
r>
current bozo signal handling is not always great, and does not<br>
necessarily need to be fixed at the same time. I've always seen bozo<br=
>
misidentify core dumps, which I thought was due to this, but I've never=
<br>
really cared.<br>
<span class=3D"HOEnZb"><font color=3D"#888888"><br>
--<br>
Andrew Deason<br>
<a href=3D"mailto:adeason@sinenomine.net">adeason@sinenomine.net</a><br>
</font></span><div class=3D"HOEnZb"><div class=3D"h5"><br>
_______________________________________________<br>
OpenAFS-devel mailing list<br>
<a href=3D"mailto:OpenAFS-devel@openafs.org">OpenAFS-devel@openafs.org</a><=
br>
<a href=3D"https://lists.openafs.org/mailman/listinfo/openafs-devel" target=
=3D"_blank">https://lists.openafs.org/mailman/listinfo/openafs-devel</a><br=
>
<br>
</div></div></blockquote></div><br><br clear=3D"all"><br>-- <br>Derrick
</div></div>
--e89a8ff2485542c67104e375cdf7--