[OpenAFS] Re: Kernel NULL pointer dereference
Ken Elkabany
Ken@elkabany.com
Fri, 20 Apr 2012 16:30:58 -0700
--0016e6de17488da8b704be24afc4
Content-Type: text/plain; charset=ISO-8859-1
On Fri, Apr 20, 2012 at 10:30 AM, Andrew Deason <adeason@sinenomine.net>wrote:
> On Thu, 19 Apr 2012 18:55:08 -0700
> Ken Elkabany <Ken@Elkabany.com> wrote:
>
> > We have 2 OpenAFS servers running 1.4.14. We have many clients that we
> > just switched over to 1.6.1pre1. Starting earlier today, we started
> > getting NULL pointer dereferences, which has been completely hosing
> > the clients. The client machines hang on any call that deals with AFS,
> > whether it's "ls /", "ls /afs", "klist", etc...
>
> 'klist' shouldn't touch AFS...
>
I agree. Perhaps I remember incorrectly, though the whole filesystem seemed
to have been in a very bad state (hence hanging on "ls /").
>
> > A "vos changeaddr" was done earlier today, whereby a large collection
> > (4000) of volumes were mistakenly assigned to another server. These
> > were corrected with "vos syncvldb" followed by "vos syncserv". I
> > mention it here, as it's the only thing we've done to the AFS cluster
> > today.
>
> I hope 'vos changeaddr' is not something you run very often. You should
> almost never run that command.
>
>
Not at all. We have a fileserver that has two IPs that map to one of its
interfaces; one accessible via LAN, one accessible via WAN. We wanted all
accesses to be done through the local IP so we used changeaddr.
Unfortunately, we pointed all the volumes to a different, incorrect server.
I'm also not sure if just running syncvldb/syncserv will entirely fix
> that; changeaddr has screwed up server entries pretty bad in the past. I
> wouldn't say you're out of the woods until you're still fine after a
> fileserver restart.
>
> It looks like it didn't. Due to some combination of changeaddr, syncvldb,
and syncserv, it looks like some volumes had 2 identical RO entries. After
removing all the RO entries for volumes that exhibited this issue, the
segfaulting went away. This was a lengthy operation as we have 4000+
volumes each with read only copies.
> > Here's what we found in the syslog:
> >
> > Apr 20 01:30:43 SERVER kernel: [12861236.027818] BUG: unable to handle
> > kernel NULL pointer dereference at 0000000000000028
> > Apr 20 01:30:43 SERVER kernel: [12861236.027836] IP: [<ffffffffa0048087>]
> > afs_Conn+0x1e7/0x260 [openafs]
>
> 1.6.1pre1 is not a great version to be running, but I can't think of
> something that's been fixed since then that would address this. If
> someone else has some idea, feel free to say otherwise.
>
> We did an early upgrade to the daily build of Ubuntu Precise, which had
1.6.1pre1. We just saw the update to 1.6.1-1 from two days ago (thanks
Sergio), so we'll be upgrading asap.
> That offset suggests dereferencing sa_flags for a NULL srvAddr. The only
> place I think that can happen is:
>
> /* First is always lowest rank, if it's up */
> if ((tv->status[0] == not_busy) && tv->serverHost[0]
> && !(tv->serverHost[0]->addr->sa_flags & SRVR_ISDOWN) &&
> !(((areq->idleError > 0) || (areq->tokenError > 0))
> && (areq->skipserver[0] == 1)))
>
> so it suggests that we have a vol struct with a server that has 0
> srvAddrs attached. I think maybe this is possible if we had another
> server struct 'steal' the srvAddr away, so we removed the srvAddr and
> we're left with none.
>
> One guess at what happened:
>
> - something tries to look up a volume A, and knows that it's on a
> server with IP address X
> - 'vos changeaddr' created a new non-mh server entry for IP X
> - we look up some other volume B, see it's on IP X, but we make a new
> server struct for the non-mh server, and steal X away from the other
> mh server struct
> - we need to look up volume A again, and serverHost[0] is pointing to a
> server that used to have the srvAddr for IP X, but since it was
> reassigned, the first srvAddr is now NULL.
>
> I'm not exactly sure what we'd want to happen in that situation. The
> quick fix is to just check for the NULL srvAddr, but I'm not immediately
> sure if that would cause us to keep trying to use a volume struct with
> no reachable servers until the vol entry expires.
>
> I (or someone) would need to experiment a bit to verify that. However,
> if something like the above is the case, this should never happen unless
> you make server identity/numbering mistakes like that. If someone else
> wants to look, I would guess this is reproducible by accessing a vol,
> changing the server uuid, accessing a different vol on the same server
> and then accessing the first vol again. Or something like that.
>
>
Thank you for taking the time to think about this. Once again, after
deleting the duplicate RO entries the problem disappeared. If you need any
help in reproducing this issue, do let me know. However, since it happened
on our live production systems, we aren't keen on making it happen again.
> --
> Andrew Deason
> adeason@sinenomine.net
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>
--0016e6de17488da8b704be24afc4
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div class=3D"gmail_quote">On Fri, Apr 20, 2012 at 10:30 AM, Andrew Deason =
<span dir=3D"ltr"><<a href=3D"mailto:adeason@sinenomine.net" target=3D"_=
blank">adeason@sinenomine.net</a>></span> wrote:<br><blockquote class=3D=
"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding=
-left:1ex">
<div>On Thu, 19 Apr 2012 18:55:08 -0700<br>
Ken Elkabany <Ken@Elkabany.com> wrote:<br>
<br>
> We have 2 OpenAFS servers running 1.4.14. We have many clients that we=
<br>
> just switched over to 1.6.1pre1. Starting earlier today, we started<br=
>
> getting NULL pointer dereferences, which has been completely hosing<br=
>
> the clients. The client machines hang on any call that deals with AFS,=
<br>
> whether it's "ls /", "ls /afs", "klist&qu=
ot;, etc...<br>
<br>
</div>'klist' shouldn't touch AFS...<br></blockquote><div><br><=
/div><div>I agree. Perhaps I remember incorrectly, though the whole filesys=
tem seemed to have been in a very bad state (hence hanging on "ls /&qu=
ot;).</div>
<div>=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;=
border-left:1px #ccc solid;padding-left:1ex">
<div><br>
> A "vos changeaddr" was done earlier today, whereby a large c=
ollection<br>
> (4000) of volumes were mistakenly assigned to another server. These<br=
>
> were corrected with "vos syncvldb" followed by "vos syn=
cserv". I<br>
> mention it here, as it's the only thing we've done to the AFS =
cluster<br>
> today.<br>
<br>
</div>I hope 'vos changeaddr' is not something you run very often. =
You should<br>
almost never run that command.<br>
<br></blockquote><div>=A0</div><div>Not at all. We have a fileserver that h=
as two IPs that map to=A0one of its interfaces; one accessible via LAN, one=
accessible via WAN. We wanted all accesses to be done through the local IP=
so we used changeaddr. Unfortunately, we pointed all the volumes to a diff=
erent, incorrect server.</div>
<div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex=
;border-left:1px #ccc solid;padding-left:1ex">
I'm also not sure if just running syncvldb/syncserv will entirely fix<b=
r>
that; changeaddr has screwed up server entries pretty bad in the past. I<br=
>
wouldn't say you're out of the woods until you're still fine af=
ter a<br>
fileserver restart.<br>
<div><br></div></blockquote><div>It looks like it didn't.=A0Due to some=
combination of changeaddr, syncvldb, and syncserv, it looks like some volu=
mes had 2 identical RO entries. After removing all the RO entries for volum=
es that exhibited this issue, the segfaulting went away. This was a lengthy=
operation as we have 4000+ volumes each with read only copies.</div>
<div>=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;=
border-left:1px #ccc solid;padding-left:1ex"><div>
> Here's what we found in the syslog:<br>
><br>
> Apr 20 01:30:43 SERVER kernel: [12861236.027818] BUG: unable to handle=
<br>
> kernel NULL pointer dereference at 0000000000000028<br>
> Apr 20 01:30:43 SERVER kernel: [12861236.027836] IP: [<ffffffffa004=
8087>]<br>
> afs_Conn+0x1e7/0x260 [openafs]<br>
<br>
</div>1.6.1pre1 is not a great version to be running, but I can't think=
of<br>
something that's been fixed since then that would address this. If<br>
someone else has some idea, feel free to say otherwise.<br>
<br></blockquote><div>We did an early upgrade to the daily build of Ubuntu =
Precise, which had 1.6.1pre1. We just saw the update to 1.6.1-1 from two da=
ys ago (thanks Sergio), so we'll be upgrading asap.</div><div>=A0</div>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
That offset suggests dereferencing sa_flags for a NULL srvAddr. The only<br=
>
place I think that can happen is:<br>
<br>
=A0 =A0/* First is always lowest rank, if it's up */<br>
=A0 =A0if ((tv->status[0] =3D=3D not_busy) && tv->serverHost=
[0]<br>
=A0 =A0 =A0 =A0&& !(tv->serverHost[0]->addr->sa_flags &am=
p; SRVR_ISDOWN) &&<br>
=A0 =A0 =A0 =A0!(((areq->idleError > 0) || (areq->tokenError >=
0))<br>
=A0 =A0 =A0 =A0 =A0&& (areq->skipserver[0] =3D=3D 1)))<br>
<br>
so it suggests that we have a vol struct with a server that has 0<br>
srvAddrs attached. I think maybe this is possible if we had another<br>
server struct 'steal' the srvAddr away, so we removed the srvAddr a=
nd<br>
we're left with none.<br>
<br>
One guess at what happened:<br>
<br>
=A0- something tries to look up a volume A, and knows that it's on a<br=
>
=A0 server with IP address X<br>
=A0- 'vos changeaddr' created a new non-mh server entry for IP X<br=
>
=A0- we look up some other volume B, see it's on IP X, but we make a ne=
w<br>
=A0 server struct for the non-mh server, and steal X away from the other<b=
r>
=A0 mh server struct<br>
=A0- we need to look up volume A again, and serverHost[0] is pointing to a<=
br>
=A0 server that used to have the srvAddr for IP X, but since it was<br>
=A0 reassigned, the first srvAddr is now NULL.<br>
<br>
I'm not exactly sure what we'd want to happen in that situation. Th=
e<br>
quick fix is to just check for the NULL srvAddr, but I'm not immediatel=
y<br>
sure if that would cause us to keep trying to use a volume struct with<br>
no reachable servers until the vol entry expires.<br>
<br>
I (or someone) would need to experiment a bit to verify that. However,<br>
if something like the above is the case, this should never happen unless<br=
>
you make server identity/numbering mistakes like that. If someone else<br>
wants to look, I would guess this is reproducible by accessing a vol,<br>
changing the server uuid, accessing a different vol on the same server<br>
and then accessing the first vol again. Or something like that.<br>
<span><font color=3D"#888888"><br></font></span></blockquote><div><br></div=
><div>Thank you for taking the time to think about this. Once again, after =
deleting the duplicate RO entries the problem disappeared. If you need any =
help in reproducing this issue, do let me know. However, since it happened =
on our live production systems, we aren't keen on making it happen agai=
n.</div>
<div>=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;=
border-left:1px #ccc solid;padding-left:1ex">
<span><font color=3D"#888888">
--<br>
Andrew Deason<br>
<a href=3D"mailto:adeason@sinenomine.net" target=3D"_blank">adeason@sinenom=
ine.net</a><br>
</font></span><div><div><br>
_______________________________________________<br>
OpenAFS-info mailing list<br>
<a href=3D"mailto:OpenAFS-info@openafs.org" target=3D"_blank">OpenAFS-info@=
openafs.org</a><br>
<a href=3D"https://lists.openafs.org/mailman/listinfo/openafs-info" target=
=3D"_blank">https://lists.openafs.org/mailman/listinfo/openafs-info</a><br>
</div></div></blockquote></div><br>
--0016e6de17488da8b704be24afc4--