[OpenAFS] RHEL 7.5 beta / 3.10.0-830.el7.x86_66 kernel lock up
Matt Vander Werf
mvanderw@nd.edu
Tue, 10 Apr 2018 14:48:56 -0400
--883d24f22a7cf2a521056982fbae
Content-Type: text/plain; charset="UTF-8"
RHEL 7.5 GA was released earlier this morning.
Just to confirm, I tested Gary's 1.6.x patch again with the new kernel for
RHEL 7.5 GA (3.10.0-862.el7.x86_64) and it still fixes the ENOTDIR issue
and I also don't see any other issues with OpenAFS and this new RHEL
release (at least not at this time).
Looking forward to a 1.6.22.3 release soon!
Thanks.
--
Matt Vander Werf
HPC System Administrator
University of Notre Dame
Center for Research Computing - Union Station
506 W. South Street
South Bend, IN 46601
Phone: (574) 631-0692
On Fri, Mar 23, 2018 at 9:50 AM, Stephan Wiesand <stephan.wiesand@desy.de>
wrote:
>
> > On 23. Mar 2018, at 12:27, Kodiak Firesmith <kfiresmith@gmail.com>
> wrote:
> >
> > I've also tested gsgatlin's 7.5beta RPMs and they work great. Any
> chance we'll see the rh75enotdir patch integrated into a release of
> 1.6.22.3 soon? I'm wondering if it'll be worth it to manually apply that
> patch to a rebuild of the official OpenAFS RPMs if this isn't on the block
> for being merged and released soon - but I don't want to blow the time
> applying that patch to a re-roll if a fixed official release is forthcoming.
>
> We are planning to release a 1.6.22.3 addressing the ENOTDIR issue with
> the EL7.5 kernel soon after the EL7.5 GA release.
>
> - Stephan
>
> > Thanks!
> > - Kodiak
> >
> >
> > On Fri, Mar 2, 2018 at 3:47 AM, Anders Nordin <anders.j.nordin@ltu.se>
> wrote:
> > Hello,
> >
> > Is there any progress on this issue? Can we expect a stable release for
> RHEL 7.5?
> >
> > MVH
> > Anders
> >
> > -----Original Message-----
> > From: openafs-info-admin@openafs.org [mailto:openafs-info-admin@ope
> nafs.org] On Behalf Of Benjamin Kaduk
> > Sent: den 9 februari 2018 01:02
> > To: Kodiak Firesmith <kfiresmith@gmail.com>
> > Cc: openafs-info <openafs-info@openafs.org>
> > Subject: Re: [OpenAFS] RHEL 7.5 beta / 3.10.0-830.el7.x86_66 kernel lock
> up
> >
> > On Wed, Feb 07, 2018 at 11:46:28AM -0500, Kodiak Firesmith wrote:
> > > Hello again All,
> > >
> > > As part of continued testing, I've been able to confirm that the
> > > SystemD double-service startup thing only happens to my hosts when
> > > going from RHEL
> > > 7.4 to RHEL 7.5beta. On a test host installed directly as RHEL
> > > 7.5beta, I get a bit farther with 1.6.18.22, in that I get to the
> > > point where OpenAFS "kind of" works.
> >
> > Thanks for tracking this down. The rpm packaging maintainers may want
> to try to track down why the double-start happens in the upgrade scenario,
> as that's pretty nasty behavior.
> >
> > > What I'm observing is that the openafs client Kernel module (built by
> > > DKMS) loads fine, and just so long as you know where you need to go in
> > > /afs, you can get there, and you can read and write files and the
> OpenAFS 'fs'
> > > command works. But doing an 'ls' of /afs or any path underneath
> > > results in
> > > "ls: reading directory /afs/: Not a directory".
> > >
> > > I ran an strace of a good RHEL 7.4 host running ls on /afs, and a RHEL
> > > 7.5beta host running ls on /afs and have created pastebins of both, as
> > > well as an inline diff.
> > >
> > > All can be seen at the following locations:
> > >
> > > works
> > > https://paste.fedoraproject.org/paste/Hiojt2~Be3wgez47bKNucQ
> > >
> > > fails
> > > https://paste.fedoraproject.org/paste/13ZXBfJIOMsuEJFwFShBfg
> > >
> > >
> > > diff
> > > https://paste.fedoraproject.org/paste/FJKRwep1fWJogIDbLnkn8A
> > >
> > > Hopefully this might help the OpenAFS devs, or someone might know what
> > > might be borking on every RHEL 7.5 beta host. It does fit with what
> > > other
> > > 7.5 beta users have observed OpenAFS doing.
> >
> > Yes, now it seems like all our reports are consistent, and we just have
> to wait for a developer to get a better look at what Red Hat changed in the
> kernel that we need to adapt to.
> >
> > -Ben
> >
> > > Thanks!
> > > - Kodiak
> > >
> > > On Mon, Feb 5, 2018 at 12:31 PM, Stephan Wiesand
> > > <stephan.wiesand@desy.de>
> > > wrote:
> > >
> > > >
> > > > > On 04.Feb 2018, at 02:11, Jeffrey Altman <jaltman@auristor.com>
> wrote:
> > > > >
> > > > > On 2/2/2018 6:04 PM, Kodiak Firesmith wrote:
> > > > >> I'm relatively new to handling OpenAFS. Are these problems part
> > > > >> of a normal "kernel release; openafs update" cycle and perhaps
> > > > >> I'm getting snagged just by being too early of an adopter? I
> > > > >> wanted to raise the alarm on this and see if anything else was
> > > > >> needed from me as the reporter of the issue, but perhaps that's
> > > > >> an overreaction to what is just part of a normal process I just
> > > > >> haven't been tuned into in prior RHEL release cycles?
> > > > >
> > > > >
> > > > > Kodiak,
> > > > >
> > > > > On RHEL, DKMS is safe to use for kernel modules that restrict
> > > > > themselves to using the restricted set of kernel interfaces (the
> > > > > RHEL KABI) that Red Hat has designated will be supported across
> > > > > the lifespan of the RHEL major version number. OpenAFS is not
> > > > > such a kernel module. As a result it is vulnerable to breakage
> each and every time a new kernel is shipped.
> > > >
> > > > Jeffrey,
> > > >
> > > > the usual way to use DKMS is to either have it build a module for a
> > > > newly installed kernel or install a prebuilt module for that kernel.
> > > > It may be possible to abuse it for providing a module built for
> > > > another kernel, but I think that won't happen accidentally.
> > > >
> > > > You may be confusing DKMS with RHEL's "KABI tracking kmods". Those
> > > > should be safe to use within a RHEL minor release (and the SL
> > > > packaging has been using them like this since EL6.4), but aren't
> > > > across minor releases (and that's why the SL packaging modifies the
> > > > kmod handling to require a build for the minor release in question.
> > > >
> > > > > There are two types of failures that can occur:
> > > > >
> > > > > 1. a change results in failure to build the OpenAFS kernel module
> > > > > for the new kernel
> > > > >
> > > > > 2. a change results in the OpenAFS kernel module building and
> > > > > successfully loading but failing to operate correctly
> > > >
> > > > The latter shouldn't happen within a minor release, but can across
> > > > minor releases.
> > > >
> > > > > It is the second of these possibilities that has taken place with
> > > > > the release of the 3.10.0-830.el7 kernel shipped as part of the
> > > > > RHEL 7.5
> > > > beta.
> > > > >
> > > > > Are you an early adopter of RHEL 7.5 beta? Absolutely, its a beta
> > > > > release and as such you should expect that there will be bugs and
> > > > > that third party kernel modules that do not adhere to the KABI
> > > > > functionality might have compatibility issues.
> > > >
> > > > The -830 kernel can break 3rd-party modules using non-whitelisted
> > > > ABIs, whether or not they adhere to the "KABI functionality".
> > > >
> > > > > There was a compatibility issue with RHEL 7.4 kernel
> > > > > (3.10.0_693.1.1.el7) as well that was only fixed in the OpenAFS
> > > > > 1.6 release series this past week as part of 1.6.22.2:
> > > > >
> > > > > http://www.openafs.org/dl/openafs/1.6.22.2/RELNOTES-1.6.22.2
> > > >
> > > > Yes, and this one was hard to fix. Thanks are due to Mark Vitale for
> > > > developing the fix and all those who reviewed and tested it.
> > > >
> > > > > Jeffrey Altman
> > > > > AuriStor, Inc.
> > > > >
> > > > > P.S. - Welcome to the community.
> > > >
> > > > Seconded. In particular, the problem report regarding the EL7.5beta
> > > > kernel was absolutely appropriate.
>
> --
> Stephan Wiesand
> DESY -DV-
> Platanenallee 6
> 15738 Zeuthen, Germany
>
>
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>
--883d24f22a7cf2a521056982fbae
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div><div><div>RHEL 7.5 GA was released earlier this morni=
ng.<br><br></div>Just to confirm, I tested Gary's 1.6.x patch again wit=
h the new kernel for RHEL 7.5 GA (3.10.0-862.el7.x86_64) and it still fixes=
the ENOTDIR issue and I also don't see any other issues with OpenAFS a=
nd this new RHEL release (at least not at this time).<br><br></div>Looking =
forward to a 1.6.22.3 release soon!<br><br></div>Thanks.<br><div><div><div =
class=3D"gmail_extra"><br clear=3D"all"><div><div class=3D"m_-8650044924045=
855289gmail_signature"><div dir=3D"ltr"><div>--<br></div><div>Matt Vander W=
erf<br>HPC System Administrator<br>University of Notre Dame<br>Center for R=
esearch Computing - Union Station<br>506 W. South Street<br>South Bend, IN =
46601<br></div>Phone: (574) 631-0692</div></div></div>
<br><div class=3D"gmail_quote">On Fri, Mar 23, 2018 at 9:50 AM, Stephan Wie=
sand <span dir=3D"ltr"><<a href=3D"mailto:stephan.wiesand@desy.de" targe=
t=3D"_blank">stephan.wiesand@desy.de</a>></span> wrote:<br><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px soli=
d rgb(204,204,204);padding-left:1ex"><span class=3D"m_-8650044924045855289g=
mail-"><br>
> On 23. Mar 2018, at 12:27, Kodiak Firesmith <<a href=3D"mailto:kfir=
esmith@gmail.com" target=3D"_blank">kfiresmith@gmail.com</a>> wrote:<br>
><br>
> I've also tested gsgatlin's 7.5beta RPMs and they work great.=
=C2=A0 Any chance we'll see the rh75enotdir patch integrated into a rel=
ease of 1.6.22.3 soon?=C2=A0 I'm wondering if it'll be worth it to =
manually apply that patch to a rebuild of the official OpenAFS RPMs if this=
isn't on the block for being merged and released soon - but I don'=
t want to blow the time applying that patch to a re-roll if a fixed officia=
l release is forthcoming.<br>
<br>
</span>We are planning to release a 1.6.22.3 addressing the ENOTDIR issue w=
ith the EL7.5 kernel soon after the EL7.5 GA release.<br>
<span class=3D"m_-8650044924045855289gmail-HOEnZb"><font color=3D"#888888">=
<br>
- Stephan<br>
</font></span><div class=3D"m_-8650044924045855289gmail-HOEnZb"><div class=
=3D"m_-8650044924045855289gmail-h5"><br>
> Thanks!<br>
>=C2=A0 - Kodiak<br>
><br>
><br>
> On Fri, Mar 2, 2018 at 3:47 AM, Anders Nordin <<a href=3D"mailto:an=
ders.j.nordin@ltu.se" target=3D"_blank">anders.j.nordin@ltu.se</a>> wrot=
e:<br>
> Hello,<br>
><br>
> Is there any progress on this issue? Can we expect a stable release fo=
r RHEL 7.5?<br>
><br>
> MVH<br>
> Anders<br>
><br>
> -----Original Message-----<br>
> From: <a href=3D"mailto:openafs-info-admin@openafs.org" target=3D"_bla=
nk">openafs-info-admin@openafs.org</a> [mailto:<a href=3D"mailto:openafs-in=
fo-admin@openafs.org" target=3D"_blank">openafs-info-admin@ope<wbr>nafs.org=
</a>] On Behalf Of Benjamin Kaduk<br>
> Sent: den 9 februari 2018 01:02<br>
> To: Kodiak Firesmith <<a href=3D"mailto:kfiresmith@gmail.com" targe=
t=3D"_blank">kfiresmith@gmail.com</a>><br>
> Cc: openafs-info <<a href=3D"mailto:openafs-info@openafs.org" targe=
t=3D"_blank">openafs-info@openafs.org</a>><br>
> Subject: Re: [OpenAFS] RHEL 7.5 beta / 3.10.0-830.el7.x86_66 kernel lo=
ck up<br>
><br>
> On Wed, Feb 07, 2018 at 11:46:28AM -0500, Kodiak Firesmith wrote:<br>
> > Hello again All,<br>
> ><br>
> > As part of continued testing, I've been able to confirm that =
the<br>
> > SystemD double-service startup thing only happens to my hosts whe=
n<br>
> > going from RHEL<br>
> > 7.4 to RHEL 7.5beta.=C2=A0 On a test host installed directly as R=
HEL<br>
> > 7.5beta, I get a bit farther with 1.6.18.22, in that I get to the=
<br>
> > point where OpenAFS "kind of" works.<br>
><br>
> Thanks for tracking this down.=C2=A0 The rpm packaging maintainers may=
want to try to track down why the double-start happens in the upgrade scen=
ario, as that's pretty nasty behavior.<br>
><br>
> > What I'm observing is that the openafs client Kernel module (=
built by<br>
> > DKMS) loads fine, and just so long as you know where you need to =
go in<br>
> > /afs, you can get there, and you can read and write files and the=
OpenAFS 'fs'<br>
> > command works.=C2=A0 But doing an 'ls' of /afs or any pat=
h underneath<br>
> > results in<br>
> > "ls: reading directory /afs/: Not a directory".<br>
> ><br>
> > I ran an strace of a good RHEL 7.4 host running ls on /afs, and a=
RHEL<br>
> > 7.5beta host running ls on /afs and have created pastebins of bot=
h, as<br>
> > well as an inline diff.<br>
> ><br>
> > All can be seen at the following locations:<br>
> ><br>
> > works<br>
> > <a href=3D"https://paste.fedoraproject.org/paste/Hiojt2~Be3wgez47=
bKNucQ" rel=3D"noreferrer" target=3D"_blank">https://paste.fedoraproject.or=
<wbr>g/paste/Hiojt2~Be3wgez47bKNucQ</a><br>
> ><br>
> > fails<br>
> > <a href=3D"https://paste.fedoraproject.org/paste/13ZXBfJIOMsuEJFw=
FShBfg" rel=3D"noreferrer" target=3D"_blank">https://paste.fedoraproject.or=
<wbr>g/paste/13ZXBfJIOMsuEJFwFShBfg</a><br>
> ><br>
> ><br>
> > diff<br>
> > <a href=3D"https://paste.fedoraproject.org/paste/FJKRwep1fWJogIDb=
Lnkn8A" rel=3D"noreferrer" target=3D"_blank">https://paste.fedoraproject.or=
<wbr>g/paste/FJKRwep1fWJogIDbLnkn8A</a><br>
> ><br>
> > Hopefully this might help the OpenAFS devs, or someone might know=
what<br>
> > might be borking on every RHEL 7.5 beta host.=C2=A0 It does fit w=
ith what<br>
> > other<br>
> > 7.5 beta users have observed OpenAFS doing.<br>
><br>
> Yes, now it seems like all our reports are consistent, and we just hav=
e to wait for a developer to get a better look at what Red Hat changed in t=
he kernel that we need to adapt to.<br>
><br>
> -Ben<br>
><br>
> > Thanks!<br>
> >=C2=A0 - Kodiak<br>
> ><br>
> > On Mon, Feb 5, 2018 at 12:31 PM, Stephan Wiesand<br>
> > <<a href=3D"mailto:stephan.wiesand@desy.de" target=3D"_blank">=
stephan.wiesand@desy.de</a>><br>
> > wrote:<br>
> ><br>
> > ><br>
> > > > On 04.Feb 2018, at 02:11, Jeffrey Altman <<a href=3D=
"mailto:jaltman@auristor.com" target=3D"_blank">jaltman@auristor.com</a>>=
; wrote:<br>
> > > ><br>
> > > > On 2/2/2018 6:04 PM, Kodiak Firesmith wrote:<br>
> > > >> I'm relatively new to handling OpenAFS.=C2=A0 A=
re these problems part<br>
> > > >> of a normal "kernel release; openafs update&qu=
ot; cycle and perhaps<br>
> > > >> I'm getting snagged just by being too early of =
an adopter?=C2=A0 I<br>
> > > >> wanted to raise the alarm on this and see if anythi=
ng else was<br>
> > > >> needed from me as the reporter of the issue, but pe=
rhaps that's<br>
> > > >> an overreaction to what is just part of a normal pr=
ocess I just<br>
> > > >> haven't been tuned into in prior RHEL release c=
ycles?<br>
> > > ><br>
> > > ><br>
> > > > Kodiak,<br>
> > > ><br>
> > > > On RHEL, DKMS is safe to use for kernel modules that re=
strict<br>
> > > > themselves to using the restricted set of kernel interf=
aces (the<br>
> > > > RHEL KABI) that Red Hat has designated will be supporte=
d across<br>
> > > > the lifespan of the RHEL major version number.=C2=A0 Op=
enAFS is not<br>
> > > > such a kernel module.=C2=A0 As a result it is vulnerabl=
e to breakage each and every time a new kernel is shipped.<br>
> > ><br>
> > > Jeffrey,<br>
> > ><br>
> > > the usual way to use DKMS is to either have it build a modul=
e for a<br>
> > > newly installed kernel or install a prebuilt module for that=
kernel.<br>
> > > It may be possible to abuse it for providing a module built =
for<br>
> > > another kernel, but I think that won't happen accidental=
ly.<br>
> > ><br>
> > > You may be confusing DKMS with RHEL's "KABI trackin=
g kmods". Those<br>
> > > should be safe to use within a RHEL minor release (and the S=
L<br>
> > > packaging has been using them like this since EL6.4), but ar=
en't<br>
> > > across minor releases (and that's why the SL packaging m=
odifies the<br>
> > > kmod handling to require a build for the minor release in qu=
estion.<br>
> > ><br>
> > > > There are two types of failures that can occur:<br>
> > > ><br>
> > > > 1. a change results in failure to build the OpenAFS ker=
nel module<br>
> > > >=C2=A0 =C2=A0 for the new kernel<br>
> > > ><br>
> > > > 2. a change results in the OpenAFS kernel module buildi=
ng and<br>
> > > >=C2=A0 =C2=A0 successfully loading but failing to operat=
e correctly<br>
> > ><br>
> > > The latter shouldn't happen within a minor release, but =
can across<br>
> > > minor releases.<br>
> > ><br>
> > > > It is the second of these possibilities that has taken =
place with<br>
> > > > the release of the 3.10.0-830.el7 kernel shipped as par=
t of the<br>
> > > > RHEL 7.5<br>
> > > beta.<br>
> > > ><br>
> > > > Are you an early adopter of RHEL 7.5 beta?=C2=A0 Absolu=
tely, its a beta<br>
> > > > release and as such you should expect that there will b=
e bugs and<br>
> > > > that third party kernel modules that do not adhere to t=
he KABI<br>
> > > > functionality might have compatibility issues.<br>
> > ><br>
> > > The -830 kernel can break 3rd-party modules using non-whitel=
isted<br>
> > > ABIs, whether or not they adhere to the "KABI functiona=
lity".<br>
> > ><br>
> > > > There was a compatibility issue with RHEL 7.4 kernel<br=
>
> > > > (3.10.0_693.1.1.el7) as well that was only fixed in the=
OpenAFS<br>
> > > > 1.6 release series this past week as part of <a href=3D=
"http://1.6.22.2" rel=3D"noreferrer" target=3D"_blank">1.6.22.2</a>:<br>
> > > ><br>
> > > >=C2=A0 <a href=3D"http://www.openafs.org/dl/openafs/1.6.=
22.2/RELNOTES-1.6.22.2" rel=3D"noreferrer" target=3D"_blank">http://www.ope=
nafs.org/dl/open<wbr>afs/1.6.22.2/RELNOTES-1.6.22.2</a><br>
> > ><br>
> > > Yes, and this one was hard to fix. Thanks are due to Mark Vi=
tale for<br>
> > > developing the fix and all those who reviewed and tested it.=
<br>
> > ><br>
> > > > Jeffrey Altman<br>
> > > > AuriStor, Inc.<br>
> > > ><br>
> > > > P.S. - Welcome to the community.<br>
> > ><br>
> > > Seconded. In particular, the problem report regarding the EL=
7.5beta<br>
> > > kernel was absolutely appropriate.<br>
<br>
--<br>
Stephan Wiesand<br>
DESY -DV-<br>
Platanenallee 6<br>
15738 Zeuthen, Germany<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
OpenAFS-info mailing list<br>
<a href=3D"mailto:OpenAFS-info@openafs.org" target=3D"_blank">OpenAFS-info@=
openafs.org</a><br>
<a href=3D"https://lists.openafs.org/mailman/listinfo/openafs-info" rel=3D"=
noreferrer" target=3D"_blank">https://lists.openafs.org/mail<wbr>man/listin=
fo/openafs-info</a><br>
</div></div></blockquote></div><br></div></div></div></div>
--883d24f22a7cf2a521056982fbae--