[OpenAFS] Re: kernel panic in 1.4.14.1 on rhel6

Aaron Knister aaronk@umbc.edu
Wed, 27 Jul 2011 15:25:15 -0400


--bcaec501604d436e6704a912037f
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jul 27, 2011 at 3:05 PM, Andrew Deason <adeason@sinenomine.net>wrote:

> On Wed, 27 Jul 2011 13:50:44 -0400
> Michael Meffie <mmeffie@sinenomine.net> wrote:
>
> > This looks similar to the panic fixed by Andrew's recent fix,
> >
> >    http://gerrit.openafs.org/5086
> >
> > which fixes a reference count imbalance.
>
> The panic that that change fixes doesn't occur (or is very unlikely to
> occur) unless you have gerrit 4625; it's not that.
>
> Aaron, if you run 'log' in crash, what do the last few lines say?
> Filter out any sensitive information or whatnot; I wouldn't expect there
> to be any, but I don't know what's in your log.
>

Hi Andrew,

Here's the output of the last few, what I think are relavent lines:

Modules linked in: openafs(P)(U) ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4
iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6
xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ppdev parport_pc
parport vmware_balloon vmxnet3 i2c_piix4 i2c_core sg shpchp ext4 mbcache
jbd2 sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic
ata_piix dm_mod [last unloaded: scsi_wait_scan]

Modules linked in: openafs(P)(U) ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4
iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6
xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ppdev parport_pc
parport vmware_balloon vmxnet3 i2c_piix4 i2c_core sg shpchp ext4 mbcache
jbd2 sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic
ata_piix dm_mod [last unloaded: scsi_wait_scan]
Pid: 3573, comm: httpd Tainted: P           ----------------
2.6.32-131.6.1.el6.x86_64 #1 VMware Virtual Platform
RIP: 0010:[<ffffffff814dbfd1>]  [<ffffffff814dbfd1>] mutex_lock+0x21/0x50
RSP: 0018:ffff8800610bdac8  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 28ec2299000000e7 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 28ec2299000000e7
RBP: ffff8800610bdad8 R08: ffff88006f8b6148 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 28ec2299000000df
R13: 0000000000000001 R14: 28ec2299000000e7 R15: ffff8800610bdc08
FS:  00007f814d64f7e0(0000) GS:ffff88000a200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f814ebbe0c8 CR3: 000000006a2e1000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process httpd (pid: 3573, threadinfo ffff8800610bc000, task
ffff88006fc174c0)
Stack:
 28ec2299000000e7 28ec2299000000df ffff8800610bdaf8 ffffffffa02b8ec6
<0> ffff8800610bdb08 ffff880061096e00 ffff8800610bdb28 ffffffffa02af261
<0> ffff8800612dea00 0000000000000000 ffff880061096e00 ffff880061096e60
Call Trace:
 [<ffffffffa02b8ec6>] afs_mutex_enter+0x16/0x50 [openafs]
 [<ffffffffa02af261>] rxi_ResetCall+0xa1/0x460 [openafs]
 [<ffffffffa02b4804>] rxi_NewCall+0x1b4/0x220 [openafs]
 [<ffffffffa02b5290>] rx_NewCall+0x110/0x280 [openafs]
 [<ffffffffa02b5a03>] ? rx_NewConnection+0x1c3/0x1e0 [openafs]
 [<ffffffffa02a8ab7>] RXAFS_FetchStatus+0x47/0x200 [openafs]
 [<ffffffffa0272cee>] ? afs_ConnBySA+0x28e/0x4e0 [openafs]
 [<ffffffffa028ad92>] ? afs_FindVCache+0x4a2/0x5c0 [openafs]
 [<ffffffff81012969>] ? read_tsc+0x9/0x20
 [<ffffffff81098920>] ? getnstimeofday+0x60/0xf0
 [<ffffffffa028b752>] afs_FetchStatus+0x112/0x620 [openafs]
 [<ffffffffa028ffa6>] afs_GetAccessBits+0xf6/0x110 [openafs]
 [<ffffffff81180cfd>] ? __link_path_walk+0x59d/0x820
 [<ffffffff811910e0>] ? mntput_no_expire+0x30/0x110
 [<ffffffffa02900d6>] afs_AccessOK+0x116/0x1c0 [openafs]
 [<ffffffffa02905a1>] afs_access+0x421/0x490 [openafs]
 [<ffffffffa02cb41c>] afs_linux_permission+0x8c/0x120 [openafs]
 [<ffffffff8117ee1e>] inode_permission+0x8e/0xd0
 [<ffffffff8117061a>] sys_chdir+0x4a/0x90
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Code: ff ff 85 c0 75 ed eb d6 66 90 55 48 89 e5 48 83 ec 10 48 89 1c 24 4c
89 64 24 08 0f 1f 44 00 00 48 89 fb e8 a2 f4 ff ff 48 89 df <f0> ff 0f 79 05
e8 25 00 00 00 65 48 8b 04 25 08 cc 00 00 48 2d
RIP  [<ffffffff814dbfd1>] mutex_lock+0x21/0x50
 RSP <ffff8800610bdac8>

Thanks,
-Aaron


>
> --
> Andrew Deason
> adeason@sinenomine.net
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>



-- 
Aaron Knister
Systems Administrator
Division of Information Technology
University of Maryland, Baltimore County
aaronk@umbc.edu

--bcaec501604d436e6704a912037f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, Jul 27, 2011 at 3:05 PM, Andrew Deason <span dir=3D"ltr">&lt;<a hre=
f=3D"mailto:adeason@sinenomine.net">adeason@sinenomine.net</a>&gt;</span> w=
rote:<br><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=
=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">

<div class=3D"im">On Wed, 27 Jul 2011 13:50:44 -0400<br>
Michael Meffie &lt;<a href=3D"mailto:mmeffie@sinenomine.net">mmeffie@sineno=
mine.net</a>&gt; wrote:<br>
<br>
&gt; This looks similar to the panic fixed by Andrew&#39;s recent fix,<br>
&gt;<br>
&gt; =A0 =A0<a href=3D"http://gerrit.openafs.org/5086" target=3D"_blank">ht=
tp://gerrit.openafs.org/5086</a><br>
&gt;<br>
&gt; which fixes a reference count imbalance.<br>
<br>
</div>The panic that that change fixes doesn&#39;t occur (or is very unlike=
ly to<br>
occur) unless you have gerrit 4625; it&#39;s not that.<br>
<br>
Aaron, if you run &#39;log&#39; in crash, what do the last few lines say?<b=
r>
Filter out any sensitive information or whatnot; I wouldn&#39;t expect ther=
e<br>
to be any, but I don&#39;t know what&#39;s in your log.<br></blockquote><di=
v><br></div><div>Hi Andrew,</div><div><br></div><div>Here&#39;s the output =
of the last few, what I think are relavent lines:</div><div><br></div>

<div><div>Modules linked in: openafs(P)(U) ipt_REJECT nf_conntrack_ipv4 nf_=
defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defra=
g_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ppdev parport_=
pc parport vmware_balloon vmxnet3 i2c_piix4 i2c_core sg shpchp ext4 mbcache=
 jbd2 sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic ata_p=
iix dm_mod [last unloaded: scsi_wait_scan]</div>

<div><br></div><div>Modules linked in: openafs(P)(U) ipt_REJECT nf_conntrac=
k_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv=
6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ppde=
v parport_pc parport vmware_balloon vmxnet3 i2c_piix4 i2c_core sg shpchp ex=
t4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_gen=
eric ata_piix dm_mod [last unloaded: scsi_wait_scan]</div>

<div>Pid: 3573, comm: httpd Tainted: P =A0 =A0 =A0 =A0 =A0 ----------------=
 =A0 2.6.32-131.6.1.el6.x86_64 #1 VMware Virtual Platform</div><div>RIP: 00=
10:[&lt;ffffffff814dbfd1&gt;] =A0[&lt;ffffffff814dbfd1&gt;] mutex_lock+0x21=
/0x50</div>

<div>RSP: 0018:ffff8800610bdac8 =A0EFLAGS: 00010246</div><div>RAX: 00000000=
00000000 RBX: 28ec2299000000e7 RCX: 0000000000000001</div><div>RDX: 0000000=
000000000 RSI: 0000000000000001 RDI: 28ec2299000000e7</div><div>RBP: ffff88=
00610bdad8 R08: ffff88006f8b6148 R09: 0000000000000000</div>

<div>R10: 0000000000000000 R11: 0000000000000000 R12: 28ec2299000000df</div=
><div>R13: 0000000000000001 R14: 28ec2299000000e7 R15: ffff8800610bdc08</di=
v><div>FS: =A000007f814d64f7e0(0000) GS:ffff88000a200000(0000) knlGS:000000=
0000000000</div>

<div>CS: =A00010 DS: 0000 ES: 0000 CR0: 0000000080050033</div><div>CR2: 000=
07f814ebbe0c8 CR3: 000000006a2e1000 CR4: 00000000000006f0</div><div>DR0: 00=
00000000000000 DR1: 0000000000000000 DR2: 0000000000000000</div><div>DR3: 0=
000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400</div>

<div>Process httpd (pid: 3573, threadinfo ffff8800610bc000, task ffff88006f=
c174c0)</div><div>Stack:</div><div>=A028ec2299000000e7 28ec2299000000df fff=
f8800610bdaf8 ffffffffa02b8ec6</div><div>&lt;0&gt; ffff8800610bdb08 ffff880=
061096e00 ffff8800610bdb28 ffffffffa02af261</div>

<div>&lt;0&gt; ffff8800612dea00 0000000000000000 ffff880061096e00 ffff88006=
1096e60</div><div>Call Trace:</div><div>=A0[&lt;ffffffffa02b8ec6&gt;] afs_m=
utex_enter+0x16/0x50 [openafs]</div><div>=A0[&lt;ffffffffa02af261&gt;] rxi_=
ResetCall+0xa1/0x460 [openafs]</div>

<div>=A0[&lt;ffffffffa02b4804&gt;] rxi_NewCall+0x1b4/0x220 [openafs]</div><=
div>=A0[&lt;ffffffffa02b5290&gt;] rx_NewCall+0x110/0x280 [openafs]</div><di=
v>=A0[&lt;ffffffffa02b5a03&gt;] ? rx_NewConnection+0x1c3/0x1e0 [openafs]</d=
iv>

<div>=A0[&lt;ffffffffa02a8ab7&gt;] RXAFS_FetchStatus+0x47/0x200 [openafs]</=
div><div>=A0[&lt;ffffffffa0272cee&gt;] ? afs_ConnBySA+0x28e/0x4e0 [openafs]=
</div><div>=A0[&lt;ffffffffa028ad92&gt;] ? afs_FindVCache+0x4a2/0x5c0 [open=
afs]</div>

<div>=A0[&lt;ffffffff81012969&gt;] ? read_tsc+0x9/0x20</div><div>=A0[&lt;ff=
ffffff81098920&gt;] ? getnstimeofday+0x60/0xf0</div><div>=A0[&lt;ffffffffa0=
28b752&gt;] afs_FetchStatus+0x112/0x620 [openafs]</div><div>=A0[&lt;fffffff=
fa028ffa6&gt;] afs_GetAccessBits+0xf6/0x110 [openafs]</div>

<div>=A0[&lt;ffffffff81180cfd&gt;] ? __link_path_walk+0x59d/0x820</div><div=
>=A0[&lt;ffffffff811910e0&gt;] ? mntput_no_expire+0x30/0x110</div><div>=A0[=
&lt;ffffffffa02900d6&gt;] afs_AccessOK+0x116/0x1c0 [openafs]</div><div>=A0[=
&lt;ffffffffa02905a1&gt;] afs_access+0x421/0x490 [openafs]</div>

<div>=A0[&lt;ffffffffa02cb41c&gt;] afs_linux_permission+0x8c/0x120 [openafs=
]</div><div>=A0[&lt;ffffffff8117ee1e&gt;] inode_permission+0x8e/0xd0</div><=
div>=A0[&lt;ffffffff8117061a&gt;] sys_chdir+0x4a/0x90</div><div>=A0[&lt;fff=
fffff8100b172&gt;] system_call_fastpath+0x16/0x1b</div>

<div>Code: ff ff 85 c0 75 ed eb d6 66 90 55 48 89 e5 48 83 ec 10 48 89 1c 2=
4 4c 89 64 24 08 0f 1f 44 00 00 48 89 fb e8 a2 f4 ff ff 48 89 df &lt;f0&gt;=
 ff 0f 79 05 e8 25 00 00 00 65 48 8b 04 25 08 cc 00 00 48 2d=A0</div><div>

RIP =A0[&lt;ffffffff814dbfd1&gt;] mutex_lock+0x21/0x50</div><div>=A0RSP &lt=
;ffff8800610bdac8&gt;</div></div><div><br></div><div>Thanks,</div><div>-Aar=
on</div><div>=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0=
 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">


<font color=3D"#888888"><br>
--<br>
Andrew Deason<br>
<a href=3D"mailto:adeason@sinenomine.net">adeason@sinenomine.net</a><br>
<br>
_______________________________________________<br>
OpenAFS-info mailing list<br>
<a href=3D"mailto:OpenAFS-info@openafs.org">OpenAFS-info@openafs.org</a><br=
>
<a href=3D"https://lists.openafs.org/mailman/listinfo/openafs-info" target=
=3D"_blank">https://lists.openafs.org/mailman/listinfo/openafs-info</a><br>
</font></blockquote></div><br><br clear=3D"all"><br>-- <br>Aaron Knister<br=
>Systems Administrator<br>Division of Information Technology<br>University =
of Maryland, Baltimore County<br><a href=3D"mailto:aaronk@umbc.edu" target=
=3D"_blank">aaronk@umbc.edu</a><br>



--bcaec501604d436e6704a912037f--