[OpenAFS] Re: kernel panic in 1.4.14.1 on rhel6
Aaron Knister
aaronk@umbc.edu
Wed, 27 Jul 2011 15:25:15 -0400
--bcaec501604d436e6704a912037f
Content-Type: text/plain; charset=ISO-8859-1
On Wed, Jul 27, 2011 at 3:05 PM, Andrew Deason <adeason@sinenomine.net>wrote:
> On Wed, 27 Jul 2011 13:50:44 -0400
> Michael Meffie <mmeffie@sinenomine.net> wrote:
>
> > This looks similar to the panic fixed by Andrew's recent fix,
> >
> > http://gerrit.openafs.org/5086
> >
> > which fixes a reference count imbalance.
>
> The panic that that change fixes doesn't occur (or is very unlikely to
> occur) unless you have gerrit 4625; it's not that.
>
> Aaron, if you run 'log' in crash, what do the last few lines say?
> Filter out any sensitive information or whatnot; I wouldn't expect there
> to be any, but I don't know what's in your log.
>
Hi Andrew,
Here's the output of the last few, what I think are relavent lines:
Modules linked in: openafs(P)(U) ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4
iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6
xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ppdev parport_pc
parport vmware_balloon vmxnet3 i2c_piix4 i2c_core sg shpchp ext4 mbcache
jbd2 sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic
ata_piix dm_mod [last unloaded: scsi_wait_scan]
Modules linked in: openafs(P)(U) ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4
iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6
xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ppdev parport_pc
parport vmware_balloon vmxnet3 i2c_piix4 i2c_core sg shpchp ext4 mbcache
jbd2 sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic
ata_piix dm_mod [last unloaded: scsi_wait_scan]
Pid: 3573, comm: httpd Tainted: P ----------------
2.6.32-131.6.1.el6.x86_64 #1 VMware Virtual Platform
RIP: 0010:[<ffffffff814dbfd1>] [<ffffffff814dbfd1>] mutex_lock+0x21/0x50
RSP: 0018:ffff8800610bdac8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 28ec2299000000e7 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 28ec2299000000e7
RBP: ffff8800610bdad8 R08: ffff88006f8b6148 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 28ec2299000000df
R13: 0000000000000001 R14: 28ec2299000000e7 R15: ffff8800610bdc08
FS: 00007f814d64f7e0(0000) GS:ffff88000a200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f814ebbe0c8 CR3: 000000006a2e1000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process httpd (pid: 3573, threadinfo ffff8800610bc000, task
ffff88006fc174c0)
Stack:
28ec2299000000e7 28ec2299000000df ffff8800610bdaf8 ffffffffa02b8ec6
<0> ffff8800610bdb08 ffff880061096e00 ffff8800610bdb28 ffffffffa02af261
<0> ffff8800612dea00 0000000000000000 ffff880061096e00 ffff880061096e60
Call Trace:
[<ffffffffa02b8ec6>] afs_mutex_enter+0x16/0x50 [openafs]
[<ffffffffa02af261>] rxi_ResetCall+0xa1/0x460 [openafs]
[<ffffffffa02b4804>] rxi_NewCall+0x1b4/0x220 [openafs]
[<ffffffffa02b5290>] rx_NewCall+0x110/0x280 [openafs]
[<ffffffffa02b5a03>] ? rx_NewConnection+0x1c3/0x1e0 [openafs]
[<ffffffffa02a8ab7>] RXAFS_FetchStatus+0x47/0x200 [openafs]
[<ffffffffa0272cee>] ? afs_ConnBySA+0x28e/0x4e0 [openafs]
[<ffffffffa028ad92>] ? afs_FindVCache+0x4a2/0x5c0 [openafs]
[<ffffffff81012969>] ? read_tsc+0x9/0x20
[<ffffffff81098920>] ? getnstimeofday+0x60/0xf0
[<ffffffffa028b752>] afs_FetchStatus+0x112/0x620 [openafs]
[<ffffffffa028ffa6>] afs_GetAccessBits+0xf6/0x110 [openafs]
[<ffffffff81180cfd>] ? __link_path_walk+0x59d/0x820
[<ffffffff811910e0>] ? mntput_no_expire+0x30/0x110
[<ffffffffa02900d6>] afs_AccessOK+0x116/0x1c0 [openafs]
[<ffffffffa02905a1>] afs_access+0x421/0x490 [openafs]
[<ffffffffa02cb41c>] afs_linux_permission+0x8c/0x120 [openafs]
[<ffffffff8117ee1e>] inode_permission+0x8e/0xd0
[<ffffffff8117061a>] sys_chdir+0x4a/0x90
[<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Code: ff ff 85 c0 75 ed eb d6 66 90 55 48 89 e5 48 83 ec 10 48 89 1c 24 4c
89 64 24 08 0f 1f 44 00 00 48 89 fb e8 a2 f4 ff ff 48 89 df <f0> ff 0f 79 05
e8 25 00 00 00 65 48 8b 04 25 08 cc 00 00 48 2d
RIP [<ffffffff814dbfd1>] mutex_lock+0x21/0x50
RSP <ffff8800610bdac8>
Thanks,
-Aaron
>
> --
> Andrew Deason
> adeason@sinenomine.net
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>
--
Aaron Knister
Systems Administrator
Division of Information Technology
University of Maryland, Baltimore County
aaronk@umbc.edu
--bcaec501604d436e6704a912037f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
On Wed, Jul 27, 2011 at 3:05 PM, Andrew Deason <span dir=3D"ltr"><<a hre=
f=3D"mailto:adeason@sinenomine.net">adeason@sinenomine.net</a>></span> w=
rote:<br><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=
=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class=3D"im">On Wed, 27 Jul 2011 13:50:44 -0400<br>
Michael Meffie <<a href=3D"mailto:mmeffie@sinenomine.net">mmeffie@sineno=
mine.net</a>> wrote:<br>
<br>
> This looks similar to the panic fixed by Andrew's recent fix,<br>
><br>
> =A0 =A0<a href=3D"http://gerrit.openafs.org/5086" target=3D"_blank">ht=
tp://gerrit.openafs.org/5086</a><br>
><br>
> which fixes a reference count imbalance.<br>
<br>
</div>The panic that that change fixes doesn't occur (or is very unlike=
ly to<br>
occur) unless you have gerrit 4625; it's not that.<br>
<br>
Aaron, if you run 'log' in crash, what do the last few lines say?<b=
r>
Filter out any sensitive information or whatnot; I wouldn't expect ther=
e<br>
to be any, but I don't know what's in your log.<br></blockquote><di=
v><br></div><div>Hi Andrew,</div><div><br></div><div>Here's the output =
of the last few, what I think are relavent lines:</div><div><br></div>
<div><div>Modules linked in: openafs(P)(U) ipt_REJECT nf_conntrack_ipv4 nf_=
defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defra=
g_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ppdev parport_=
pc parport vmware_balloon vmxnet3 i2c_piix4 i2c_core sg shpchp ext4 mbcache=
jbd2 sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic ata_p=
iix dm_mod [last unloaded: scsi_wait_scan]</div>
<div><br></div><div>Modules linked in: openafs(P)(U) ipt_REJECT nf_conntrac=
k_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv=
6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ppde=
v parport_pc parport vmware_balloon vmxnet3 i2c_piix4 i2c_core sg shpchp ex=
t4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_gen=
eric ata_piix dm_mod [last unloaded: scsi_wait_scan]</div>
<div>Pid: 3573, comm: httpd Tainted: P =A0 =A0 =A0 =A0 =A0 ----------------=
=A0 2.6.32-131.6.1.el6.x86_64 #1 VMware Virtual Platform</div><div>RIP: 00=
10:[<ffffffff814dbfd1>] =A0[<ffffffff814dbfd1>] mutex_lock+0x21=
/0x50</div>
<div>RSP: 0018:ffff8800610bdac8 =A0EFLAGS: 00010246</div><div>RAX: 00000000=
00000000 RBX: 28ec2299000000e7 RCX: 0000000000000001</div><div>RDX: 0000000=
000000000 RSI: 0000000000000001 RDI: 28ec2299000000e7</div><div>RBP: ffff88=
00610bdad8 R08: ffff88006f8b6148 R09: 0000000000000000</div>
<div>R10: 0000000000000000 R11: 0000000000000000 R12: 28ec2299000000df</div=
><div>R13: 0000000000000001 R14: 28ec2299000000e7 R15: ffff8800610bdc08</di=
v><div>FS: =A000007f814d64f7e0(0000) GS:ffff88000a200000(0000) knlGS:000000=
0000000000</div>
<div>CS: =A00010 DS: 0000 ES: 0000 CR0: 0000000080050033</div><div>CR2: 000=
07f814ebbe0c8 CR3: 000000006a2e1000 CR4: 00000000000006f0</div><div>DR0: 00=
00000000000000 DR1: 0000000000000000 DR2: 0000000000000000</div><div>DR3: 0=
000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400</div>
<div>Process httpd (pid: 3573, threadinfo ffff8800610bc000, task ffff88006f=
c174c0)</div><div>Stack:</div><div>=A028ec2299000000e7 28ec2299000000df fff=
f8800610bdaf8 ffffffffa02b8ec6</div><div><0> ffff8800610bdb08 ffff880=
061096e00 ffff8800610bdb28 ffffffffa02af261</div>
<div><0> ffff8800612dea00 0000000000000000 ffff880061096e00 ffff88006=
1096e60</div><div>Call Trace:</div><div>=A0[<ffffffffa02b8ec6>] afs_m=
utex_enter+0x16/0x50 [openafs]</div><div>=A0[<ffffffffa02af261>] rxi_=
ResetCall+0xa1/0x460 [openafs]</div>
<div>=A0[<ffffffffa02b4804>] rxi_NewCall+0x1b4/0x220 [openafs]</div><=
div>=A0[<ffffffffa02b5290>] rx_NewCall+0x110/0x280 [openafs]</div><di=
v>=A0[<ffffffffa02b5a03>] ? rx_NewConnection+0x1c3/0x1e0 [openafs]</d=
iv>
<div>=A0[<ffffffffa02a8ab7>] RXAFS_FetchStatus+0x47/0x200 [openafs]</=
div><div>=A0[<ffffffffa0272cee>] ? afs_ConnBySA+0x28e/0x4e0 [openafs]=
</div><div>=A0[<ffffffffa028ad92>] ? afs_FindVCache+0x4a2/0x5c0 [open=
afs]</div>
<div>=A0[<ffffffff81012969>] ? read_tsc+0x9/0x20</div><div>=A0[<ff=
ffffff81098920>] ? getnstimeofday+0x60/0xf0</div><div>=A0[<ffffffffa0=
28b752>] afs_FetchStatus+0x112/0x620 [openafs]</div><div>=A0[<fffffff=
fa028ffa6>] afs_GetAccessBits+0xf6/0x110 [openafs]</div>
<div>=A0[<ffffffff81180cfd>] ? __link_path_walk+0x59d/0x820</div><div=
>=A0[<ffffffff811910e0>] ? mntput_no_expire+0x30/0x110</div><div>=A0[=
<ffffffffa02900d6>] afs_AccessOK+0x116/0x1c0 [openafs]</div><div>=A0[=
<ffffffffa02905a1>] afs_access+0x421/0x490 [openafs]</div>
<div>=A0[<ffffffffa02cb41c>] afs_linux_permission+0x8c/0x120 [openafs=
]</div><div>=A0[<ffffffff8117ee1e>] inode_permission+0x8e/0xd0</div><=
div>=A0[<ffffffff8117061a>] sys_chdir+0x4a/0x90</div><div>=A0[<fff=
fffff8100b172>] system_call_fastpath+0x16/0x1b</div>
<div>Code: ff ff 85 c0 75 ed eb d6 66 90 55 48 89 e5 48 83 ec 10 48 89 1c 2=
4 4c 89 64 24 08 0f 1f 44 00 00 48 89 fb e8 a2 f4 ff ff 48 89 df <f0>=
ff 0f 79 05 e8 25 00 00 00 65 48 8b 04 25 08 cc 00 00 48 2d=A0</div><div>
RIP =A0[<ffffffff814dbfd1>] mutex_lock+0x21/0x50</div><div>=A0RSP <=
;ffff8800610bdac8></div></div><div><br></div><div>Thanks,</div><div>-Aar=
on</div><div>=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0=
0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<font color=3D"#888888"><br>
--<br>
Andrew Deason<br>
<a href=3D"mailto:adeason@sinenomine.net">adeason@sinenomine.net</a><br>
<br>
_______________________________________________<br>
OpenAFS-info mailing list<br>
<a href=3D"mailto:OpenAFS-info@openafs.org">OpenAFS-info@openafs.org</a><br=
>
<a href=3D"https://lists.openafs.org/mailman/listinfo/openafs-info" target=
=3D"_blank">https://lists.openafs.org/mailman/listinfo/openafs-info</a><br>
</font></blockquote></div><br><br clear=3D"all"><br>-- <br>Aaron Knister<br=
>Systems Administrator<br>Division of Information Technology<br>University =
of Maryland, Baltimore County<br><a href=3D"mailto:aaronk@umbc.edu" target=
=3D"_blank">aaronk@umbc.edu</a><br>
--bcaec501604d436e6704a912037f--