[OpenAFS] /afs area is hanging
Mark Henry
mark.henry@infoprint.com
Tue, 12 May 2009 14:37:48 -0600
This is a multipart message in MIME format.
--=_alternative 00715152872575B4_=
Content-Type: text/plain; charset="US-ASCII"
> I took the liberty to paste the interesting parts to
> http://pastebin.com/m53578cd5. Notice the bottom, which was the original
> bottom as well. Mark, you've been asked to look at dmesg before this, so
I
> suppose this didn't happen before you tried this call-tracing?
> Besides, it would be interesting if an upgrade to 1.4.10 makes the
problem
> go away. Can you try that?
We upgraded to 1.4.10. We got the same errors. Here is the cmdebug
output:
=> cmdebug HOSTNAME
** Cache entry @ 0x172d1880 for 2.536870937.28.1820
[afs.dev.infoprint.com]
locks: (none_waiting, write_locked(pid:-246839824 at:681))
3 bytes DV 0 refcnt 2
callback 22174d40 expires 1242158668
0 opens 0 writers
mount point
states (0x5), stat'd, read-only
The volume # 536870937 above just happens to be root.cell but it has been
different based on which dir I do the ls in.
> Finally, it looks like ls and sshd got locked up trying to determine
your
> client machine's home cell. I can see that happening (only) if no cell
has
> been set at that point. The output of fs wscell would be interesting in
> this situation, but I'm not sure wether that would lock up as well (and
if
> it is at all helpful).
We tried the fs wscell command. It worked fine if the fs command was
local and hung if the fs was being retrieved from afs.
Also, here is a bit of interesting output from dmesg when the system is
hung:
AssertProcessEntry: pohm_main, pid=6518
openafs: Can't open inode 95550
------------[ cut here ]------------
kernel BUG at
/compile/openafs-1.4.10/src/libafs/MODLOAD-2.6.22.5-31-default-MP/osi_file.c:87!
invalid opcode: 0000 [1] SMP
last sysfs file: /class/scsi_host/host0/model
CPU 6
Modules linked in: tun iptable_filter ip_tables x_tables amk ipv6
libafs(P) microcode firmware_class usbhid hid ff_memless tp_dd af_packet
apparmor ext2 loop dm_mod parport_pc parport bnx2 rtc_cmos rtc_core
i2c_i801 rtc_lib ide_cd i2c_core cdrom shpchp tg3 pci_hotplug container
button sg ehci_hcd uhci_hcd usbcore sd_mod ata_piix libata edd ext3
mbcache jbd fan aacraid scsi_mod piix ide_core thermal processor
Pid: 6519, comm: ASCIIMast Tainted: P N 2.6.22.5-31-default #1
RIP: 0010:[<ffffffff8834dde9>] [<ffffffff8834dde9>]
:libafs:osi_UFSOpen+0x155/0x1f2
RSP: 0000:ffff81031de299c8 EFLAGS: 00010296
RAX: 0000000000000023 RBX: ffff81031a92b418 RCX: 0000000000000001
RDX: ffffffff804bdfe8 RSI: 0000000000000096 RDI: ffffffff804bdfe0
RBP: ffff81042145f000 R08: 0000000000000001 R09: ffff810001086bc0
R10: 0000000000000046 R11: ffff81042fa74ec0 R12: ffff81042af37000
R13: 000000000001753e R14: 000000000001753e R15: 0000000000000003
FS: 0000000000000000(0000) GS:ffff81042ee9f1c0(0000)
knlGS:0000000000000000
CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000f5d6050c CR3: 000000039e0b9000 CR4: 00000000000006e0
Process ASCIIMast (pid: 6519, threadinfo ffff81031de28000, task
ffff81031ad770c0)
Stack: ffffc2000770eb90 ffffc2000770caa8 ffff81042a631000
ffff8104172d1880
ffff81039800839c ffffffff8832f678 0000000300000000 0000000000000003
0000000000000000 ffffc2000770eb90 ffff8104172d1880 0000000000000000
Call Trace:
[<ffffffff8832f678>] :libafs:afs_UFSHandleLink+0xf7/0x1bd
[<ffffffff8832aadf>] :libafs:afs_lookup+0xbb2/0x115f
[<ffffffff88352713>] :libafs:afs_linux_dentry_revalidate+0x422/0x434
[<ffffffff883521ac>] :libafs:afs_linux_lookup+0x85/0x1ca
[<ffffffff883188c7>] :libafs:PagInCred+0x30/0xa9
[<ffffffff8028f972>] do_lookup+0xc4/0x1ae
[<ffffffff8029179a>] __link_path_walk+0x36c/0xd8b
[<ffffffff80299115>] dput+0x26/0x115
[<ffffffff80292066>] __link_path_walk+0xc38/0xd8b
[<ffffffff80292211>] link_path_walk+0x58/0xe0
[<ffffffff802877e9>] do_filp_open+0x1c/0x3d
[<ffffffff80292589>] do_path_lookup+0x1ab/0x227
[<ffffffff80292fb9>] __path_lookup_intent_open+0x56/0x97
[<ffffffff80293148>] open_namei+0x7a/0x674
[<ffffffff802877e9>] do_filp_open+0x1c/0x3d
[<ffffffff8028784e>] do_sys_open+0x44/0xc1
[<ffffffff80220bb2>] ia32_sysret+0x0/0xa
We still can't seem to get this system to stop hanging in the /afs area.
Mark Henry
_____________________________________________________________________________
"This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you." _____________________________________________________________________________
--=_alternative 00715152872575B4_=
Content-Type: text/html; charset="US-ASCII"
<br><tt><font size=2><br>
<br>
> I took the liberty to paste the interesting parts to <br>
> http://pastebin.com/m53578cd5. Notice the bottom, which was the original<br>
> bottom as well. Mark, you've been asked to look at dmesg before this,
so I<br>
> suppose this didn't happen before you tried this call-tracing?<br>
<br>
> Besides, it would be interesting if an upgrade to 1.4.10 makes the
problem<br>
> go away. Can you try that?<br>
</font></tt>
<br><tt><font size=2>We upgraded to 1.4.10. We got the same errors.
Here is the cmdebug output:</font></tt>
<br>
<br><tt><font size=2>=> cmdebug HOSTNAME</font></tt>
<br><tt><font size=2>** Cache entry @ 0x172d1880 for 2.536870937.28.1820
[afs.dev.infoprint.com]</font></tt>
<br><tt><font size=2> locks: (none_waiting, write_locked(pid:-246839824
at:681))</font></tt>
<br><tt><font size=2> 3
bytes DV 0 refcnt
2</font></tt>
<br><tt><font size=2> callback 22174d40 expires 1242158668</font></tt>
<br><tt><font size=2> 0 opens 0 writers</font></tt>
<br><tt><font size=2> mount point</font></tt>
<br><tt><font size=2> states (0x5), stat'd, read-only</font></tt>
<br>
<br><tt><font size=2>The volume # 536870937 above just happens to be root.cell
but it has been different based on which dir I do the ls in.</font></tt>
<br><tt><font size=2><br>
> Finally, it looks like ls and sshd got locked up trying to determine
your<br>
> client machine's home cell. I can see that happening (only) if no
cell has <br>
> been set at that point. The output of fs wscell would be interesting
in <br>
> this situation, but I'm not sure wether that would lock up as well
(and if <br>
> it is at all helpful).</font></tt>
<br>
<br><tt><font size=2>We tried the fs wscell command. It worked fine
if the fs command was local and hung if the fs was being retrieved from
afs.</font></tt>
<br>
<br><tt><font size=2>Also, here is a bit of interesting output from dmesg
when the system is hung:</font></tt>
<br>
<br><tt><font size=2>AssertProcessEntry: pohm_main, pid=6518<br>
openafs: Can't open inode 95550<br>
------------[ cut here ]------------<br>
kernel BUG at /compile/openafs-1.4.10/src/libafs/MODLOAD-2.6.22.5-31-default-MP/osi_file.c:87!<br>
invalid opcode: 0000 [1] SMP <br>
last sysfs file: /class/scsi_host/host0/model<br>
CPU 6 <br>
Modules linked in: tun iptable_filter ip_tables x_tables amk ipv6 libafs(P)
microcode firmware_class usbhid hid ff_memless tp_dd af_packet apparmor
ext2 loop dm_mod parport_pc parport bnx2 rtc_cmos rtc_core i2c_i801 rtc_lib
ide_cd i2c_core cdrom shpchp tg3 pci_hotplug container button sg ehci_hcd
uhci_hcd usbcore sd_mod ata_piix libata edd ext3 mbcache jbd fan aacraid
scsi_mod piix ide_core thermal processor<br>
Pid: 6519, comm: ASCIIMast Tainted: P N 2.6.22.5-31-default
#1<br>
RIP: 0010:[<ffffffff8834dde9>] [<ffffffff8834dde9>] :libafs:osi_UFSOpen+0x155/0x1f2<br>
RSP: 0000:ffff81031de299c8 EFLAGS: 00010296<br>
RAX: 0000000000000023 RBX: ffff81031a92b418 RCX: 0000000000000001<br>
RDX: ffffffff804bdfe8 RSI: 0000000000000096 RDI: ffffffff804bdfe0<br>
RBP: ffff81042145f000 R08: 0000000000000001 R09: ffff810001086bc0<br>
R10: 0000000000000046 R11: ffff81042fa74ec0 R12: ffff81042af37000<br>
R13: 000000000001753e R14: 000000000001753e R15: 0000000000000003<br>
FS: 0000000000000000(0000) GS:ffff81042ee9f1c0(0000) knlGS:0000000000000000<br>
CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b<br>
CR2: 00000000f5d6050c CR3: 000000039e0b9000 CR4: 00000000000006e0<br>
Process ASCIIMast (pid: 6519, threadinfo ffff81031de28000, task ffff81031ad770c0)<br>
Stack: ffffc2000770eb90 ffffc2000770caa8 ffff81042a631000 ffff8104172d1880<br>
ffff81039800839c ffffffff8832f678 0000000300000000 0000000000000003<br>
0000000000000000 ffffc2000770eb90 ffff8104172d1880 0000000000000000<br>
Call Trace:<br>
[<ffffffff8832f678>] :libafs:afs_UFSHandleLink+0xf7/0x1bd<br>
[<ffffffff8832aadf>] :libafs:afs_lookup+0xbb2/0x115f<br>
[<ffffffff88352713>] :libafs:afs_linux_dentry_revalidate+0x422/0x434<br>
[<ffffffff883521ac>] :libafs:afs_linux_lookup+0x85/0x1ca<br>
[<ffffffff883188c7>] :libafs:PagInCred+0x30/0xa9<br>
[<ffffffff8028f972>] do_lookup+0xc4/0x1ae<br>
[<ffffffff8029179a>] __link_path_walk+0x36c/0xd8b<br>
[<ffffffff80299115>] dput+0x26/0x115<br>
[<ffffffff80292066>] __link_path_walk+0xc38/0xd8b<br>
[<ffffffff80292211>] link_path_walk+0x58/0xe0<br>
[<ffffffff802877e9>] do_filp_open+0x1c/0x3d<br>
[<ffffffff80292589>] do_path_lookup+0x1ab/0x227<br>
[<ffffffff80292fb9>] __path_lookup_intent_open+0x56/0x97<br>
[<ffffffff80293148>] open_namei+0x7a/0x674<br>
[<ffffffff802877e9>] do_filp_open+0x1c/0x3d<br>
[<ffffffff8028784e>] do_sys_open+0x44/0xc1<br>
[<ffffffff80220bb2>] ia32_sysret+0x0/0xa<br>
<br>
<br>
We still can't seem to get this system to stop hanging in the /afs area.<br>
</font></tt><font size=2 face="sans-serif"><br>
Mark Henry<br>
</font>
<br>
<BR>
_____________________________________________________________________________<BR>
"This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you." _____________________________________________________________________________<BR>
--=_alternative 00715152872575B4_=--