[OpenAFS] /afs area is hanging

[Derrick Brashear]

On Tue, May 12, 2009 at 4:37 PM, Mark Henry <mark.henry@infoprint.com> wrot=
e:
>
>
>
>> I took the liberty to paste the interesting parts to
>> http://pastebin.com/m53578cd5. Notice the bottom, which was the original
>> bottom as well. Mark, you've been asked to look at dmesg before this, so=
 I
>> suppose this didn't happen before you tried this call-tracing?
>
>> Besides, it would be interesting if an upgrade to 1.4.10 makes the probl=
em
>> go away. Can you try that?
>
> We upgraded to 1.4.10. =A0We got the same errors. =A0Here is the cmdebug =
output:
>
> =3D> cmdebug HOSTNAME
> ** Cache entry @ 0x172d1880 for 2.536870937.28.1820 [afs.dev.infoprint.co=
m]
> =A0 =A0 locks: (none_waiting, write_locked(pid:-246839824 at:681))
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A03 bytes =A0DV =A0 =A0 =A0 =A0 =A0 =A00 =A0=
refcnt =A0 =A0 2
> =A0 =A0 callback 22174d40 =A0 expires 1242158668
> =A0 =A0 0 opens =A0 =A0 0 writers
> =A0 =A0 mount point
> =A0 =A0 states (0x5), stat'd, read-only

which is afs_HandleLink, called from afs_lookup. Memcache or disk
cache? (different implementations of that function)
>
> The volume # 536870937 above just happens to be root.cell but it has been
> different based on which dir I do the ls in.
>
>> Finally, it looks like ls and sshd got locked up trying to determine you=
r
>> client machine's home cell. I can see that happening (only) if no cell h=
as
>> been set at that point. The output of fs wscell would be interesting in
>> this situation, but I'm not sure wether that would lock up as well (and =
if
>> it is at all helpful).
>
> We tried the fs wscell command. =A0It worked fine if the fs command was l=
ocal
> and hung if the fs was being retrieved from afs.

well, if afs is unhappy, one presumes afs is unhappy.

> Also, here is a bit of interesting output from dmesg when the system is
> hung:

ah, that'd be disk cache.

> AssertProcessEntry: pohm_main, pid=3D6518
> openafs: Can't open inode 95550

yes, that's very interesting. you've probably told us what we need to.
you got an oops in the thread holding the lock, the machine will never
recover. what filesystem is behind your afs cache?

> ------------[ cut here ]------------
> kernel BUG at
> /compile/openafs-1.4.10/src/libafs/MODLOAD-2.6.22.5-31-default-MP/osi_fil=
e.c:87!
> invalid opcode: 0000 [1] SMP
> last sysfs file: /class/scsi_host/host0/model
> CPU 6
> Modules linked in: tun iptable_filter ip_tables x_tables amk ipv6 libafs(=
P)
> microcode firmware_class usbhid hid ff_memless tp_dd af_packet apparmor e=
xt2
> loop dm_mod parport_pc parport bnx2 rtc_cmos rtc_core i2c_i801 rtc_lib
> ide_cd i2c_core cdrom shpchp tg3 pci_hotplug container button sg ehci_hcd
> uhci_hcd usbcore sd_mod ata_piix libata edd ext3 mbcache jbd fan aacraid
> scsi_mod piix ide_core thermal processor
> Pid: 6519, comm: ASCIIMast Tainted: P =A0 =A0 =A0N 2.6.22.5-31-default #1
> RIP: 0010:[<ffffffff8834dde9>] =A0[<ffffffff8834dde9>]
> :libafs:osi_UFSOpen+0x155/0x1f2
> RSP: 0000:ffff81031de299c8 =A0EFLAGS: 00010296
> RAX: 0000000000000023 RBX: ffff81031a92b418 RCX: 0000000000000001
> RDX: ffffffff804bdfe8 RSI: 0000000000000096 RDI: ffffffff804bdfe0
> RBP: ffff81042145f000 R08: 0000000000000001 R09: ffff810001086bc0
> R10: 0000000000000046 R11: ffff81042fa74ec0 R12: ffff81042af37000
> R13: 000000000001753e R14: 000000000001753e R15: 0000000000000003
> FS: =A00000000000000000(0000) GS:ffff81042ee9f1c0(0000) knlGS:00000000000=
00000
> CS: =A00010 DS: 002b ES: 002b CR0: 000000008005003b
> CR2: 00000000f5d6050c CR3: 000000039e0b9000 CR4: 00000000000006e0
> Process ASCIIMast (pid: 6519, threadinfo ffff81031de28000, task
> ffff81031ad770c0)
> Stack: =A0ffffc2000770eb90 ffffc2000770caa8 ffff81042a631000 ffff8104172d=
1880
> ffff81039800839c ffffffff8832f678 0000000300000000 0000000000000003
> 0000000000000000 ffffc2000770eb90 ffff8104172d1880 0000000000000000
> Call Trace:
> [<ffffffff8832f678>] :libafs:afs_UFSHandleLink+0xf7/0x1bd
> [<ffffffff8832aadf>] :libafs:afs_lookup+0xbb2/0x115f
> [<ffffffff88352713>] :libafs:afs_linux_dentry_revalidate+0x422/0x434
> [<ffffffff883521ac>] :libafs:afs_linux_lookup+0x85/0x1ca
> [<ffffffff883188c7>] :libafs:PagInCred+0x30/0xa9
> [<ffffffff8028f972>] do_lookup+0xc4/0x1ae
> [<ffffffff8029179a>] __link_path_walk+0x36c/0xd8b
> [<ffffffff80299115>] dput+0x26/0x115
> [<ffffffff80292066>] __link_path_walk+0xc38/0xd8b
> [<ffffffff80292211>] link_path_walk+0x58/0xe0
> [<ffffffff802877e9>] do_filp_open+0x1c/0x3d
> [<ffffffff80292589>] do_path_lookup+0x1ab/0x227
> [<ffffffff80292fb9>] __path_lookup_intent_open+0x56/0x97
> [<ffffffff80293148>] open_namei+0x7a/0x674
> [<ffffffff802877e9>] do_filp_open+0x1c/0x3d
> [<ffffffff8028784e>] do_sys_open+0x44/0xc1
> [<ffffffff80220bb2>] ia32_sysret+0x0/0xa
>
>
> We still can't seem to get this system to stop hanging in the /afs area.
>
> Mark Henry
>
>
> _________________________________________________________________________=
____
> "This message and any attachments are solely for the intended recipient a=
nd
> may contain confidential or privileged information. If you are not the
> intended recipient, any disclosure, copying, use, or distribution of the
> information included in this message and any attachments is prohibited. I=
f
> you have received this communication in error, please notify us by reply
> e-mail and immediately and permanently delete this message and any
> attachments. Thank you."
> _________________________________________________________________________=
____
>



--=20
Derrick