[OpenAFS] Stack overflows on Red Hat 2.4.20 kernels

[Steve Snodgrass]

--PEIAKu/WMn1b1Hv9
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Greetings,

I suspect this is the same problem that Lee Damon posted about at the end
of June, but he was never able to produce a full stack dump.  I have lots
of them.  :-)

Quite a few of our servers, all of which run OpenAFS, have started stack
dumping and then (sometimes) crashing since Red Hat released their 2.4.20
kernel series for Red Hat 7.3.  This happens with OpenAFS 1.2.9 (I'm pretty
sure 1.2.10 is affected as well) with kernels 2.4.20-13, 2.4.20-18, and
2.4.20-19.  I have not tested 2.4.20-20 yet.  I first saw the problem on
June 5th and it has continued up to the present; I now have 22 stack dumps
from 11 different machines.  Kernel 2.4.18-27 does not exhibit this problem.

Some common factors: all of the stack dumps start with a sys_stat64 call
and involve OpenAFS and netfilter calls.  All of these system are running
netfilter/iptables, which may be a contributing factor.  Interestingly, this
problem may not be isolated to OpenAFS; it appears that someone got hit by
an NFS version of it, see this Red Hat bug:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=91566

I have no idea what the root cause of this is; it may be some complex
kernel/AFS/netfilter interaction, but if anyone has any ideas I'd be much
obliged to hear them.  I've included one stack dump; I have plenty more if
they are desired.  Unfortunately Red Hat isn't too interested in my problem
since I'm running OpenAFS.

-- 
Steve Snodgrass * ssnodgra@pheran.com * Network and Unix Guru(?) at Large
Geek Code: GCS d? s: a C++ U++++$ P+++ L++ w PS+ 5++ b++ DI+ D++ e++ r+++ y+*
"If you want to be somebody else, change your mind."  -Sister Hazel

--PEIAKu/WMn1b1Hv9
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="dump20030606.txt"

2.4.20-13
[-- MARK -- Fri Jun  6 11:30:00 2003]
do_IRQ: stack overflow: 708
c0251065 000002c4 00000000 de19fa80 00000000 de19fabc da8bed00 c024a50c 
       de19fa80 ddb2cf00 de19faac 00000000 de19fabc da8bed00 00000120 420e0018 
       40010018 ffffff11 c0231c59 00000010 00000246 de19faac 00000030 420e1222 
Call Trace:   [<c0231c59>] packet_rcv_spkt [kernel] 0xd9 (0xc4bc89a4))
[<c01e94df>] dev_queue_xmit_nit [kernel] 0x8f (0xc4bc8a04))
[<c01e979d>] dev_queue_xmit [kernel] 0x1ed (0xc4bc8a24))
[<c01edd4f>] neigh_resolve_output [kernel] 0x15f (0xc4bc8a68))
[<c01edd8a>] neigh_resolve_output [kernel] 0x19a (0xc4bc8a7c))
[<e08f7416>] ip_refrag [ip_conntrack] 0x26 (0xc4bc8a98))
[<c0202d80>] ip_finish_output2 [kernel] 0x0 (0xc4bc8aac))
[<c0202d80>] ip_finish_output2 [kernel] 0x0 (0xc4bc8ab8))
[<c01f045e>] nf_iterate [kernel] 0x2e (0xc4bc8abc))
[<c0202d80>] ip_finish_output2 [kernel] 0x0 (0xc4bc8ad4))
[<c0202e3d>] ip_finish_output2 [kernel] 0xbd (0xc4bc8ad8))
[<c0202d80>] ip_finish_output2 [kernel] 0x0 (0xc4bc8ae0))
[<c01f078f>] nf_hook_slow [kernel] 0xcf (0xc4bc8ae4))
[<c01f07c6>] nf_hook_slow [kernel] 0x106 (0xc4bc8afc))
[<c0202d70>] output_maybe_reroute [kernel] 0x0 (0xc4bc8b28))
[<c0202d70>] output_maybe_reroute [kernel] 0x0 (0xc4bc8b38))
[<c0201838>] ip_output [kernel] 0x158 (0xc4bc8b3c))
[<c0202d80>] ip_finish_output2 [kernel] 0x0 (0xc4bc8b54))
[<c0202d70>] output_maybe_reroute [kernel] 0x0 (0xc4bc8b60))
[<c0202d70>] output_maybe_reroute [kernel] 0x0 (0xc4bc8b70))
[<c01f078f>] nf_hook_slow [kernel] 0xcf (0xc4bc8b74))
[<c0202d7b>] output_maybe_reroute [kernel] 0xb (0xc4bc8b84))
[<c01f07c6>] nf_hook_slow [kernel] 0x106 (0xc4bc8b8c))
[<c02025b9>] ip_build_xmit [kernel] 0x2f9 (0xc4bc8bcc))
[<c0202d70>] output_maybe_reroute [kernel] 0x0 (0xc4bc8be4))
[<e0982ccb>] vlan_dev_hwaccel_hard_start_xmit [8021q] 0x7b (0xc4bc8bfc))
[<c021dfcf>] udp_sendmsg [kernel] 0x3cf (0xc4bc8c20))
[<c021dad0>] udp_getfrag [kernel] 0x0 (0xc4bc8c28))
[<c01edd8a>] neigh_resolve_output [kernel] 0x19a (0xc4bc8c68))
[<c0202d80>] ip_finish_output2 [kernel] 0x0 (0xc4bc8c98))
[<c0224b55>] inet_sendmsg [kernel] 0x35 (0xc4bc8cdc))
[<c01e204c>] sock_sendmsg [kernel] 0x6c (0xc4bc8cf0))
[<e09ce0fd>] osi_NetSend [libafs-2.4.20-13.7-i686.mp] 0xdd (0xc4bc8d54))
[<c0231d32>] packet_rcv_spkt [kernel] 0x1b2 (0xc4bc8d74))
[<c01e96fe>] dev_queue_xmit [kernel] 0x14e (0xc4bc8d88))
[<c02025b9>] ip_build_xmit [kernel] 0x2f9 (0xc4bc8db8))
[<e0982ccb>] vlan_dev_hwaccel_hard_start_xmit [8021q] 0x7b (0xc4bc8ddc))
[<c01e97a7>] dev_queue_xmit [kernel] 0x1f7 (0xc4bc8df0))
[<c021dad0>] udp_getfrag [kernel] 0x0 (0xc4bc8e14))
[<e09d1346>] rxi_SendPacket [libafs-2.4.20-13.7-i686.mp] 0x146 (0xc4bc8e34))
[<e08f7416>] ip_refrag [ip_conntrack] 0x26 (0xc4bc8e64))
[<e09c53fd>] rxi_SendList [libafs-2.4.20-13.7-i686.mp] 0x56d (0xc4bc8e84))
[<c0202d80>] ip_finish_output2 [kernel] 0x0 (0xc4bc8ea0))
[<e09c5697>] rxi_SendXmitList [libafs-2.4.20-13.7-i686.mp] 0x227 (0xc4bc8ec4))
[<e09c5f4a>] rxi_Start [libafs-2.4.20-13.7-i686.mp] 0x7ba (0xc4bc8f14))
[<c01f07c6>] nf_hook_slow [kernel] 0x106 (0xc4bc8f58))
[<e09d743d>] rxkad_PreparePacket [libafs-2.4.20-13.7-i686.mp] 0x11d (0xc4bc8f64))
[<e09d1f93>] rxi_PrepareSendPacket [libafs-2.4.20-13.7-i686.mp] 0x183 (0xc4bc8f94))
[<e0a1013c>] rx_freePktQ_lock [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc8fa8))
[<e0a1013c>] rx_freePktQ_lock [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc8fac))
[<e09cd20c>] rxi_FlushWrite [libafs-2.4.20-13.7-i686.mp] 0x3bc (0xc4bc8fc4))
[<c021dfcf>] udp_sendmsg [kernel] 0x3cf (0xc4bc8fec))
[<e09ca129>] rxi_ReadProc [libafs-2.4.20-13.7-i686.mp] 0x99 (0xc4bc9034))
[<e09c6602>] rxi_ScheduleKeepAliveEvent [libafs-2.4.20-13.7-i686.mp] 0x52 (0xc4bc9044))
[<e09c64b0>] rxi_KeepAliveEvent [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc904c))
[<e09bdafc>] rx_NewCall [libafs-2.4.20-13.7-i686.mp] 0x63c (0xc4bc9074))
[<e09caa7d>] rx_ReadProc32 [libafs-2.4.20-13.7-i686.mp] 0xdd (0xc4bc90a4))
[<e09d22e5>] xdrrx_getint32 [libafs-2.4.20-13.7-i686.mp] 0x15 (0xc4bc90d4))
[<e09d771b>] afs_xdr_u_int [libafs-2.4.20-13.7-i686.mp] 0x1b (0xc4bc90f4))
[<e09b7424>] xdr_AFSFetchStatus [libafs-2.4.20-13.7-i686.mp] 0x14 (0xc4bc9114))
[<e09b8b16>] RXAFS_FetchStatus [libafs-2.4.20-13.7-i686.mp] 0x66 (0xc4bc9134))
[<e098e815>] afs_Conn [libafs-2.4.20-13.7-i686.mp] 0x185 (0xc4bc9154))
[<e0a06900>] xdrrx_ops [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc915c))
[<e0a06c00>] afs_global_lock [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc9178))
[<e09a50ef>] afs_FetchStatus [libafs-2.4.20-13.7-i686.mp] 0xef (0xc4bc9184))
[<e098df32>] afs_IsPrimaryCellNum [libafs-2.4.20-13.7-i686.mp] 0x22 (0xc4bc91b4))
[<e098df32>] afs_IsPrimaryCellNum [libafs-2.4.20-13.7-i686.mp] 0x22 (0xc4bc91c4))
[<e09a5a9b>] afs_FindVCache [libafs-2.4.20-13.7-i686.mp] 0x35b (0xc4bc91d4))
[<e09b5561>] afs_GetVolume [libafs-2.4.20-13.7-i686.mp] 0x21 (0xc4bc91e4))
[<e09a3a26>] afs_GetVCache [libafs-2.4.20-13.7-i686.mp] 0x3c6 (0xc4bc9214))
[<e09957a8>] afs_dir_GetBlob [libafs-2.4.20-13.7-i686.mp] 0x18 (0xc4bc9244))
[<e09c62f0>] rxi_Send [libafs-2.4.20-13.7-i686.mp] 0xb0 (0xc4bc9250))
[<e099586c>] FindItem [libafs-2.4.20-13.7-i686.mp] 0x5c (0xc4bc9264))
[<e099557d>] afs_dir_LookupOffset [libafs-2.4.20-13.7-i686.mp] 0x6d (0xc4bc9294))
[<e09ae33d>] afs_lookup [libafs-2.4.20-13.7-i686.mp] 0xabd (0xc4bc92b4))
[<e099b030>] afs_InitReq [libafs-2.4.20-13.7-i686.mp] 0x30 (0xc4bc9364))
[<e09dd175>] afs_linux_dentry_revalidate [libafs-2.4.20-13.7-i686.mp] 0x135 (0xc4bc9384))
[<e0a06c00>] afs_global_lock [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc93c4))
[<e09de348>] afs_linux_permission [libafs-2.4.20-13.7-i686.mp] 0xd8 (0xc4bc93c8))
[<c0150bcd>] cached_lookup [kernel] 0x2d (0xc4bc93f4))
[<c01515cd>] link_path_walk [kernel] 0x79d (0xc4bc9404))
[<e0999fdd>] osi_dnlc_enter [libafs-2.4.20-13.7-i686.mp] 0x1dd (0xc4bc9480))
[<e0a0a220>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x1f80 (0xc4bc9484))
[<e0a0a530>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x2290 (0xc4bc9488))
[<e0999f9e>] osi_dnlc_enter [libafs-2.4.20-13.7-i686.mp] 0x19e (0xc4bc9490))
[<e0a0a220>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x1f80 (0xc4bc94a4))
[<e0a0a220>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x1f80 (0xc4bc94a8))
[<e09aea76>] afs_lookup [libafs-2.4.20-13.7-i686.mp] 0x11f6 (0xc4bc94b0))
[<c0132d12>] read_cache_page [kernel] 0x42 (0xc4bc9514))
[<c0132d85>] read_cache_page [kernel] 0xb5 (0xc4bc9520))
[<c0154562>] page_getlink [kernel] 0x22 (0xc4bc9544))
[<e09de750>] afs_symlink_filler [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc9550))
[<e09dd20e>] afs_linux_dentry_revalidate [libafs-2.4.20-13.7-i686.mp] 0x1ce (0xc4bc9570))
[<c01547b8>] page_follow_link [kernel] 0x138 (0xc4bc9584))
[<e0a06c00>] afs_global_lock [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc95b0))
[<e09de348>] afs_linux_permission [libafs-2.4.20-13.7-i686.mp] 0xd8 (0xc4bc95b4))
[<e09dd620>] afs_linux_lookup [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc95d4))
[<c015174e>] link_path_walk [kernel] 0x91e (0xc4bc95f0))
[<e0999fdd>] osi_dnlc_enter [libafs-2.4.20-13.7-i686.mp] 0x1dd (0xc4bc966c))
[<e0a0a530>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x2290 (0xc4bc9670))
[<e0a0a648>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x23a8 (0xc4bc9674))
[<e0999f9e>] osi_dnlc_enter [libafs-2.4.20-13.7-i686.mp] 0x19e (0xc4bc967c))
[<e0a0a530>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x2290 (0xc4bc9690))
[<e0a0a530>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x2290 (0xc4bc9694))
[<e09aea76>] afs_lookup [libafs-2.4.20-13.7-i686.mp] 0x11f6 (0xc4bc969c))
[<c0132d12>] read_cache_page [kernel] 0x42 (0xc4bc9700))
[<c0132d85>] read_cache_page [kernel] 0xb5 (0xc4bc970c))
[<c0154562>] page_getlink [kernel] 0x22 (0xc4bc9730))
[<e09de750>] afs_symlink_filler [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc973c))
[<e09dd20e>] afs_linux_dentry_revalidate [libafs-2.4.20-13.7-i686.mp] 0x1ce (0xc4bc975c))
[<c01547b8>] page_follow_link [kernel] 0x138 (0xc4bc9770))
[<e0a06c00>] afs_global_lock [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc979c))
[<e09de348>] afs_linux_permission [libafs-2.4.20-13.7-i686.mp] 0xd8 (0xc4bc97a0))
[<e09dd620>] afs_linux_lookup [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc97c0))
[<c015174e>] link_path_walk [kernel] 0x91e (0xc4bc97dc))
[<e0999fdd>] osi_dnlc_enter [libafs-2.4.20-13.7-i686.mp] 0x1dd (0xc4bc9858))
[<e0a0a648>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x23a8 (0xc4bc985c))
[<e0a0b9c0>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x3720 (0xc4bc9860))
[<e0999f9e>] osi_dnlc_enter [libafs-2.4.20-13.7-i686.mp] 0x19e (0xc4bc9868))
[<e0a0a648>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x23a8 (0xc4bc987c))
[<e0a0a648>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x23a8 (0xc4bc9880))
[<e09aea76>] afs_lookup [libafs-2.4.20-13.7-i686.mp] 0x11f6 (0xc4bc9888))
[<c0132d12>] read_cache_page [kernel] 0x42 (0xc4bc98ec))
[<c0132d85>] read_cache_page [kernel] 0xb5 (0xc4bc98f8))
[<c015a21c>] dput [kernel] 0x1c (0xc4bc9908))
[<c0154562>] page_getlink [kernel] 0x22 (0xc4bc991c))
[<e09de750>] afs_symlink_filler [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc9928))
[<e09dd20e>] afs_linux_dentry_revalidate [libafs-2.4.20-13.7-i686.mp] 0x1ce (0xc4bc9948))
[<c01547b8>] page_follow_link [kernel] 0x138 (0xc4bc995c))
[<e0a06c00>] afs_global_lock [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc9988))
[<e09de348>] afs_linux_permission [libafs-2.4.20-13.7-i686.mp] 0xd8 (0xc4bc998c))
[<c015174e>] link_path_walk [kernel] 0x91e (0xc4bc99c8))
[<e0999fdd>] osi_dnlc_enter [libafs-2.4.20-13.7-i686.mp] 0x1dd (0xc4bc9a44))
[<e0a0a728>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x2488 (0xc4bc9a48))
[<e0a0aa70>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x27d0 (0xc4bc9a4c))
[<e0999f9e>] osi_dnlc_enter [libafs-2.4.20-13.7-i686.mp] 0x19e (0xc4bc9a54))
[<e0a0a728>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x2488 (0xc4bc9a68))
[<e0a0a728>] nameCache [libafs-2.4.20-13.7-i686.mp] 0x2488 (0xc4bc9a6c))
[<e09aea76>] afs_lookup [libafs-2.4.20-13.7-i686.mp] 0x11f6 (0xc4bc9a74))
[<c0132d12>] read_cache_page [kernel] 0x42 (0xc4bc9ad8))
[<c0132d85>] read_cache_page [kernel] 0xb5 (0xc4bc9ae4))
[<c0154562>] page_getlink [kernel] 0x22 (0xc4bc9b08))
[<e09de750>] afs_symlink_filler [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc9b14))
[<e09dd20e>] afs_linux_dentry_revalidate [libafs-2.4.20-13.7-i686.mp] 0x1ce (0xc4bc9b34))
[<c01547b8>] page_follow_link [kernel] 0x138 (0xc4bc9b48))
[<e0a06c00>] afs_global_lock [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc9b74))
[<e09de348>] afs_linux_permission [libafs-2.4.20-13.7-i686.mp] 0xd8 (0xc4bc9b78))
[<e09dd620>] afs_linux_lookup [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc9b98))
[<c015174e>] link_path_walk [kernel] 0x91e (0xc4bc9bb4))
[<c0200220>] ip_forward_finish [kernel] 0x0 (0xc4bc9bd8))
[<c0202d65>] ip_finish_output [kernel] 0x125 (0xc4bc9bdc))
[<e09957a8>] afs_dir_GetBlob [libafs-2.4.20-13.7-i686.mp] 0x18 (0xc4bc9c04))
[<c01f045e>] nf_iterate [kernel] 0x2e (0xc4bc9c10))
[<e09958fb>] FindItem [libafs-2.4.20-13.7-i686.mp] 0xeb (0xc4bc9c24))
[<e09b5561>] afs_GetVolume [libafs-2.4.20-13.7-i686.mp] 0x21 (0xc4bc9c34))
[<e09b5561>] afs_GetVolume [libafs-2.4.20-13.7-i686.mp] 0x21 (0xc4bc9c50))
[<e09a6a70>] afs_CopyOutAttrs [libafs-2.4.20-13.7-i686.mp] 0x1f0 (0xc4bc9c64))
[<e09a6a70>] afs_CopyOutAttrs [libafs-2.4.20-13.7-i686.mp] 0x1f0 (0xc4bc9c80))
[<e09db4e1>] vcache2inode [libafs-2.4.20-13.7-i686.mp] 0x21 (0xc4bc9ca0))
[<c0132d12>] read_cache_page [kernel] 0x42 (0xc4bc9cc4))
[<c0132d85>] read_cache_page [kernel] 0xb5 (0xc4bc9cd0))
[<c015a21c>] dput [kernel] 0x1c (0xc4bc9ce0))
[<c0154562>] page_getlink [kernel] 0x22 (0xc4bc9cf4))
[<e09de750>] afs_symlink_filler [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc9d00))
[<e09dd20e>] afs_linux_dentry_revalidate [libafs-2.4.20-13.7-i686.mp] 0x1ce (0xc4bc9d20))
[<c01547b8>] page_follow_link [kernel] 0x138 (0xc4bc9d34))
[<e0a06c00>] afs_global_lock [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc9d60))
[<e09de348>] afs_linux_permission [libafs-2.4.20-13.7-i686.mp] 0xd8 (0xc4bc9d64))
[<e09dd620>] afs_linux_lookup [libafs-2.4.20-13.7-i686.mp] 0x0 (0xc4bc9d84))
[<c015174e>] link_path_walk [kernel] 0x91e (0xc4bc9da0))
[<e098df32>] afs_IsPrimaryCellNum [libafs-2.4.20-13.7-i686.mp] 0x22 (0xc4bc9dc4))
[<c01e57fc>] kfree_skbmem [kernel] 0xc (0xc4bc9ddc))
[<c01e597e>] __kfree_skb [kernel] 0x11e (0xc4bc9dec))
[<c0231d32>] packet_rcv_spkt [kernel] 0x1b2 (0xc4bc9df8))
[<e09b5561>] afs_GetVolume [libafs-2.4.20-13.7-i686.mp] 0x21 (0xc4bc9e08))
[<e09d91ae>] osi_UFSOpen [libafs-2.4.20-13.7-i686.mp] 0x16e (0xc4bc9e14))
[<e09a6a70>] afs_CopyOutAttrs [libafs-2.4.20-13.7-i686.mp] 0x1f0 (0xc4bc9e38))
[<e09db4e1>] vcache2inode [libafs-2.4.20-13.7-i686.mp] 0x21 (0xc4bc9e58))
[<c01e561f>] alloc_skb [kernel] 0xef (0xc4bc9e6c))
[<e095c739>] tg3_rx [tg3] 0x289 (0xc4bc9ec8))
[<e09dcef1>] afs_linux_revalidate [libafs-2.4.20-13.7-i686.mp] 0x171 (0xc4bc9ed8))
[<c015a21c>] dput [kernel] 0x1c (0xc4bc9ee4))
[<c01508ee>] getname [kernel] 0x5e (0xc4bc9f0c))
[<c0151a8b>] path_lookup [kernel] 0x1b (0xc4bc9f20))
[<c0151d84>] __user_walk [kernel] 0x24 (0xc4bc9f30))
[<c014dbc7>] vfs_stat [kernel] 0x17 (0xc4bc9f44))
[<c014e171>] sys_stat64 [kernel] 0x11 (0xc4bc9f70))
[<c0108be3>] system_call [kernel] 0x33 (0xc4bc9fc0))

--PEIAKu/WMn1b1Hv9--