[OpenAFS-devel] Fileserver deadlocks on Ubuntu Hardy during peak load
Linus Widströmer
linus.widstromer@it.su.se
Mon, 06 Sep 2010 13:30:43 +0200
Hi everyone,
we have started to experience fileserver hangs on our Ubuntu Hardy fileservers running 1.4.6.dfsg1-2+ubuntu0.1
The problem seems to be related to high users load, the prime suspects are the IMAP servers which read maildirs from AFS. We have seen the problem a couple of times, the effect is that one of the fileservers all by a sudden build up a queue of ~200 calls waiting for a thread and stops responding, with client hangs as a result.
We have managed to catch a core dump of a stalled fileserver and it seems as all active threads are waiting for a lock (see bt below). After rebooting one of the clients the backlog quickly goes down and the server returns to normal operation within a minute or so.
In the backtrace we see five threads waiting for a lock in h_Lock_r - two threads trying to lock host1 and three of them trying to get a lock for host2. Both hosts are part of the IMAP system.
client1
=======
#2 0x08074a2b in Afs_Lock_Obtain (lock=0x8134694, how=2) at ../lwp/lock.c:94
#3 0x08061e23 in h_Lock_r (host=0x8134640) at ../viced/host.c:534
#2 0x08074a2b in Afs_Lock_Obtain (lock=0x8134694, how=2) at ../lwp/lock.c:94
#3 0x08061e23 in h_Lock_r (host=0x8134640) at ../viced/host.c:534
lock->wait_states = 2
lock->excl_locked = 2
lock->readers_reading = 0
lock->num_waiting = 2
client2
=======
#2 0x08074a2b in Afs_Lock_Obtain (lock=0x8134194, how=2) at ../lwp/lock.c:94
#3 0x08061e23 in h_Lock_r (host=0x8134140) at ../viced/host.c:534
#2 0x08074a2b in Afs_Lock_Obtain (lock=0x8134194, how=2) at ../lwp/lock.c:94
#3 0x08061e23 in h_Lock_r (host=0x8134140) at ../viced/host.c:534
#2 0x08074a2b in Afs_Lock_Obtain (lock=0x8134194, how=2) at ../lwp/lock.c:94
#3 0x08061e23 in h_Lock_r (host=0x8134140) at ../viced/host.c:534
lock->wait_states = 2
lock->excl_locked = 2
lock->readers_reading = 0
lock->num_waiting = 3
Has anyone experienced similar problems with this version of openafs? Is it a known bug, has it perhaps been fixed in newer releases?
Any hints on how to track this down further; what to look for in the logs/corefiles, how to gather even more debugging next time the problem occurs etc are very much welcome. And of course - and ideas on how to get it fixed even more so.
Feel free to contact me if more information from the corefile is needed.
Best regards,
Linus Widströmer
Stockholm University
Backtrace of all threads
==============
Thread 20 (process 5060):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb76edd76 in nanosleep () from /lib/tls/i686/cmov/libc.so.6
#2 0xb76edb87 in sleep () from /lib/tls/i686/cmov/libc.so.6
#3 0x0804de2c in main (argc=856, argv=0x29389510) at ../viced/viced.c:2046
Thread 19 (process 5155):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c4dd2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x0804c69a in FsyncCheckLWP () at ../viced/viced.c:504
#3 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#4 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 18 (process 5154):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb76edd76 in nanosleep () from /lib/tls/i686/cmov/libc.so.6
#2 0xb76edb87 in sleep () from /lib/tls/i686/cmov/libc.so.6
#3 0x0804c792 in HostCheckLWP () at ../viced/viced.c:468
#4 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#5 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 17 (process 5153):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb76edd76 in nanosleep () from /lib/tls/i686/cmov/libc.so.6
#2 0xb76edb87 in sleep () from /lib/tls/i686/cmov/libc.so.6
#3 0x0804cd6d in FiveMinuteCheckLWP () at ../viced/viced.c:423
#4 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#5 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 16 (process 5144):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb76edd76 in nanosleep () from /lib/tls/i686/cmov/libc.so.6
#2 0xb76edb87 in sleep () from /lib/tls/i686/cmov/libc.so.6
#3 0x08088d02 in ih_sync_thread () at ../vol/ihandle.c:911
#4 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#5 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 15 (process 5131):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb7727941 in select () from /lib/tls/i686/cmov/libc.so.6
#2 0x08084b75 in FSYNC_sync () at ../vol/fssync.c:395
#3 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#4 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 14 (process 5130):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x08074a2b in Afs_Lock_Obtain (lock=0x8134694, how=2) at ../lwp/lock.c:94
#3 0x08061e23 in h_Lock_r (host=0x8134640) at ../viced/host.c:534
#4 0x08065256 in h_Lookup_r (haddr=3126717826, hport=22811, heldp=0xb09a5de4) at ../viced/host.c:840
#5 0x080654ed in h_GetHost_r (tcon=0xad29fe00) at ../viced/host.c:1420
#6 0x080672ae in h_FindClient_r (tcon=0xad29fe00) at ../viced/host.c:2032
#7 0x0804fe82 in CallPreamble (acall=<value optimized out>, activecall=1, tconn=0xb09a61c8, ahostp=0xb09a61c4) at ../viced/afsfileprocs.c:321
#8 0x0805e6e1 in SRXAFS_FetchStatus (acall=0xad86c968, Fid=0xb09a6300, OutStatus=0xb09a6210, CallBack=0xb09a6318, Sync=0xb09a62b8) at ../viced/afsfileprocs.c:2962
#9 0x08094369 in RXAFS_ExecuteRequest (z_call=0xad86c968) at ../fsint/afsint.ss.c:159
#10 0x080b97cc in rxi_ServerProc (threadID=11, newcall=0x81fc880, socketp=0xb09a63b8) at ./../rx/rx.c:1418
#11 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
#12 0x08096df9 in server_entry (argp=0x8097800) at ../rx/rx_pthread.c:100
#13 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#14 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 13 (process 5129):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x08074a2b in Afs_Lock_Obtain (lock=0x8134694, how=2) at ../lwp/lock.c:94
#3 0x08061e23 in h_Lock_r (host=0x8134640) at ../viced/host.c:534
#4 0x08065256 in h_Lookup_r (haddr=3126717826, hport=22811, heldp=0xb11a6de4) at ../viced/host.c:840
#5 0x080654ed in h_GetHost_r (tcon=0xad30b0b8) at ../viced/host.c:1420
#6 0x080672ae in h_FindClient_r (tcon=0xad30b0b8) at ../viced/host.c:2032
#7 0x0804fe82 in CallPreamble (acall=<value optimized out>, activecall=1, tconn=0xb11a71c8, ahostp=0xb11a71c4) at ../viced/afsfileprocs.c:321
#8 0x0805e6e1 in SRXAFS_FetchStatus (acall=0x8230110, Fid=0xb11a7300, OutStatus=0xb11a7210, CallBack=0xb11a7318, Sync=0xb11a72b8) at ../viced/afsfileprocs.c:2962
#9 0x08094369 in RXAFS_ExecuteRequest (z_call=0x8230110) at ../fsint/afsint.ss.c:159
#10 0x080b97cc in rxi_ServerProc (threadID=5, newcall=0xad862f00, socketp=0xb11a73b8) at ./../rx/rx.c:1418
#11 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
#12 0x08096df9 in server_entry (argp=0x8097800) at ../rx/rx_pthread.c:100
#13 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#14 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 12 (process 5128):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c7db8 in recvmsg () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x080bed67 in rxi_ReadPacket (socket=5, p=0x8186c00, host=0xb19a8384, port=0xb19a838a) at ./../rx/rx_packet.c:1411
#3 0x08097296 in rxi_ListenerProc (sock=5, tnop=0xb19a83b4, newcallp=0xb19a83b0) at ../rx/rx_pthread.c:230
#4 0x080978ad in rx_ServerProc () at ../rx/rx_pthread.c:305
#5 0x08096df9 in server_entry (argp=0x8097800) at ../rx/rx_pthread.c:100
#6 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#7 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 11 (process 5127):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x08074a2b in Afs_Lock_Obtain (lock=0x8134194, how=2) at ../lwp/lock.c:94
#3 0x08061e23 in h_Lock_r (host=0x8134140) at ../viced/host.c:534
#4 0x08065256 in h_Lookup_r (haddr=3143495042, hport=22811, heldp=0xb21a8de4) at ../viced/host.c:840
#5 0x080654ed in h_GetHost_r (tcon=0x824bbe0) at ../viced/host.c:1420
#6 0x080672ae in h_FindClient_r (tcon=0x824bbe0) at ../viced/host.c:2032
#7 0x0804fe82 in CallPreamble (acall=<value optimized out>, activecall=1, tconn=0xb21a91c8, ahostp=0xb21a91c4) at ../viced/afsfileprocs.c:321
#8 0x0805e6e1 in SRXAFS_FetchStatus (acall=0xad44a448, Fid=0xb21a9300, OutStatus=0xb21a9210, CallBack=0xb21a9318, Sync=0xb21a92b8) at ../viced/afsfileprocs.c:2962
#9 0x08094369 in RXAFS_ExecuteRequest (z_call=0xad44a448) at ../fsint/afsint.ss.c:159
#10 0x080b97cc in rxi_ServerProc (threadID=7, newcall=0x832ebf8, socketp=0xb21a93b8) at ./../rx/rx.c:1418
#11 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
#12 0x08096df9 in server_entry (argp=0x8097800) at ../rx/rx_pthread.c:100
#13 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#14 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 10 (process 5126):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x080c2180 in rxi_ReadProc (call=0xad814d60, buf=0xb29a9adb "²`M\201\034\233\232²\030\202\203X\233\232²fâ\b\b`M\201", nbytes=1) at ./../rx/rx_rdwr.c:234
#3 0x080b9667 in rx_EndCall (call=0xad814d60, rc=0) at ./../rx/rx.c:1898
#4 0x0808e266 in RXAFSCB_CallBack (z_conn=0xad838218, Fids_Array=0xb29a9de4, CallBacks_Array=0xb29a9ddc) at ../fsint/afscbint.cs.c:32
#5 0x0806c043 in XCallBackBulk_r (ahost=0x8139840, fids=0xb29a9e30, nfids=0) at ../viced/callback.c:601
#6 0x0806c172 in BreakDelayedCallBacks_r (host=0x8139840) at ../viced/callback.c:1151
#7 0x0805018d in CallPreamble (acall=<value optimized out>, activecall=0, tconn=0xb29aa168, ahostp=0xb29aa164) at ../viced/afsfileprocs.c:381
#8 0x0805091e in SRXAFS_GetTime (acall=0x8233440, Seconds=0xb29aa1c8, USeconds=0xb29aa1c4) at ../viced/afsfileprocs.c:6845
#9 0x08090623 in _RXAFS_GetTime (z_call=0x8233440, z_xdrs=0x5561) at ../fsint/afsint.ss.c:1109
#10 0x0809380a in RXAFS_ExecuteRequest (z_call=0x8233440) at ../fsint/afsint.ss.c:1941
#11 0x080b97cc in rxi_ServerProc (threadID=13, newcall=0xad32ac30, socketp=0xb29aa3b8) at ./../rx/rx.c:1418
#12 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
#13 0x08096df9 in server_entry (argp=0x8097800) at ../rx/rx_pthread.c:100
#14 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#15 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 9 (process 5125):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x080c2180 in rxi_ReadProc (call=0xad648dc8, buf=0xb31aab14 "D«\032³8«\032³ 6\f\b\234«\032³4«\032³H«\032³Å?\f\bÈ\215dD«\032³h«\032³Îa\t\b\234«\032³8¬\032³h«\032³z6\f\b\234«\032³d«\032³¥Õ\016", nbytes=4) at ./../rx/rx_rdwr.c:234
#3 0x080c25f6 in rx_ReadProc32 (call=0xad648dc8, value=0xb31aab14) at ./../rx/rx_rdwr.c:388
#4 0x080c3feb in xdrrx_getint32 (axdrs=0xb31aab9c, lp=0xb31aab34) at ./../rx/xdr_rx.c:211
#5 0x080c36a0 in xdr_int (xdrs=0x80, ip=0xb31aac38) at ./../rx/xdr.c:99
#6 0x080961ce in xdr_interfaceAddr (xdrs=0xb31aab9c, objp=0xb31aac38) at ../fsint/afsint.xdr.c:363
#7 0x0808c486 in RXAFSCB_TellMeAboutYourself (z_conn=0x83312e0, addr=0xb31aac38, capabilities=0xb31aaddc) at ../fsint/afscbint.cs.c:826
#8 0x080658d2 in h_GetHost_r (tcon=0x8260d70) at ../viced/host.c:1447
#9 0x080672ae in h_FindClient_r (tcon=0x8260d70) at ../viced/host.c:2032
#10 0x0804fe82 in CallPreamble (acall=<value optimized out>, activecall=1, tconn=0xb31ab1c8, ahostp=0xb31ab1c4) at ../viced/afsfileprocs.c:321
#11 0x0805e6e1 in SRXAFS_FetchStatus (acall=0x81f7cb8, Fid=0xb31ab300, OutStatus=0xb31ab210, CallBack=0xb31ab318, Sync=0xb31ab2b8) at ../viced/afsfileprocs.c:2962
#12 0x08094369 in RXAFS_ExecuteRequest (z_call=0x81f7cb8) at ../fsint/afsint.ss.c:159
#13 0x080b97cc in rxi_ServerProc (threadID=10, newcall=0xad85e410, socketp=0xb31ab3b8) at ./../rx/rx.c:1418
#14 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
#15 0x08096df9 in server_entry (argp=0x8097800) at ../rx/rx_pthread.c:100
#16 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#17 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 8 (process 5124):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x08074a2b in Afs_Lock_Obtain (lock=0x8134194, how=2) at ../lwp/lock.c:94
#3 0x08061e23 in h_Lock_r (host=0x8134140) at ../viced/host.c:534
#4 0x08065256 in h_Lookup_r (haddr=3143495042, hport=22811, heldp=0xb39abde4) at ../viced/host.c:840
#5 0x080654ed in h_GetHost_r (tcon=0x823a128) at ../viced/host.c:1420
#6 0x080672ae in h_FindClient_r (tcon=0x823a128) at ../viced/host.c:2032
#7 0x0804fe82 in CallPreamble (acall=<value optimized out>, activecall=1, tconn=0xb39ac1c8, ahostp=0xb39ac1c4) at ../viced/afsfileprocs.c:321
#8 0x0805e6e1 in SRXAFS_FetchStatus (acall=0xad8b1918, Fid=0xb39ac300, OutStatus=0xb39ac210, CallBack=0xb39ac318, Sync=0xb39ac2b8) at ../viced/afsfileprocs.c:2962
#9 0x08094369 in RXAFS_ExecuteRequest (z_call=0xad8b1918) at ../fsint/afsint.ss.c:159
#10 0x080b97cc in rxi_ServerProc (threadID=3, newcall=0x820d0a8, socketp=0xb39ac3b8) at ./../rx/rx.c:1418
#11 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
#12 0x08096df9 in server_entry (argp=0x8097800) at ../rx/rx_pthread.c:100
#13 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#14 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 7 (process 5123):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x080b66e2 in rx_GetCall (tno=8, cur_service=0x811c6b8, socketp=0xb41ad3b8) at ./../rx/rx.c:1613
#3 0x080b982f in rxi_ServerProc (threadID=8, newcall=0x825b448, socketp=0xb41ad3b8) at ./../rx/rx.c:1376
#4 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
#5 0x08096df9 in server_entry (argp=0x8097800) at ../rx/rx_pthread.c:100
#6 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#7 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 6 (process 5122):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x080c2180 in rxi_ReadProc (call=0x8219080, buf=0xb49add6b "", nbytes=1) at ./../rx/rx_rdwr.c:234
#3 0x080b9667 in rx_EndCall (call=0x8219080, rc=0) at ./../rx/rx.c:1898
#4 0x0808db56 in RXAFSCB_InitCallBackState3 (z_conn=0x83e7078, serverUuid=0x80ff494) at ../fsint/afscbint.cs.c:460
#5 0x0806c2c8 in BreakDelayedCallBacks_r (host=0x8133740) at ../viced/callback.c:1097
#6 0x0805018d in CallPreamble (acall=<value optimized out>, activecall=1, tconn=0xb49ae154, ahostp=0xb49ae150) at ../viced/afsfileprocs.c:381
#7 0x08055a53 in common_GiveUpCallBacks (acall=0xad6fa6f8, FidArray=0xb49ae1b4, CallBackArray=0xb49ae1ac) at ../viced/afsfileprocs.c:6106
#8 0x0808f73b in _RXAFS_GiveUpCallBacks (z_call=0xad6fa6f8, z_xdrs=0xb49ae2e8) at ../fsint/afsint.ss.c:838
#9 0x080931f6 in RXAFS_ExecuteRequest (z_call=0xad6fa6f8) at ../fsint/afsint.ss.c:1923
#10 0x080b97cc in rxi_ServerProc (threadID=6, newcall=0xad6fa6f8, socketp=0xb49ae3b8) at ./../rx/rx.c:1418
#11 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
#12 0x08096df9 in server_entry (argp=0x8097800) at ../rx/rx_pthread.c:100
#13 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#14 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 5 (process 5121):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x080b66e2 in rx_GetCall (tno=12, cur_service=0x811c6b8, socketp=0xb51af3b8) at ./../rx/rx.c:1613
#3 0x080b982f in rxi_ServerProc (threadID=12, newcall=0x825b448, socketp=0xb51af3b8) at ./../rx/rx.c:1376
#4 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
#5 0x08096df9 in server_entry (argp=0x8097800) at ../rx/rx_pthread.c:100
#6 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#7 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 4 (process 5120):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x080c2180 in rxi_ReadProc (call=0x82a81c0, buf=0xb59afb14 "Dû\232µ8û\232µ 6\f\b\234û\232µ4û\232µHû\232µÅ?\f\bÀ\201*\bDû\232µhû\232µÎa\t\b\234û\232µ8ü\232µhû\232µz6\f\b\234û\232µdû\232µ`ù\016", nbytes=4) at ./../rx/rx_rdwr.c:234
#3 0x080c25f6 in rx_ReadProc32 (call=0x82a81c0, value=0xb59afb14) at ./../rx/rx_rdwr.c:388
#4 0x080c3feb in xdrrx_getint32 (axdrs=0xb59afb9c, lp=0xb59afb34) at ./../rx/xdr_rx.c:211
#5 0x080c36a0 in xdr_int (xdrs=0x80, ip=0xb59afc38) at ./../rx/xdr.c:99
#6 0x080961ce in xdr_interfaceAddr (xdrs=0xb59afb9c, objp=0xb59afc38) at ../fsint/afsint.xdr.c:363
#7 0x0808c486 in RXAFSCB_TellMeAboutYourself (z_conn=0xad6f89c8, addr=0xb59afc38, capabilities=0xb59afddc) at ../fsint/afscbint.cs.c:826
#8 0x080658d2 in h_GetHost_r (tcon=0xad281a88) at ../viced/host.c:1447
#9 0x080672ae in h_FindClient_r (tcon=0xad281a88) at ../viced/host.c:2032
#10 0x0804fe82 in CallPreamble (acall=<value optimized out>, activecall=1, tconn=0xb59b01c8, ahostp=0xb59b01c4) at ../viced/afsfileprocs.c:321
#11 0x0805e6e1 in SRXAFS_FetchStatus (acall=0x82147a8, Fid=0xb59b0300, OutStatus=0xb59b0210, CallBack=0xb59b0318, Sync=0xb59b02b8) at ../viced/afsfileprocs.c:2962
#12 0x08094369 in RXAFS_ExecuteRequest (z_call=0x82147a8) at ../fsint/afsint.ss.c:159
#13 0x080b97cc in rxi_ServerProc (threadID=4, newcall=0xad8065e8, socketp=0xb59b03b8) at ./../rx/rx.c:1418
#14 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
#15 0x08096df9 in server_entry (argp=0x8097800) at ../rx/rx_pthread.c:100
#16 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#17 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 3 (process 5081):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c4dd2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x0809778f in event_handler (argp=0x0) at ../rx/rx_pthread.c:173
#3 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#4 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 2 (process 5080):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0x08074a2b in Afs_Lock_Obtain (lock=0x8134194, how=2) at ../lwp/lock.c:94
#3 0x08061e23 in h_Lock_r (host=0x8134140) at ../viced/host.c:534
#4 0x08065256 in h_Lookup_r (haddr=3143495042, hport=22811, heldp=0xb6e26df4) at ../viced/host.c:840
#5 0x080654ed in h_GetHost_r (tcon=0x8374160) at ../viced/host.c:1420
#6 0x080672ae in h_FindClient_r (tcon=0x8374160) at ../viced/host.c:2032
#7 0x0804fe82 in CallPreamble (acall=<value optimized out>, activecall=1, tconn=0xb6e271d8, ahostp=0xb6e271d4) at ../viced/afsfileprocs.c:321
#8 0x0805e6e1 in SRXAFS_FetchStatus (acall=0xad424f70, Fid=0xb6e27310, OutStatus=0xb6e27220, CallBack=0xb6e27328, Sync=0xb6e272c8) at ../viced/afsfileprocs.c:2962
#9 0x08094369 in RXAFS_ExecuteRequest (z_call=0xad424f70) at ../fsint/afsint.ss.c:159
#10 0x080b97cc in rxi_ServerProc (threadID=9, newcall=0x8488710, socketp=0xb6e273c4) at ./../rx/rx.c:1418
#11 0x08097376 in rx_ListenerProc (argp=0x5) at ../rx/rx_pthread.c:261
#12 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#13 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
Thread 1 (process 5079):
#0 0xb77db410 in __kernel_vsyscall ()
#1 0xb77c8b1a in do_sigwait () from /lib/tls/i686/cmov/libpthread.so.0
#2 0xb77c8bbf in sigwait () from /lib/tls/i686/cmov/libpthread.so.0
#3 0x08079610 in softsig_thread (arg=0x0) at ../util/softsig.c:93
#4 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#5 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6