[OpenAFS-devel] Fileserver deadlocks on Ubuntu Hardy during peak load
Derrick Brashear
shadow@gmail.com
Mon, 6 Sep 2010 09:21:58 -0400
funny how you define "active". so who holds the lock? i bet it's those
threads which are in TellMeAboutYourself, waiting on network. (4, 9)
there is thread starvation avoidance in newer versions. and you need
more threads. i suggest the -p 128 argument to fileserver in addition
to whatever else you have, at least, and that's a minimum
configuration improvement.
2010/9/6 Linus Widstr=F6mer <linus.widstromer@it.su.se>:
>
> Hi everyone,
>
> we have started to experience fileserver hangs on our Ubuntu Hardy filese=
rvers running 1.4.6.dfsg1-2+ubuntu0.1
>
> The problem seems to be related to high users load, the prime suspects ar=
e the IMAP servers which read maildirs from AFS. We have seen the problem a=
couple of times, the effect is that one of the fileservers all by a sudden=
build up a queue of ~200 calls waiting for a thread and stops responding, =
with client hangs as a result.
>
> We have managed to catch a core dump of a stalled fileserver and it seems=
as all active threads are waiting for a lock (see bt below). After rebooti=
ng one of the clients the backlog quickly goes down and the server returns =
to normal operation within a minute or so.
>
> In the backtrace we see five threads waiting for a lock in h_Lock_r - two=
threads trying to lock host1 and three of them trying to get a lock for ho=
st2. Both hosts are part of the IMAP system.
>
> client1
> =3D=3D=3D=3D=3D=3D=3D
> #2 =A00x08074a2b in Afs_Lock_Obtain (lock=3D0x8134694, how=3D2) at ../lwp=
/lock.c:94
> #3 =A00x08061e23 in h_Lock_r (host=3D0x8134640) at ../viced/host.c:534
>
> #2 =A00x08074a2b in Afs_Lock_Obtain (lock=3D0x8134694, how=3D2) at ../lwp=
/lock.c:94
> #3 =A00x08061e23 in h_Lock_r (host=3D0x8134640) at ../viced/host.c:534
>
> lock->wait_states =3D 2
> lock->excl_locked =3D 2
> lock->readers_reading =3D 0
> lock->num_waiting =3D 2
>
>
> client2
> =3D=3D=3D=3D=3D=3D=3D
> #2 =A00x08074a2b in Afs_Lock_Obtain (lock=3D0x8134194, how=3D2) at ../lwp=
/lock.c:94
> #3 =A00x08061e23 in h_Lock_r (host=3D0x8134140) at ../viced/host.c:534
>
> #2 =A00x08074a2b in Afs_Lock_Obtain (lock=3D0x8134194, how=3D2) at ../lwp=
/lock.c:94
> #3 =A00x08061e23 in h_Lock_r (host=3D0x8134140) at ../viced/host.c:534
>
> #2 =A00x08074a2b in Afs_Lock_Obtain (lock=3D0x8134194, how=3D2) at ../lwp=
/lock.c:94
> #3 =A00x08061e23 in h_Lock_r (host=3D0x8134140) at ../viced/host.c:534
>
> lock->wait_states =3D 2
> lock->excl_locked =3D 2
> lock->readers_reading =3D 0
> lock->num_waiting =3D 3
>
>
> Has anyone experienced similar problems with this version of openafs? Is =
it a known bug, has it perhaps been fixed in newer releases?
>
> Any hints on how to track this down further; what to look for in the logs=
/corefiles, how to gather even more debugging next time the problem occurs =
etc are very much welcome. And of course - =A0and ideas on how to get it fi=
xed even more so.
>
> Feel free to contact me if more information from the corefile is needed.
>
> Best regards,
> Linus Widstr=F6mer
> Stockholm University
>
>
>
> Backtrace of all threads
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> Thread 20 (process 5060):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb76edd76 in nanosleep () from /lib/tls/i686/cmov/libc.so.6
> #2 =A00xb76edb87 in sleep () from /lib/tls/i686/cmov/libc.so.6
> #3 =A00x0804de2c in main (argc=3D856, argv=3D0x29389510) at ../viced/vice=
d.c:2046
>
> Thread 19 (process 5155):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c4dd2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/=
i686/cmov/libpthread.so.0
> #2 =A00x0804c69a in FsyncCheckLWP () at ../viced/viced.c:504
> #3 =A00xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so=
.0
> #4 =A00xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 18 (process 5154):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb76edd76 in nanosleep () from /lib/tls/i686/cmov/libc.so.6
> #2 =A00xb76edb87 in sleep () from /lib/tls/i686/cmov/libc.so.6
> #3 =A00x0804c792 in HostCheckLWP () at ../viced/viced.c:468
> #4 =A00xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so=
.0
> #5 =A00xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 17 (process 5153):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb76edd76 in nanosleep () from /lib/tls/i686/cmov/libc.so.6
> #2 =A00xb76edb87 in sleep () from /lib/tls/i686/cmov/libc.so.6
> #3 =A00x0804cd6d in FiveMinuteCheckLWP () at ../viced/viced.c:423
> #4 =A00xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so=
.0
> #5 =A00xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 16 (process 5144):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb76edd76 in nanosleep () from /lib/tls/i686/cmov/libc.so.6
> #2 =A00xb76edb87 in sleep () from /lib/tls/i686/cmov/libc.so.6
> #3 =A00x08088d02 in ih_sync_thread () at ../vol/ihandle.c:911
> #4 =A00xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so=
.0
> #5 =A00xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 15 (process 5131):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb7727941 in select () from /lib/tls/i686/cmov/libc.so.6
> #2 =A00x08084b75 in FSYNC_sync () at ../vol/fssync.c:395
> #3 =A00xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so=
.0
> #4 =A00xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 14 (process 5130):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/=
cmov/libpthread.so.0
> #2 =A00x08074a2b in Afs_Lock_Obtain (lock=3D0x8134694, how=3D2) at ../lwp=
/lock.c:94
> #3 =A00x08061e23 in h_Lock_r (host=3D0x8134640) at ../viced/host.c:534
> #4 =A00x08065256 in h_Lookup_r (haddr=3D3126717826, hport=3D22811, heldp=
=3D0xb09a5de4) at ../viced/host.c:840
> #5 =A00x080654ed in h_GetHost_r (tcon=3D0xad29fe00) at ../viced/host.c:14=
20
> #6 =A00x080672ae in h_FindClient_r (tcon=3D0xad29fe00) at ../viced/host.c=
:2032
> #7 =A00x0804fe82 in CallPreamble (acall=3D<value optimized out>, activeca=
ll=3D1, tconn=3D0xb09a61c8, ahostp=3D0xb09a61c4) at ../viced/afsfileprocs.c=
:321
> #8 =A00x0805e6e1 in SRXAFS_FetchStatus (acall=3D0xad86c968, Fid=3D0xb09a6=
300, OutStatus=3D0xb09a6210, CallBack=3D0xb09a6318, Sync=3D0xb09a62b8) at .=
./viced/afsfileprocs.c:2962
> #9 =A00x08094369 in RXAFS_ExecuteRequest (z_call=3D0xad86c968) at ../fsin=
t/afsint.ss.c:159
> #10 0x080b97cc in rxi_ServerProc (threadID=3D11, newcall=3D0x81fc880, soc=
ketp=3D0xb09a63b8) at ./../rx/rx.c:1418
> #11 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
> #12 0x08096df9 in server_entry (argp=3D0x8097800) at ../rx/rx_pthread.c:1=
00
> #13 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
> #14 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 13 (process 5129):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/=
cmov/libpthread.so.0
> #2 =A00x08074a2b in Afs_Lock_Obtain (lock=3D0x8134694, how=3D2) at ../lwp=
/lock.c:94
> #3 =A00x08061e23 in h_Lock_r (host=3D0x8134640) at ../viced/host.c:534
> #4 =A00x08065256 in h_Lookup_r (haddr=3D3126717826, hport=3D22811, heldp=
=3D0xb11a6de4) at ../viced/host.c:840
> #5 =A00x080654ed in h_GetHost_r (tcon=3D0xad30b0b8) at ../viced/host.c:14=
20
> #6 =A00x080672ae in h_FindClient_r (tcon=3D0xad30b0b8) at ../viced/host.c=
:2032
> #7 =A00x0804fe82 in CallPreamble (acall=3D<value optimized out>, activeca=
ll=3D1, tconn=3D0xb11a71c8, ahostp=3D0xb11a71c4) at ../viced/afsfileprocs.c=
:321
> #8 =A00x0805e6e1 in SRXAFS_FetchStatus (acall=3D0x8230110, Fid=3D0xb11a73=
00, OutStatus=3D0xb11a7210, CallBack=3D0xb11a7318, Sync=3D0xb11a72b8) at ..=
/viced/afsfileprocs.c:2962
> #9 =A00x08094369 in RXAFS_ExecuteRequest (z_call=3D0x8230110) at ../fsint=
/afsint.ss.c:159
> #10 0x080b97cc in rxi_ServerProc (threadID=3D5, newcall=3D0xad862f00, soc=
ketp=3D0xb11a73b8) at ./../rx/rx.c:1418
> #11 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
> #12 0x08096df9 in server_entry (argp=3D0x8097800) at ../rx/rx_pthread.c:1=
00
> #13 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
> #14 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 12 (process 5128):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c7db8 in recvmsg () from /lib/tls/i686/cmov/libpthread.so.0
> #2 =A00x080bed67 in rxi_ReadPacket (socket=3D5, p=3D0x8186c00, host=3D0xb=
19a8384, port=3D0xb19a838a) at ./../rx/rx_packet.c:1411
> #3 =A00x08097296 in rxi_ListenerProc (sock=3D5, tnop=3D0xb19a83b4, newcal=
lp=3D0xb19a83b0) at ../rx/rx_pthread.c:230
> #4 =A00x080978ad in rx_ServerProc () at ../rx/rx_pthread.c:305
> #5 =A00x08096df9 in server_entry (argp=3D0x8097800) at ../rx/rx_pthread.c=
:100
> #6 =A00xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so=
.0
> #7 =A00xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 11 (process 5127):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/=
cmov/libpthread.so.0
> #2 =A00x08074a2b in Afs_Lock_Obtain (lock=3D0x8134194, how=3D2) at ../lwp=
/lock.c:94
> #3 =A00x08061e23 in h_Lock_r (host=3D0x8134140) at ../viced/host.c:534
> #4 =A00x08065256 in h_Lookup_r (haddr=3D3143495042, hport=3D22811, heldp=
=3D0xb21a8de4) at ../viced/host.c:840
> #5 =A00x080654ed in h_GetHost_r (tcon=3D0x824bbe0) at ../viced/host.c:142=
0
> #6 =A00x080672ae in h_FindClient_r (tcon=3D0x824bbe0) at ../viced/host.c:=
2032
> #7 =A00x0804fe82 in CallPreamble (acall=3D<value optimized out>, activeca=
ll=3D1, tconn=3D0xb21a91c8, ahostp=3D0xb21a91c4) at ../viced/afsfileprocs.c=
:321
> #8 =A00x0805e6e1 in SRXAFS_FetchStatus (acall=3D0xad44a448, Fid=3D0xb21a9=
300, OutStatus=3D0xb21a9210, CallBack=3D0xb21a9318, Sync=3D0xb21a92b8) at .=
./viced/afsfileprocs.c:2962
> #9 =A00x08094369 in RXAFS_ExecuteRequest (z_call=3D0xad44a448) at ../fsin=
t/afsint.ss.c:159
> #10 0x080b97cc in rxi_ServerProc (threadID=3D7, newcall=3D0x832ebf8, sock=
etp=3D0xb21a93b8) at ./../rx/rx.c:1418
> #11 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
> #12 0x08096df9 in server_entry (argp=3D0x8097800) at ../rx/rx_pthread.c:1=
00
> #13 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
> #14 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 10 (process 5126):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/=
cmov/libpthread.so.0
> #2 =A00x080c2180 in rxi_ReadProc (call=3D0xad814d60, buf=3D0xb29a9adb "=
=B2`M\201=AD\034\233\232=B2\030\202\203=ADX\233\232=B2f=E2\b\b`M\201=AD", n=
bytes=3D1) at ./../rx/rx_rdwr.c:234
> #3 =A00x080b9667 in rx_EndCall (call=3D0xad814d60, rc=3D0) at ./../rx/rx.=
c:1898
> #4 =A00x0808e266 in RXAFSCB_CallBack (z_conn=3D0xad838218, Fids_Array=3D0=
xb29a9de4, CallBacks_Array=3D0xb29a9ddc) at ../fsint/afscbint.cs.c:32
> #5 =A00x0806c043 in XCallBackBulk_r (ahost=3D0x8139840, fids=3D0xb29a9e30=
, nfids=3D0) at ../viced/callback.c:601
> #6 =A00x0806c172 in BreakDelayedCallBacks_r (host=3D0x8139840) at ../vice=
d/callback.c:1151
> #7 =A00x0805018d in CallPreamble (acall=3D<value optimized out>, activeca=
ll=3D0, tconn=3D0xb29aa168, ahostp=3D0xb29aa164) at ../viced/afsfileprocs.c=
:381
> #8 =A00x0805091e in SRXAFS_GetTime (acall=3D0x8233440, Seconds=3D0xb29aa1=
c8, USeconds=3D0xb29aa1c4) at ../viced/afsfileprocs.c:6845
> #9 =A00x08090623 in _RXAFS_GetTime (z_call=3D0x8233440, z_xdrs=3D0x5561) =
at ../fsint/afsint.ss.c:1109
> #10 0x0809380a in RXAFS_ExecuteRequest (z_call=3D0x8233440) at ../fsint/a=
fsint.ss.c:1941
> #11 0x080b97cc in rxi_ServerProc (threadID=3D13, newcall=3D0xad32ac30, so=
cketp=3D0xb29aa3b8) at ./../rx/rx.c:1418
> #12 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
> #13 0x08096df9 in server_entry (argp=3D0x8097800) at ../rx/rx_pthread.c:1=
00
> #14 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
> #15 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 9 (process 5125):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/=
cmov/libpthread.so.0
> #2 =A00x080c2180 in rxi_ReadProc (call=3D0xad648dc8, buf=3D0xb31aab14 "D=
=AB\032=B38=AB\032=B3=A06\f\b\234=AB\032=B34=AB\032=B3H=AB\032=B3=C5?\f\b=
=C8\215d=ADD=AB\032=B3h=AB\032=B3=CEa\t\b\234=AB\032=B38=AC\032=B3h=AB\032=
=B3z6\f\b\234=AB\032=B3d=AB\032=B3=A5=D5\016", nbytes=3D4) at ./../rx/rx_rd=
wr.c:234
> #3 =A00x080c25f6 in rx_ReadProc32 (call=3D0xad648dc8, value=3D0xb31aab14)=
at ./../rx/rx_rdwr.c:388
> #4 =A00x080c3feb in xdrrx_getint32 (axdrs=3D0xb31aab9c, lp=3D0xb31aab34) =
at ./../rx/xdr_rx.c:211
> #5 =A00x080c36a0 in xdr_int (xdrs=3D0x80, ip=3D0xb31aac38) at ./../rx/xdr=
.c:99
> #6 =A00x080961ce in xdr_interfaceAddr (xdrs=3D0xb31aab9c, objp=3D0xb31aac=
38) at ../fsint/afsint.xdr.c:363
> #7 =A00x0808c486 in RXAFSCB_TellMeAboutYourself (z_conn=3D0x83312e0, addr=
=3D0xb31aac38, capabilities=3D0xb31aaddc) at ../fsint/afscbint.cs.c:826
> #8 =A00x080658d2 in h_GetHost_r (tcon=3D0x8260d70) at ../viced/host.c:144=
7
> #9 =A00x080672ae in h_FindClient_r (tcon=3D0x8260d70) at ../viced/host.c:=
2032
> #10 0x0804fe82 in CallPreamble (acall=3D<value optimized out>, activecall=
=3D1, tconn=3D0xb31ab1c8, ahostp=3D0xb31ab1c4) at ../viced/afsfileprocs.c:3=
21
> #11 0x0805e6e1 in SRXAFS_FetchStatus (acall=3D0x81f7cb8, Fid=3D0xb31ab300=
, OutStatus=3D0xb31ab210, CallBack=3D0xb31ab318, Sync=3D0xb31ab2b8) at ../v=
iced/afsfileprocs.c:2962
> #12 0x08094369 in RXAFS_ExecuteRequest (z_call=3D0x81f7cb8) at ../fsint/a=
fsint.ss.c:159
> #13 0x080b97cc in rxi_ServerProc (threadID=3D10, newcall=3D0xad85e410, so=
cketp=3D0xb31ab3b8) at ./../rx/rx.c:1418
> #14 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
> #15 0x08096df9 in server_entry (argp=3D0x8097800) at ../rx/rx_pthread.c:1=
00
> #16 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
> #17 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 8 (process 5124):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/=
cmov/libpthread.so.0
> #2 =A00x08074a2b in Afs_Lock_Obtain (lock=3D0x8134194, how=3D2) at ../lwp=
/lock.c:94
> #3 =A00x08061e23 in h_Lock_r (host=3D0x8134140) at ../viced/host.c:534
> #4 =A00x08065256 in h_Lookup_r (haddr=3D3143495042, hport=3D22811, heldp=
=3D0xb39abde4) at ../viced/host.c:840
> #5 =A00x080654ed in h_GetHost_r (tcon=3D0x823a128) at ../viced/host.c:142=
0
> #6 =A00x080672ae in h_FindClient_r (tcon=3D0x823a128) at ../viced/host.c:=
2032
> #7 =A00x0804fe82 in CallPreamble (acall=3D<value optimized out>, activeca=
ll=3D1, tconn=3D0xb39ac1c8, ahostp=3D0xb39ac1c4) at ../viced/afsfileprocs.c=
:321
> #8 =A00x0805e6e1 in SRXAFS_FetchStatus (acall=3D0xad8b1918, Fid=3D0xb39ac=
300, OutStatus=3D0xb39ac210, CallBack=3D0xb39ac318, Sync=3D0xb39ac2b8) at .=
./viced/afsfileprocs.c:2962
> #9 =A00x08094369 in RXAFS_ExecuteRequest (z_call=3D0xad8b1918) at ../fsin=
t/afsint.ss.c:159
> #10 0x080b97cc in rxi_ServerProc (threadID=3D3, newcall=3D0x820d0a8, sock=
etp=3D0xb39ac3b8) at ./../rx/rx.c:1418
> #11 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
> #12 0x08096df9 in server_entry (argp=3D0x8097800) at ../rx/rx_pthread.c:1=
00
> #13 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
> #14 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 7 (process 5123):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/=
cmov/libpthread.so.0
> #2 =A00x080b66e2 in rx_GetCall (tno=3D8, cur_service=3D0x811c6b8, socketp=
=3D0xb41ad3b8) at ./../rx/rx.c:1613
> #3 =A00x080b982f in rxi_ServerProc (threadID=3D8, newcall=3D0x825b448, so=
cketp=3D0xb41ad3b8) at ./../rx/rx.c:1376
> #4 =A00x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
> #5 =A00x08096df9 in server_entry (argp=3D0x8097800) at ../rx/rx_pthread.c=
:100
> #6 =A00xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so=
.0
> #7 =A00xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 6 (process 5122):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/=
cmov/libpthread.so.0
> #2 =A00x080c2180 in rxi_ReadProc (call=3D0x8219080, buf=3D0xb49add6b "", =
nbytes=3D1) at ./../rx/rx_rdwr.c:234
> #3 =A00x080b9667 in rx_EndCall (call=3D0x8219080, rc=3D0) at ./../rx/rx.c=
:1898
> #4 =A00x0808db56 in RXAFSCB_InitCallBackState3 (z_conn=3D0x83e7078, serve=
rUuid=3D0x80ff494) at ../fsint/afscbint.cs.c:460
> #5 =A00x0806c2c8 in BreakDelayedCallBacks_r (host=3D0x8133740) at ../vice=
d/callback.c:1097
> #6 =A00x0805018d in CallPreamble (acall=3D<value optimized out>, activeca=
ll=3D1, tconn=3D0xb49ae154, ahostp=3D0xb49ae150) at ../viced/afsfileprocs.c=
:381
> #7 =A00x08055a53 in common_GiveUpCallBacks (acall=3D0xad6fa6f8, FidArray=
=3D0xb49ae1b4, CallBackArray=3D0xb49ae1ac) at ../viced/afsfileprocs.c:6106
> #8 =A00x0808f73b in _RXAFS_GiveUpCallBacks (z_call=3D0xad6fa6f8, z_xdrs=
=3D0xb49ae2e8) at ../fsint/afsint.ss.c:838
> #9 =A00x080931f6 in RXAFS_ExecuteRequest (z_call=3D0xad6fa6f8) at ../fsin=
t/afsint.ss.c:1923
> #10 0x080b97cc in rxi_ServerProc (threadID=3D6, newcall=3D0xad6fa6f8, soc=
ketp=3D0xb49ae3b8) at ./../rx/rx.c:1418
> #11 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
> #12 0x08096df9 in server_entry (argp=3D0x8097800) at ../rx/rx_pthread.c:1=
00
> #13 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
> #14 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 5 (process 5121):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/=
cmov/libpthread.so.0
> #2 =A00x080b66e2 in rx_GetCall (tno=3D12, cur_service=3D0x811c6b8, socket=
p=3D0xb51af3b8) at ./../rx/rx.c:1613
> #3 =A00x080b982f in rxi_ServerProc (threadID=3D12, newcall=3D0x825b448, s=
ocketp=3D0xb51af3b8) at ./../rx/rx.c:1376
> #4 =A00x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
> #5 =A00x08096df9 in server_entry (argp=3D0x8097800) at ../rx/rx_pthread.c=
:100
> #6 =A00xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so=
.0
> #7 =A00xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 4 (process 5120):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/=
cmov/libpthread.so.0
> #2 =A00x080c2180 in rxi_ReadProc (call=3D0x82a81c0, buf=3D0xb59afb14 "D=
=FB\232=B58=FB\232=B5=A06\f\b\234=FB\232=B54=FB\232=B5H=FB\232=B5=C5?\f\b=
=C0\201*\bD=FB\232=B5h=FB\232=B5=CEa\t\b\234=FB\232=B58=FC\232=B5h=FB\232=
=B5z6\f\b\234=FB\232=B5d=FB\232=B5`=F9\016", nbytes=3D4) at ./../rx/rx_rdwr=
.c:234
> #3 =A00x080c25f6 in rx_ReadProc32 (call=3D0x82a81c0, value=3D0xb59afb14) =
at ./../rx/rx_rdwr.c:388
> #4 =A00x080c3feb in xdrrx_getint32 (axdrs=3D0xb59afb9c, lp=3D0xb59afb34) =
at ./../rx/xdr_rx.c:211
> #5 =A00x080c36a0 in xdr_int (xdrs=3D0x80, ip=3D0xb59afc38) at ./../rx/xdr=
.c:99
> #6 =A00x080961ce in xdr_interfaceAddr (xdrs=3D0xb59afb9c, objp=3D0xb59afc=
38) at ../fsint/afsint.xdr.c:363
> #7 =A00x0808c486 in RXAFSCB_TellMeAboutYourself (z_conn=3D0xad6f89c8, add=
r=3D0xb59afc38, capabilities=3D0xb59afddc) at ../fsint/afscbint.cs.c:826
> #8 =A00x080658d2 in h_GetHost_r (tcon=3D0xad281a88) at ../viced/host.c:14=
47
> #9 =A00x080672ae in h_FindClient_r (tcon=3D0xad281a88) at ../viced/host.c=
:2032
> #10 0x0804fe82 in CallPreamble (acall=3D<value optimized out>, activecall=
=3D1, tconn=3D0xb59b01c8, ahostp=3D0xb59b01c4) at ../viced/afsfileprocs.c:3=
21
> #11 0x0805e6e1 in SRXAFS_FetchStatus (acall=3D0x82147a8, Fid=3D0xb59b0300=
, OutStatus=3D0xb59b0210, CallBack=3D0xb59b0318, Sync=3D0xb59b02b8) at ../v=
iced/afsfileprocs.c:2962
> #12 0x08094369 in RXAFS_ExecuteRequest (z_call=3D0x82147a8) at ../fsint/a=
fsint.ss.c:159
> #13 0x080b97cc in rxi_ServerProc (threadID=3D4, newcall=3D0xad8065e8, soc=
ketp=3D0xb59b03b8) at ./../rx/rx.c:1418
> #14 0x0809789a in rx_ServerProc () at ../rx/rx_pthread.c:302
> #15 0x08096df9 in server_entry (argp=3D0x8097800) at ../rx/rx_pthread.c:1=
00
> #16 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
> #17 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 3 (process 5081):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c4dd2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/=
i686/cmov/libpthread.so.0
> #2 =A00x0809778f in event_handler (argp=3D0x0) at ../rx/rx_pthread.c:173
> #3 =A00xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so=
.0
> #4 =A00xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 2 (process 5080):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c4aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/=
cmov/libpthread.so.0
> #2 =A00x08074a2b in Afs_Lock_Obtain (lock=3D0x8134194, how=3D2) at ../lwp=
/lock.c:94
> #3 =A00x08061e23 in h_Lock_r (host=3D0x8134140) at ../viced/host.c:534
> #4 =A00x08065256 in h_Lookup_r (haddr=3D3143495042, hport=3D22811, heldp=
=3D0xb6e26df4) at ../viced/host.c:840
> #5 =A00x080654ed in h_GetHost_r (tcon=3D0x8374160) at ../viced/host.c:142=
0
> #6 =A00x080672ae in h_FindClient_r (tcon=3D0x8374160) at ../viced/host.c:=
2032
> #7 =A00x0804fe82 in CallPreamble (acall=3D<value optimized out>, activeca=
ll=3D1, tconn=3D0xb6e271d8, ahostp=3D0xb6e271d4) at ../viced/afsfileprocs.c=
:321
> #8 =A00x0805e6e1 in SRXAFS_FetchStatus (acall=3D0xad424f70, Fid=3D0xb6e27=
310, OutStatus=3D0xb6e27220, CallBack=3D0xb6e27328, Sync=3D0xb6e272c8) at .=
./viced/afsfileprocs.c:2962
> #9 =A00x08094369 in RXAFS_ExecuteRequest (z_call=3D0xad424f70) at ../fsin=
t/afsint.ss.c:159
> #10 0x080b97cc in rxi_ServerProc (threadID=3D9, newcall=3D0x8488710, sock=
etp=3D0xb6e273c4) at ./../rx/rx.c:1418
> #11 0x08097376 in rx_ListenerProc (argp=3D0x5) at ../rx/rx_pthread.c:261
> #12 0xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
> #13 0xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
>
> Thread 1 (process 5079):
> #0 =A00xb77db410 in __kernel_vsyscall ()
> #1 =A00xb77c8b1a in do_sigwait () from /lib/tls/i686/cmov/libpthread.so.0
> #2 =A00xb77c8bbf in sigwait () from /lib/tls/i686/cmov/libpthread.so.0
> #3 =A00x08079610 in softsig_thread (arg=3D0x0) at ../util/softsig.c:93
> #4 =A00xb77c04fb in start_thread () from /lib/tls/i686/cmov/libpthread.so=
.0
> #5 =A00xb772ef5e in clone () from /lib/tls/i686/cmov/libc.so.6
> _______________________________________________
> OpenAFS-devel mailing list
> OpenAFS-devel@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-devel
>
--=20
Derrick