[OpenAFS] Re: busy .backup volumes
Kristen J. Webb
kwebb@teradactyl.com
Thu, 02 May 2013 13:03:30 -0600
I'm not sure if this is enough info to be useful:
(gdb) thread apply all
Thread 13 (Thread 0x7f83705ed950
#0 0x00007f8370b6bdab in recvmsg
#1 0x000000000042d649 in rxi_Recvmsg
#2 0x0000000000456fa5 in rxi_ReadPacket
#3 0x000000000042da58 in rxi_ListenerProc
#4 0x000000000042db5f in rx_ListenerProc
#5 0x00007f8370b65070 in start_thread
#6 0x00007f83706c110d in clone
#7 0x0000000000000000 in ??
Thread 12 (Thread 0x7f836fdec950
#0 0x00007f8370b68fdd in pthread_cond_timedwait@@GLIBC_2.3.2
#1 0x000000000042def8 in event_handler
#2 0x00007f8370b65070 in start_thread
#3 0x00007f83706c110d in clone
#4 0x0000000000000000 in ??
Thread 11 (Thread 0x7f836f5eb950
#0 0x00007f83706ba662 in select
#1 0x00000000004045ca in BKGLoop
#2 0x00007f8370b65070 in start_thread
#3 0x00007f83706c110d in clone
#4 0x0000000000000000 in ??
Thread 10 (Thread 0x7f836edea950
#0 0x00007f8370b68d59 in pthread_cond_wait@@GLIBC_2.3.2
#1 0x000000000044eafa in rx_GetCall
#2 0x0000000000451b12 in rxi_ServerProc
#3 0x000000000042dff8 in rx_ServerProc
#4 0x000000000042d658 in server_entry
#5 0x00007f8370b65070 in start_thread
#6 0x00007f83706c110d in clone
#7 0x0000000000000000 in ??
Thread 9 (Thread 0x7f836e5e9950
#0 0x00007f8370b68d59 in pthread_cond_wait@@GLIBC_2.3.2
#1 0x000000000044eafa in rx_GetCall
#2 0x0000000000451b12 in rxi_ServerProc
#3 0x000000000042dff8 in rx_ServerProc
#4 0x000000000042d658 in server_entry
#5 0x00007f8370b65070 in start_thread
#6 0x00007f83706c110d in clone
#7 0x0000000000000000 in ??
Thread 8 (Thread 0x7f836dde8950
#0 0x00007f8370b68d59 in pthread_cond_wait@@GLIBC_2.3.2
#1 0x000000000044eafa in rx_GetCall
#2 0x0000000000451b12 in rxi_ServerProc
#3 0x000000000042dff8 in rx_ServerProc
#4 0x000000000042d658 in server_entry
#5 0x00007f8370b65070 in start_thread
#6 0x00007f83706c110d in clone
#7 0x0000000000000000 in ??
Thread 7 (Thread 0x7f836d5e7950
#0 0x00007f8370b68d59 in pthread_cond_wait@@GLIBC_2.3.2
---Type <return> to continue,
#1 0x000000000044eafa in rx_GetCall
#2 0x0000000000451b12 in rxi_ServerProc
#3 0x000000000042dff8 in rx_ServerProc
#4 0x000000000042d658 in server_entry
#5 0x00007f8370b65070 in start_thread
#6 0x00007f83706c110d in clone
#7 0x0000000000000000 in ??
Thread 6 (Thread 0x7f836cde6950
#0 0x00007f8370b68d59 in pthread_cond_wait@@GLIBC_2.3.2
#1 0x000000000044eafa in rx_GetCall
#2 0x0000000000451b12 in rxi_ServerProc
#3 0x000000000042dff8 in rx_ServerProc
#4 0x000000000042d658 in server_entry
#5 0x00007f8370b65070 in start_thread
#6 0x00007f83706c110d in clone
#7 0x0000000000000000 in ??
Thread 5 (Thread 0x7f8367fff950
#0 0x00007f8370b68d59 in pthread_cond_wait@@GLIBC_2.3.2
#1 0x000000000044eafa in rx_GetCall
#2 0x0000000000451b12 in rxi_ServerProc
#3 0x000000000042dff8 in rx_ServerProc
#4 0x000000000042d658 in server_entry
#5 0x00007f8370b65070 in start_thread
#6 0x00007f83706c110d in clone
#7 0x0000000000000000 in ??
Thread 4 (Thread 0x7f83677fe950
#0 0x00007f8370b68d59 in pthread_cond_wait@@GLIBC_2.3.2
#1 0x000000000044eafa in rx_GetCall
#2 0x0000000000451b12 in rxi_ServerProc
#3 0x000000000042dff8 in rx_ServerProc
#4 0x000000000042d658 in server_entry
#5 0x00007f8370b65070 in start_thread
#6 0x00007f83706c110d in clone
#7 0x0000000000000000 in ??
Thread 3 (Thread 0x7f8365ffb950
#0 0x00007f8370b68d59 in pthread_cond_wait@@GLIBC_2.3.2
#1 0x000000000044eafa in rx_GetCall
#2 0x0000000000451b12 in rxi_ServerProc
#3 0x000000000042dff8 in rx_ServerProc
#4 0x000000000042d658 in server_entry
#5 0x00007f8370b65070 in start_thread
#6 0x00007f83706c110d in clone
#7 0x0000000000000000 in ??
Thread 2 (Thread 0x7f83657fa950
#0 0x00007f837068fce1 in nanosleep
#1 0x00007f837068fadc in sleep
#2 0x0000000000429e62 in ih_sync_thread
#3 0x00007f8370b65070 in start_thread
#4 0x00007f83706c110d in clone
#5 0x0000000000000000 in ??
Thread 1 (Thread 0x7f8370f5b6f0
---Type <return> to continue,
#0 0x00007f8370b68d59 in pthread_cond_wait@@GLIBC_2.3.2
#1 0x000000000044eafa in rx_GetCall
#2 0x0000000000451b12 in rxi_ServerProc
#3 0x000000000042dff8 in rx_ServerProc
#4 0x000000000045317f in rx_StartServer
#5 0x0000000000403d58 in main
On 5/2/13 11:44 AM, Andrew Deason wrote:
> On Thu, 02 May 2013 11:19:41 -0600
> "Kristen J. Webb" <kwebb@teradactyl.com> wrote:
>
>> I've omitted the volumes from backup so I can control the
>> order of things better next time. Is there anything else
>> useful I can get before restarting the server?
>
> Capture a core. What seems likely is the Dump RPC is executing, the
> relevant Rx call is errored, but for whatever reason the Dump RPC is not
> returning. I don't know what it could be hanging on if the Rx call is
> errored, so all you can do is see where it is stuck.
>
> A core itself obviously has a lot of sensitive information in it.
> However, just the stack trace will not, as long as you make sure to trim
> all of the function arguments that may appear in it. (In gdb, get a
> stack trace with 'thread apply all bt'.) But just save the core even if
> you don't do anything besides get a stack trace, so you can poke around
> in it later if necessary.
>
--
This message is NOT encrypted
--------------------------------
Mr. Kristen J. Webb
Chief Technology Officer
Teradactyl LLC.
2450 Baylor Dr. S.E.
Albuquerque, New Mexico 87106
Phone: 1-505-338-6000
Email: kwebb@teradactyl.com
Web: http://www.teradactyl.com
Providers of Scalable Backup Solutions
for Unique Data Environments
--------------------------------
NOTICE TO RECIPIENTS: Any information contained in or attached to this message
is intended solely for the use of the intended recipient(s). If you are not the
intended recipient of this transmittal, you are hereby notified that you
received this transmittal in error, and we request that you please delete and
destroy all copies and attachments in your possession, notify the sender that
you have received this communication in error, and note that any review or
dissemination of, or the taking of any action in reliance on, this communication
is expressly prohibited.
Regular internet e-mail transmission cannot be guaranteed to be secure or
error-free. Therefore, we do not represent that this information is complete or
accurate, and it should not be relied upon as such. If you prefer to communicate
with Teradactyl LLC. using secure (i.e., encrypted and/or digitally signed)
e-mail transmission, please notify the sender. Otherwise, you will be deemed to
have consented to communicate with Teradactyl via regular internet e-mail
transmission. Please note that Teradactyl reserves the right to intercept,
monitor, and retain all e-mail messages (including secure e-mail messages) sent
to or from its systems as permitted by applicable law.