[OpenAFS] Re: Help with wedged Solaris box
Kevin Hildebrand
kevin@umd.edu
Thu, 29 Nov 2012 16:31:30 -0500 (EST)
::findlocks is not returning anything useful-
> ::findlocks
findlocks: nota bene: locks may be held
::stacks is showing some AFS threads awake, but I'm not sure which (if
any) are relevant-
> ::stacks
THREAD STATE SOBJ COUNT
3006c632420 SLEEP CV 2296
lwp_park+0x130
syslwp_park+0x54
syscall_trap+0xac
2a10000fc80 FREE <NONE> 511
<no consistent stack found>
2a100077c80 SLEEP CV 284
taskq_thread_wait+0x50
taskq_thread+0x358
thread_start+4
2a100557c80 SLEEP CV 136
taskq_thread+0x358
thread_start+4
3000c694440 SLEEP CV 127
lwp_park+0x130
syslwp_park+0x54
syscall_trap32+0xcc
30076144440 SLEEP MUTEX 89
mutex_vector_enter+0x428
afs_root+0x3c
fsop_root+0x10
traverse+0x7c
lookuppnvp+0x3d0
lookuppnat+0x120
lookupnameat+0xa8
cstatat_getvp+0x198
cstatat64_32+0x40
syscall_trap32+0xcc
2a10967bc80 SLEEP CV 88
taskq_d_thread+0x158
thread_start+4
2a100e59c80 FREE <NONE> 74
cbe_level10+8
intr_thread+0x198
2a100d93c80 ONPROC <NONE> 73
thread_start+4
2a1072f3c80 SLEEP CV 67
squeue_worker+0x29c
thread_start+4
3000994e3a0 SLEEP SHUTTLE 63
syscall_trap32+0xcc
2a10001fc80 ONPROC <NONE> 54
ktl0+0x64
cpu_halt+0x104
idle+0x128
thread_start+4
3007612a4a0 SLEEP MUTEX 38
mutex_vector_enter+0x428
afs_xsetgroups+0x40
syscall_trap32+0xcc
3000994a000 SLEEP CV 34
cv_waituntil_sig+0xb0
semop+0x564
syscall_trap+0xac
30053620b40 SLEEP CV 33
str_cv_wait+0x28
strwaitq+0x238
kstrgetmsg+0xdd8
sotpi_recvmsg+0x2b8
socktpi_read+0x44
fop_read+0x20
read+0x274
syscall_trap+0xac
300535cc0e0 SLEEP CV 26
aio_cleanup_thread+0x1b4
kaioc+0xe0
syscall_trap+0xac
300645be700 SLEEP MUTEX 20
mutex_vector_enter+0x428
afs_root+0x3c
fsop_root+0x10
traverse+0x7c
lookuppnvp+0x3d0
lookuppnat+0x120
lookupnameat+0x5c
cstatat_getvp+0x198
cstatat+0x40
syscall_trap+0xac
30009a76ea0 SLEEP CV 17
cv_waituntil_sig+0xb0
nanosleep+0x100
syscall_trap32+0xcc
30076130360 SLEEP MUTEX 16
mutex_vector_enter+0x428
afs_root+0x3c
fsop_root+0x10
traverse+0x7c
lookuppnvp+0x3d0
lookuppnat+0x120
lookupnameat+0x5c
access+0xb4
syscall_trap+0xac
2a101201c80 FREE <NONE> 13
cyclic_softint+0x120
cbe_level10+8
intr_thread+0x198
30009932ac0 SLEEP CV 13
poll_common+0x4e8
pollsys+0xf8
syscall_trap32+0xcc
2a100dfbc80 FREE <NONE> 12
cbe_level1+8
intr_thread+0x198
2a103107c80 SLEEP CV 12
md_daemon+0x144
thread_start+4
2a100c43c80 SLEEP CV 12
nfs_async_manager+0x1c0
thread_start+4
2a103157c80 SLEEP CV 11
trans_roll_wait+0x7c
trans_roll+0x29c
thread_start+4
2a10063fc80 SLEEP CV 11
ufs_thread_run+0x134
ufs_thread_delete+0x54
thread_start+4
2a1006d7c80 SLEEP CV 10
emlxs_thread+0x144
thread_start+4
30009944140 SLEEP CV 9
cv_waituntil_sig+0xb0
lwp_park+0x130
syslwp_park+0x54
syscall_trap32+0xcc
2a100d63c80 SLEEP CV 8
evch_delivery_hold+0x64
evch_delivery_thr+0x228
thread_start+4
30009a7a4c0 SLEEP CV 8
pause+0x10
syscall_trap32+0xcc
3004f18fb60 SLEEP CV 7
cv_waituntil_sig+0xb0
poll_common+0x4e8
pollsys+0xf8
syscall_trap+0xac
30009818380 SLEEP CV 6
sigtimedwait+0x148
syscall_trap32+0xcc
2a1010a9c80 FREE <NONE> 5
cyclic_softint+0xac
cbe_level10+8
intr_thread+0x198
3000c130d80 SLEEP CV 5
cv_waituntil_sig+0xb0
nanosleep+0x100
syscall_trap+0xac
300099bdba0 SLEEP CV 5
cv_waituntil_sig+0xb0
poll_common+0x4e8
pollsys+0xf8
syscall_trap32+0xcc
3007537f1a0 SLEEP CV 5
waitid+0x48c
waitsys32+0x10
syscall_trap32+0xcc
2a101129c80 FREE <NONE> 4
cyclic_softint+0x120
cbe_level1+8
intr_thread+0x198
2a1073d7c80 SLEEP CV 4
i_mac_notify_thread+0x94
thread_start+4
2a1024b5c80 SLEEP CV 4
soft_ring_worker+0x11c
thread_start+4
2a1011f1c80 FREE <NONE> 3
taskq_dispatch+0x108
callout_realtime+0x14
cyclic_softint+0xac
cbe_level10+8
intr_thread+0x198
2a103b43c80 SLEEP CV 3
afs_osi_Sleep+0x80
afs_BackgroundDaemon+0x454
afsd_thread+0x1e8
thread_start+4
3007612b580 SLEEP CV 3
fifo_read+0x90
fop_read+0x20
read+0x274
syscall_trap+0xac
30009940120 SLEEP CV 3
sigsuspend+0xa8
syscall_trap32+0xcc
30053552d80 ONPROC <NONE> 3
ktl0+0x64
default_lock_delay+0x6c
mutex_vector_enter+0x460
afs_root+0x3c
fsop_root+0x10
traverse+0x7c
lookuppnvp+0x3d0
lookuppnat+0x120
lookupnameat+0xa8
cstatat_getvp+0x198
cstatat64_32+0x40
syscall_trap32+0xcc
2a101399c80 FREE <NONE> 2
callout_normal+0xa0
cyclic_softint+0xac
cbe_level1+8
intr_thread+0x198
2a1011b1c80 FREE <NONE> 2
callout_normal+0xa0
cyclic_softint+0xac
cbe_level10+8
intr_thread+0x198
2a103b6bc80 SLEEP CV 2
afs_osi_Sleep+0x80
afs_GetDCache+0x1724
BPrefetch+0x68
afs_BackgroundDaemon+0x2f4
afsd_thread+0x1e8
thread_start+4
3007524d7e0 SLEEP CV 2
afs_osi_Sleep+0x80
afs_GetDCache+0x1724
afs_GetOnePage+0x490
afs_getpage+0xe8
fop_getpage+0x44
segmap_fault+0x230
afs_nfsrdwr+0x1130
afs_vmread+0x70
fop_read+0x20
read+0x274
syscall_trap+0xac
30073165c60 SLEEP CV 2
afs_osi_Sleep+0x80
afs_nfsrdwr+0x2fc
afs_vmwrite+0x70
fop_write+0x20
write+0x268
syscall_trap32+0xcc
30009818a40 SLEEP CV 2
cte_get_event+0x5c
fop_ioctl+0x2c
ioctl+0x184
syscall_trap32+0xcc
3000c5c7140 SLEEP CV 2
cv_waituntil_sig+0xb0
lwp_park+0x130
syslwp_park+0x54
syscall_trap+0xac
3000991db40 SLEEP CV 2
door_unref+0xa8
syscall_trap32+0xcc
2a100747c80 SLEEP CV 2
fp_job_handler+0xe90
thread_start+4
3004f7d2ea0 SLEEP MUTEX 2
mutex_vector_enter+0x428
afs_getpage+0x8c
fop_getpage+0x44
segvn_fault_vnodepages+0x644
segvn_fault+0x4ec
as_fault+0x4c8
pagefault+0x68
trap+0xd50
utl0+0x4c
3000c20c6e0 SLEEP CV 2
poll_common+0x4e8
pollsys+0xf8
syscall_trap+0xac
2a1039f9c80 SLEEP CV 2
svc_thread_creator+0xa0
thread_start+4
3000992e080 SLEEP CV 2
svc_wait+0xbc
nfssys+0x194
syscall_trap32+0xcc
2a100e51c80 FREE <NONE> 1
0
px_msiq_intr+0x214
intr_thread+0x198
2a1028bfc80 FREE <NONE> 1
2
callout_expire+0x1c
callout_realtime+0x14
cyclic_softint+0xac
cbe_level1+8
intr_thread+0x198
2a102947c80 FREE <NONE> 1
0x1580
callout_expire+0x1c
callout_realtime+0x14
cyclic_softint+0xac
cbe_level1+8
intr_thread+0x198
2a100047c80 FREE <NONE> 1
clock_tick_execute_common+0x84
clock_tick_schedule+0x114
clock+0x498
cyclic_softint+0xac
cbe_level10+8
intr_thread+0x198
2a10210bc80 FREE <NONE> 1
ehci_hcdi_callback+0xb0
ehci_handle_normal_qtd+0x1c
ehci_traverse_active_qtd_list+0x134
ehci_intr+0x188
px_intx_intr+0x6c
intr_thread+0x198
2a100f49c80 FREE <NONE> 1
emlxs_sli3_msi_intr+0x208
px_msiq_intr+0x214
intr_thread+0x198
2a101029c80 FREE <NONE> 1
intr_thread+0x198
2a10004fc80 FREE <NONE> 1
lgrp_loadavg+0x11c
clock+0x498
cyclic_softint+0xac
cbe_level10+8
intr_thread+0x198
2a100dc3c80 FREE <NONE> 1
scsi_init_pkt+0x5c
sd_setup_rw_pkt+0xc0
sd_initpkt_for_buf+0x68
sd_start_cmds+0x1b4
sd_return_command+0x230
sdintr+0x28c
mpt_doneq_empty+0x74
mpt_intr+0xdc
px_msiq_intr+0x214
intr_thread+0x198
2a1021d3c80 FREE <NONE> 1
taskq_dispatch+0x28
callout_realtime+0x14
cyclic_softint+0xac
cbe_level10+8
intr_thread+0x198
2a1029cfc80 FREE <NONE> 1
taskq_dispatch+0x108
callout_normal+0xa0
cyclic_softint+0xac
cbe_level1+8
intr_thread+0x198
2a102bcbc80 FREE <NONE> 1
taskq_dispatch+0x108
callout_normal+0xa0
cyclic_softint+0xac
cbe_level10+8
intr_thread+0x198
2a10254fc80 FREE <NONE> 1
turnstile_block+0x5fc
mutex_vector_enter+0x428
callout_normal+4
cyclic_softint+0xac
cbe_level10+8
intr_thread+0x198
300637f8e80 SLEEP CV 1
afs_osi_Sleep+0x80
afs_GetDCache+0x1724
afs_GetOnePage+0x490
afs_getpage+0xe8
fop_getpage+0x44
segvn_fault+0xb00
as_fault+0x4c8
pagefault+0x68
trap+0xd50
utl0+0x4c
2a1073efc80 SLEEP CV 1
arc_reclaim_thread+0x164
thread_start+4
2a10311fc80 SLEEP CV 1
crypto_bufcall_service+0x160
thread_start+4
2a100397c80 SLEEP CV 1
cv_wait_sig+0x224
str_cv_wait+0x28
strwaitq+0x238
kstrgetmsg+0xdd8
sotpi_recvmsg+0x2b8
osi_NetReceive+0x118
rxk_ReadPacket+0x154
rxk_Listener+0xc0
afsd_thread+0x350
thread_start+4
3000994d100 SLEEP CV 1
cv_waituntil_sig+0xb0
sigtimedwait+0x148
syscall_trap32+0xcc
300546ee740 SLEEP CV 1
dpioctl+0x3d0
fop_ioctl+0x2c
ioctl+0x184
syscall_trap32+0xcc
30059bbae20 SLEEP CV 1
ds_pri_ioctl+0x1d8
fop_ioctl+0x2c
ioctl+0x184
syscall_trap32+0xcc
3000c692ae0 SLEEP CV 1
fifo_open+0x1d4
fop_open+0x78
vn_openat+0x500
copen+0x260
syscall_trap32+0xcc
3000c6840a0 SLEEP CV 1
fifo_read+0x90
fop_read+0x20
read+0x274
syscall_trap32+0xcc
2a1035fbc80 SLEEP CV 1
fsflush+0x1dc
thread_start+4
2a1009d3c80 SLEEP CV 1
idm_wd_thread+0x80
thread_start+4
2a100737c80 SLEEP CV 1
ipsec_loader+0xa0
thread_start+4
2a1024d5c80 SLEEP CV 1
irm_balance_thread+0x1e8
thread_start+4
2a1035f3c80 SLEEP CV 1
kcage_cageout+0xa4
thread_start+4
2a1030c7c80 SLEEP CV 1
kcf_failover_thread+0x1b0
thread_start+4
3000c20d460 SLEEP CV 1
kcf_svc_do_run+0x88
cryptoadm_ioctl+0x1e4
fop_ioctl+0x2c
ioctl+0x184
syscall_trap32+0xcc
30009946160 SLEEP CV 1
kcf_svc_wait+0xac
cryptoadm_ioctl+0x174
fop_ioctl+0x2c
ioctl+0x184
syscall_trap32+0xcc
2a1073bfc80 SLEEP CV 1
l2arc_feed_thread+0x98
thread_start+4
2a1024a5c80 SLEEP CV 1
log_event_deliver+0x268
thread_start+4
2a100797c80 SLEEP CV 1
mcast_restart_timers_thread+0xd8
thread_start+4
2a1035e3c80 SLEEP CV 1
mod_uninstall_daemon+0xe4
thread_start+4
2a1004e7c80 SLEEP MUTEX 1
mutex_vector_enter+0x428
SRXAFSCB_CallBack+0x50
_RXAFSCB_CallBack+0x6c
RXAFSCB_ExecuteRequest+0x44
rxi_ServerProc+0x150
rx_ServerProc+0xa4
afs_RXCallBackServer+0x4c
afsd_thread+0xa0
thread_start+4
30059190780 SLEEP MUTEX 1
mutex_vector_enter+0x428
afs_getpage+0x8c
fop_getpage+0x44
segvn_fault+0xb00
as_fault+0x4c8
pagefault+0x68
trap+0xd50
utl0+0x4c
30054712780 SLEEP MUTEX 1
mutex_vector_enter+0x428
afs_root+0x3c
fsop_root+0x10
traverse+0x7c
lookuppnvp+0x3d0
lookuppnat+0x120
exec_common+0x190
exece+0x10
syscall_trap+0xac
2a103b4bc80 SLEEP MUTEX 1
mutex_vector_enter+0x428
cv_timedwait_hires+0xd0
afs_osi_TimedSleep+0xfc
afs_osi_Wait+0xcc
afs_CheckServerDaemon+0x2b4
afsd_thread+0x27c
thread_start+4
2a103b5bc80 SLEEP MUTEX 1
mutex_vector_enter+0x428
cv_timedwait_hires+0xd0
afs_osi_TimedSleep+0xfc
afs_osi_Wait+0xcc
afs_Daemon+0x7c0
afsd_thread+0x14c
thread_start+4
2a10030fc80 SLEEP MUTEX 1
mutex_vector_enter+0x428
cv_timedwait_hires+0xd0
afs_osi_TimedSleep+0xfc
afs_osi_Wait+0xcc
afs_rxevent_daemon+0x34
afsd_thread+0x2e8
thread_start+4
300637f98a0 SLEEP MUTEX 1
mutex_vector_enter+0x428
gafs_lookup+0x30
fop_lookup+0x28
lookuppnvp+0x354
lookuppnat+0x120
lookupnameat+0x5c
chdir+0x24
syscall_trap32+0xcc
2a100337c80 SLEEP CV 1
nfsauth_refresh_thread+0xa4
thread_start+4
2a10053fc80 SLEEP CV 1
page_retire_thread+0x1f0
thread_start+4
2a1035cbc80 SLEEP CV 1
pageout+0x258
thread_start+4
2a1035c3c80 SLEEP CV 1
pageout_scanner+0x1a4
thread_start+4
2a100527c80 SLEEP CV 1
pm_dep_thread+0x8c
thread_start+4
3000991c700 SLEEP CV 1
port_getn+0x2d8
portfs+0x13c
syscall_trap32+0xcc
2a10006fc80 SLEEP CV 1
scsi_hba_barrier_daemon+0x90
thread_start+4
2a100067c80 SLEEP CV 1
scsi_lunchg1_daemon+0x80
thread_start+4
2a10009fc80 SLEEP CV 1
scsi_lunchg2_daemon+0x88
thread_start+4
2a1004cfc80 SLEEP CV 1
scsi_watch_thread+0x38c
thread_start+4
300099c00c0 SLEEP CV 1
sd_check_media+0x120
sdioctl+0xdb8
fop_ioctl+0x2c
ioctl+0x184
syscall_trap32+0xcc
2a10361bc80 SLEEP CV 1
seg_pasync_thread+0xdc
thread_start+4
3000993ab20 SLEEP CV 1
str_cv_wait+0x28
strwaitq+0x238
kstrgetmsg+0xdd8
sotpi_recvmsg+0x2b8
recvit+0x68
recvfrom+0x88
syscall_trap32+0xcc
300535b5840 SLEEP CV 1
str_cv_wait+0x28
strwaitq+0x238
kstrgetmsg+0xdd8
sotpi_recvmsg+0x2b8
socktpi_read+0x44
fop_read+0x20
read+0x274
syscall_trap32+0xcc
2a100517c80 SLEEP CV 1
streams_bufcall_service+0x170
thread_start+4
2a10050fc80 SLEEP CV 1
streams_qbkgrnd_service+0x90
thread_start+4
2a100507c80 SLEEP CV 1
streams_sqbkgrnd_service+0x90
thread_start+4
2a100017c80 SLEEP CV 1
thread_reaper+0xc0
thread_start+4
2a100537c80 SLEEP CV 1
ufs_thread_idle+0x178
thread_start+4
2a10052fc80 SLEEP CV 1
ufs_thread_run+0x134
ufs_thread_hlock+0x50
thread_start+4
2a102967c80 RUN <NONE> 1
squeue_worker+0x29c
thread_start+4
2a1095e3c80 RUN <NONE> 1
taskq_d_thread+0x158
thread_start+4
2a100597c80 RUN <NONE> 1
taskq_thread_wait+0x50
taskq_thread+0x358
thread_start+4
2a1013b1c80 ONPROC <NONE> 1
cpu_halt+0x104
idle+0x128
thread_start+4
2a100297c80 ONPROC <NONE> 1
ktl0+0x64
0x70400
afsd_thread+0x254
thread_start+4
3004f5bd520 ONPROC <NONE> 1
ktl0+0x64
default_lock_delay+4
mutex_vector_enter+0x460
afs_root+0x3c
fsop_root+0x10
traverse+0x7c
lookuppnvp+0x3d0
lookuppnat+0x120
lookupnameat+0x5c
access+0xb4
syscall_trap+0xac
300099387a0 ONPROC <NONE> 1
ktl0+0x64
default_lock_delay+4
mutex_vector_enter+0x460
afs_root+0x3c
fsop_root+0x10
traverse+0x7c
lookuppnvp+0x3d0
lookuppnat+0x120
lookupnameat+0x5c
cstatat_getvp+0x198
cstatat+0x40
syscall_trap+0xac
3005871a3c0 ONPROC <NONE> 1
ktl0+0x64
default_lock_delay+0x6c
mutex_vector_enter+0x460
afs_root+0x3c
fsop_root+0x10
traverse+0x7c
lookuppnvp+0x3d0
lookuppnat+0x120
lookupnameat+0x5c
cstatat_getvp+0x198
cstatat+0x40
syscall_trap+0xac
3006471f140 ONPROC <NONE> 1
ktl0+0x64
mutex_vector_enter+0x364
afs_root+0x3c
fsop_root+0x10
traverse+0x7c
lookuppnvp+0x3d0
lookuppnat+0x120
lookupnameat+0xa8
cstatat_getvp+0x198
cstatat64_32+0x40
syscall_trap32+0xcc
300546ef4c0 ONPROC <NONE> 1
ktl0+0x64
mutex_vector_enter+0x460
afs_root+0x3c
fsop_root+0x10
traverse+0x7c
lookuppnvp+0x3d0
lookuppnat+0x120
lookupnameat+0xa8
cstatat_getvp+0x198
cstatat64_32+0x40
syscall_trap32+0xcc
180e000 STOPPED <NONE> 1
_start+0x100
2a1013e1c80 PANIC <NONE> 1
sync_handler+0x150
vx_handler+0x80
promif_enter_mon+0x14c
kern_cif_handler+0x28
client_handler+0x2c
prom_enter_mon+0x24
debug_enter+0x118
abort_seq_softintr+0x94
intr_thread+0x198
On Thu, 29 Nov 2012, Andrew Deason wrote:
> On Thu, 29 Nov 2012 09:08:18 -0500 (EST)
> Kevin Hildebrand <kevin@umd.edu> wrote:
>
>> For this particular crash dump, I have hundreds of threads that are
>> stuck in this location.
>
> You should be able to find at least one of ours that's somewhere else.
> The ::stacks dcmd can help sort through a lot of duplicate stacks; or if
> you don't have that, there's a script that can help do the same thing at
> <http://hub.opensolaris.org/bin/download/Community+Group+mdb/tips/munges>
>
> Or look at the locks directly like Derrick said.
>
> --
> Andrew Deason
> adeason@sinenomine.net
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>