[OpenAFS-devel] Solaris panic when non-networked machine is shut down

Mon, 28 Oct 2002 00:39:00 +0100 (CET)

Hi all,

when a AFS client on a Solaris box is not able to make network contact
and then is shut down prior to get any network it panics in the manner
described below. As these are not my own boxes, I do not have first
hand experience, but I am told that this is behaviour which has been
repeated successfully. This is not a mission critical problem but
frustrating for the sysadmin which has to deal with savecore, fsck and
such stuff when doing pre install testing without net. Openafs version
should be 1.2.6.

Harald.

(messages from the swedish chef can be found in cut-and-paste section below)
----------------------------------------------------------------------------
-----------------------------------------------------
Kerneldump på SUN Ultra-10
----------------------------------------------------------------------------
-----------------------------------------------------

afs: Lost contact with volume location server 164.37.205.20 in cell
test.apoteket.se

afs: Lost contact with volume location server 164.37.205.20 in cell
test.apoteket.se

umount: /afs no such file or directory
The system is down.
WARM shutting down of: CB... afs... BkG... CTrunc... AFSDB... RxEvent...
RxListener... osi_NetReceive, infinite t_kspoll timed out
anic[cpu0]/thread=0x6075c4e0: osi_NetReceive, infinite t_kspoll timed out

syncing file systems... [8] 11 done
3917 static and sysmap kernel pages
31 dynamic kernel data pages
178 kernel-pageable pages
0 segkmap kernel pages
0 segvn kernel pages
0 current user process pages
4126 total pages (4126 chunks)

dumping to vp 6045cdd4, offset 983264

3614 pages left
3102 pages left
2590 pages left
2078 pages left
1566 pages left
1054 pages left
542 pages left
30 pages left
4126 total pages, dump succeeded
WARNING: interrupt level 4 not serviced
rebooting...
Resetting ...

----------------------------------------------------------------------------
-----------------------------------------------------
crash-analys av dump på SUN Ultra-10:
----------------------------------------------------------------------------
-----------------------------------------------------

# crash -d wmcore.1 -n unix.1
dumpfile = vmcore.1, namelist = unix.1, outfile = stdout
> t
STACK TRACE FOR THREAD 60806720:
locals |0 0x00000000 0x00000003 0x00000000 0x00000001
|4 0x00000000 0xfff9b500 0x00000000 0x3023d3d7
| 0x00000000
ins |0 0x00000000 0x00000000 0x60104e38 0x60444b3c | outs
|4 0x80000000 0x00000000 0xffffffff 0x00000000 |
|
read error on register set at ffffffff
> p
PROC TABLE SIZE = 4010
SLOT ST PID PPID PGID SID UID PRI NAME FLAGS
0 t 0 0 0 0 0 96 sched load sys lock
1 z 1 0 0 0 0 99 zombie load nowait
2 s 2 0 0 0 0 98 pageout load sys lock nowait
3 s 3 0 0 0 0 60 fsflush load sys lock nowait
20 s 4064 1 0 0 0 60 uadmin load nowait
40 p 276 1 0 0 0 59 afsd load sys lock nowait
> f
ADDRESS RCNT TYPE/ADDR OFFSET FLAGS
6084fef0 1 SPEC/6082fe4c 0 read write
6016e5a0 1 SPEC/6045d18c 0 read
6016e8a0 1 ?? /60733ea0 0 read write
6016fc80 1 SPEC/6045cc3c 0 read
6016fcb0 1 SPEC/6045cd4c 0 read
6016fd40 2 SPEC/6045d18c 380 write
6016fe30 1 SPEC/6045d544 0 read write
6016fe60 1 SPEC/6045d654 0 read
6016fe90 1 SPEC/6045d874 0 read
6016fec0 1 SPEC/6045d984 0 read write
6016fef0 3 SPEC/6045da94 0 read write
6016ff20 3 SPEC/6045db1c 0 read write
6016ff50 3 SPEC/6045dba4 0 read write
6016ff80 3 SPEC/6045dc2c 0 read write
6016ffb0 3 SPEC/6045dd3c 0 read write
> p -f 40
PROC TABLE SIZE = 4010
SLOT ST PID PPID PGID SID UID PRI NAME FLAGS
40 p 276 1 0 0 0 59 afsd load sys lock nowait

Session: sid: 0, ctty: -
Process Credentials: uid: 0, gid: 0, real uid: 0, real gid: 0
as: 10418c4c
wait code: 0, wait data: 0
sig: effff500 link 0
parent: 60145338 child: 0
sibling: 0 threadp: 60806720
utime: 0 stime: 0 cutime: 0 cstime: 0
trace: 0 sigmask: effff500 class: 0
lwptotal: 1 lwpcnt: 1 lwprcnt: 1
lwpblocked: -1

> ?
as kmastat pcfsnode t (trace)
b (buffer) kmausers prnode trace
base l (lck) proc thread
buf (bufhdr) lck pty ts
buffer linkblk q (quit) tsdptbl
bufhdr lwp qrun tsproc
c (callout) m (vfs) queue tty
callout major quit u (user)
class map rd (od) user
cpu mblk redirect ui (uinode)
defproc mblkusers rtdptbl uinode
defthread mode rtproc v (var)
dispq mount (vfs) rwlock var
ds mutex s (stack) vfs
f (file) mutextable search vfssw
file nfs (nfsnode) sema vnode
findaddr nfsnode size vtop
findslot nm snode ?
fs (vfssw) od stack !cmd
help p (proc) status
hment page stream
kfp pcb strstat

> lwp
LWP:
oldcontext 0 ap 0 eosys 0
lwp_regs 3023db68 qsav effff9c8
args: arg[0]: 0 arg[1]: 0 arg[2]: 0
arg[3]: 0 arg[4]: 0 arg[5]: 0
arg[6]: 0 arg[7]: 0
sysabort 0 asleep 0 cursig 0 curflt 0 curinfo 0
thread 60806720 procp 60950cc8
> s

kernel stack:
FP: 3023d300 end of stack: 3023db08

3023d300: 00000000 00000003 00000000 00000001
3023d310: 00000000 fff9b500 00000000 3023d3d7
3023d320: 00000000 00000000 60104e38 60444b3c
3023d330: 80000000 00000000 ffffffff 00000000
3023d340: 00000000 60163f40 10433b24 00001030
3023d350: 00000020 000f9fe0 00000000 0000e2d8
3023d360: 00000000 00000000 3023d390 10032b44
3023d370: 00000000 00000000 00000000 00000000
3023d380: 00000000 00000024 87000000 60098400
3023d390: 00000000 00000020 00000001 600985a0
3023d3a0: 600985b0 60439528 00000000 00020000
3023d3b0: 00000020 10435c00 60439528 60936f18
3023d3c0: 6043f0d0 00000000 3023d3f0 1001e5f4
3023d3d0: 00000000 00000020 00000000 601c3abc
3023d3e0: 100308d0 100308cc 00000000 00001861
3023d3f0: 600985e4 600985ac 6008b8e0 60098594
3023d400: 600985cc 00000000 104166fc 00000001
3023d410: 3023d430 00000011 00000000 00000000
3023d420: 00020000 00000000 60104e38 00000000
3023d430: 00000000 ffffffff ffffffff 3023d4a8
3023d440: 00000000 00000000 0000000c 60098400
3023d450: 600985e8 0001f92a 0001f92a 3023d45c
3023d460: 3023d45c 0001f92a 00000001 10441cb8
3023d470: 3023d5a2 00000016 ffffffff 00000000
3023d480: 00000003 00000000 3023d4b0 60439528
3023d490: 00000000 104d2b78 00000000 3023d570
3023d4a0: 00000000 60103d1c 00000000 dc504000
3023d4b0: 00020000 00000000 00020000 00000000
3023d4c0: 00000000 0000000a 00000000 60103ce8
3023d4d0: 00000000 600eb900 00000000 00000000
3023d4e0: 00000000 60098400 00000000 601bb880
3023d4f0: 600e7248 60098400 60103ce8 00000001
3023d500: 3023d530 601b8ec8 3023d530 601c2530
3023d510: 00000000 00000630 00000000 00000000
3023d520: 601c3260 00000000 00000001 00000630
3023d530: 00000001 00000630 3023d71c 00000001
3023d540: 00000000 600eb900 00000000 00000000
3023d550: 601bb880 00000000 601bb898 601bb8a0
3023d560: 00000000 ffffffff 3023d590 601b76e4
3023d570: 00000000 00000003 00000010 10464430
3023d580: 00000008 00000001 00000000 00000000
3023d590: 600971d0 001fa2d0 00000100 601b9d04
3023d5a0: 601b9ce0 601b9ca8 60093538 601bb884
3023d5b0: 60093538 fffffffb 601bb880 601bb8ac
3023d5c0: 00000005 fffffffd 3023d670 1001db38
3023d5d0: 00000000 00000003 00000010 00000000
3023d5e0: 10431798 6009a00c 00000000 00000002
3023d5f0: 00000000 00000000 00000001 00000000
3023d600: 00000000 00000000 00000000 00000000
3023d610: 00020000 601f0000 00000000 00000000
3023d620: 00000000 00000000 00000000 00000000
3023d630: 00000000 00000000 00000000 00000000
3023d640: 00000000 00000000 00000000 00000000
3023d650: 00000000 00000000 00000000 00000000
3023d660: 00000000 00000000 00000000 00000000
3023d670: 10413400 10418c00 00000080 00002000
3023d680: 60210000 00000000 0000000f 10406000
3023d690: 6045cdd4 0000e3aa 601f0000 00000010
3023d6a0: 00000001 00000000 3023d6d0 100a595c
3023d6b0: 00000000 3023d6d0 00000000 100a59a8
3023d6c0: 00000100 100a5a14 00000000 00000080
3023d6d0: 00001c75 601b1838 601b184c 6056e000
3023d6e0: 601b1848 0000e3aa 10433b24 00001030
3023d6f0: 00000001 000fa0d0 00000000 0000e3aa
3023d700: 00000000 00000a0f 3023d730 100a56b4
3023d710: 00000000 3023d730 00000630 00000a00
3023d720: 0000000d 00000000 00000001 00000064
3023d730: 00002000 00000007 00001030 00000001
3023d740: 00001031 02066000 00002000 601b1838
3023d750: 601b1838 601b1858 3dac033b 60936f18
3023d760: 601b1838 00000052 3023d7e0 1001d7bc
3023d770: 00000000 00000000 00001030 00001030
3023d780: 00000000 00000000 104187cc 1042f400
3023d790: 003b0000 00010000 00001030 3023d764
3023d7a0: 00000000 10418c4c 00000000 10427960
3023d7b0: 0000ff88 107b41c0 00000000 00000000
3023d7c0: 10427960 00000000 100a52b0 3023d730
3023d7d0: 00001030 00000001 00000000 601b1838
3023d7e0: 00000000 104146a8 00000000 00000000
3023d7f0: 00000001 10406000 00000000 00000000
3023d800: 00000009 3023d950 3023d7e0 00000000
3023d810: 00000000 00000001 3023d848 1001d3c8
3023d820: 3023d848 1001d370 104146b0 104146e0
3023d830: 3023d950 104146dc 00000000 00000000
3023d840: 00000009 00000000 608067ac 60950d64
3023d850: 608342cb fffffffe 00000000 04000000
3023d860: 60950da4 60950d6c 60936f18 3023d950
3023d870: 00000000 00000003 00200000 00000000
3023d880: 3023d8a8 1001d31c 00000001 00000000
3023d890: 3023d950 00000000 00000003 10418800
3023d8a0: 00000000 00000000 00000000 00008000
3023d8b0: 00000000 00008000 00008000 00020219
3023d8c0: 00000000 00000000 60936f18 ffffffff
3023d8d0: 00000004 608343f2 60834230 00000001
3023d8e0: 3023d908 608fc938 00000000 00000000
3023d8f0: 00000000 00000000 00000000 6008a398
3023d900: 00000000 6084ff10 60936f18 ffffffff
3023d910: 00000004 608343f2 6008a3b8 00000001
3023d920: 00000000 6004df84 60936f18 ffffffff
3023d930: 00000004 608343f2 6080fd70 6082dbf8
3023d940: 3023d968 608fbf7c 00000000 00000000
3023d950: ffffffff 00000004 608343f2 60834230
3023d960: 00000001 60834230 60936f18 ffffffff
3023d970: 00000002 3023d9d4 003b0000 ffffffff
3023d980: 10000000 00000001 6008a398 3023da80
3023d990: 60954044 0000000d 3023da7c 609586d0
3023d9a0: 3023da10 608fd4c8 00000000 60936f18
3023d9b0: ffffffff 00000004 608343f2 00000000
3023d9c0: 00000000 00000000 00000000 00000000
3023d9d0: 00000000 00000000 00000000 00000000
3023d9e0: 00000000 00000000 00000000 00000003
3023d9f0: 00000004 00000000 00000000 00000000
3023da00: 6008a370 6008a398 0000000d fffffd84
3023da10: 0000000d 6008a398 3023da80 60954044
3023da20: 3023da7c 104146ee 104146f0 60097874
3023da30: 6008a398 60954000 3023dafc 3023daf8
3023da40: 00000002 00004000 3023da98 608fd870
3023da50: 00000000 6008a398 3023da80 60954044
3023da60: 0000000d 3023da7c 00000000 00000000
3023da70: 00000000 00000588 00004000 00004004
3023da80: 00000000 00000000 00000000 00000000
3023da90: 00000000 60954000 6008a398 60954000
3023daa0: 3023dafc 3023daf8 3002bec0 00000000
3023dab0: 104146f0 00008ded 00000000 00000000
3023dac0: 60951988 60950cc8 104146a8 00000000
3023dad0: 3023db08 10025470 00000000 6008a398
3023dae0: 60954000 3023dafc 3023daf8 00000000
3023daf0: 00000000 00000000 00000000 00000000
3023db00: 00000000 60954000
> t
STACK TRACE FOR THREAD 60806720:
locals |0 0x00000000 0x00000003 0x00000000 0x00000001
|4 0x00000000 0xfff9b500 0x00000000 0x3023d3d7
| 0x00000000
ins |0 0x00000000 0x00000000 0x60104e38 0x60444b3c | outs
|4 0x80000000 0x00000000 0xffffffff 0x00000000 |
|
read error on register set at ffffffff
> ^D
----------------------------------------------------------------------------
-----------------------------------------------------
Kerneldump på SUN Ultra Enterprise 2
----------------------------------------------------------------------------
-----------------------------------------------------

afs: Lost contact with volume location server 164.37.205.40 in cell
test.apoteket.se

afs: Lost contact with volume location server 164.37.205.40 in cell
test.apoteket.se

SUNW,hme0: Link Down - cable problem?
BkG... CTrunc... AFSDB... RxEvent... Rxpiatnnirc..[ocipNut0e]e/vt,hirfenatd
=_0sxo6l0tam2d4o7t2
0: osi_NetReceive, infinite t_kspoll timed out

syncing file systems... [11] 7 done
4721 static and sysmap kernel pages
49 dynamic kernel data pages
189 kernel-pageable pages
0 segkmap kernel pages
0 segvn kernel pages
0 current user process pages
4959 total pages (4959 chunks)

dumping to vp 60262aa4, offset 447632

4447 pages left
3935 pages left
3423 pages left
2911 pages left
2399 pages left
1887 pages left
1375 pages left
863 pages left
351 pages left
4959 total pages, dump succeeded
rebooting...
Resetting ...

Sun Ultra 2 UPA/SBus (2 X UltraSPARC-II 296MHz), Keyboard Present
OpenBoot 3.19, 768 MB memory installed, Serial #11677347.
Ethernet address 8:0:20:b2:2e:a3, Host ID: 80b22ea3.