[OpenAFS-devel] Latest build kernel panic on AIX 6.1 6100-09-12
Ben Huntsman
ben@huntsmans.net
Tue, 16 Aug 2022 06:50:50 +0000
--_000_MWHPR0701MB367403D583B4D648B36695EDA76B9MWHPR0701MB3674_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hi guys-
Ok, I got the build done, but as predicted, it kernel panics the system:
# /opt/openafs/sbin/afsd -stat 300 -dcache 100 -daemons 2 -volumes 50
afs: Binding rx to 0.0.0.0:7001
... and then it panics. Before running afsd, I loaded the export and kerne=
l module, and those went just fine. The panic only occurs when starting af=
sd.
I configured a dump device and did some basic looking. Here's the stack tr=
ace from the dump:
CRASH INFORMATION:
CPU 0 CSA F00000002FF47600 at time of crash, error code for LEDs: 30000000
pvthread+01DB00 STACK:
[0000E864]___memset64+00005C ()
[F1000000C04C1AF4]rxevent_alloc+000094 ()
[F1000000C04C2058]rxevent_Post+000038 (F00000002FF46D28, F00000002FF46D20,
F1000000C0592ED8, 0000000000000000, 0000000000000000, 0000000000000000)
[F1000000C04AE890]rxi_ReapConnections+000630 (0000000000000000, 00000000000=
00000,
0000000000000000, 0000000000000000)
[F1000000C04AE9E0]rx_StartServer+0000E0 (0000000000000000)
[F1000000C0536E78]afs_ResourceInit+0002D8 (??)
[F1000000C0542A30]afs_InitSetup+0001B0 (0000019000000190)
[F1000000C0543160]afs_syscall_call+000120 (0000000000000000, 00000000000001=
90,
0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000)
[F1000000C0557D5C]syscall+0000DC (0000001C0000001C, 0000000000000000,
0000019000000190, 0000000000000000, 0000000000000000, 0000000000000000,
0000000000000000)
[00014D70].hkey_legacy_gate+00004C ()
[00003888]mfspurr_sc_flih01+0000E4 ()
[kdb_get_virtual_memory] no real storage @ FFFFFFFF3FFFE60
[kdb_read_mem] no real storage @ FFFFFFFFFFF6150
And here's the mst:
Machine State Save Area
iar : 000000000000E864 msr : 8000000000009032 cr : 24224244
lr : F1000000C04C1AF8 ctr : 0000000000000000 xer : 00000008
mq : DEADBEEF asr : FFFFFFFFFFFFFFFF amr : 0008C00000000000
r0 : FFFFFFFFFFFFFFFF r1 : F00000002FF46B70 r2 : F1000000C05945A0
r3 : 00000000DEADBEEF r4 : 0000000000000068 r5 : 0000000000000000
r6 : 0000000000000000 r7 : 0000000000000000 r8 : 0000000000000000
r9 : 0000000000000000 r10 : 00000000DEADBEEF r11 : 00000000DEADBF57
r12 : 0000000000000007 r13 : F1000A03E05B7000 r14 : 0000000000000009
r15 : 000000002FF22CAC r16 : 000000002FF22CD4 r17 : 00000000DEADBEEF
r18 : 00000000DEADBEEF r19 : 00000000F135E1D8 r20 : 00000000DEADBEEF
r21 : 00000000DEADBEEF r22 : 0000000020000444 r23 : 00000000DEADBEEF
r24 : 00000000DEADBEEF r25 : 0000000020020CD8 r26 : 0000000020022F08
r27 : 000000002FF1F5A0 r28 : 0000000000000000 r29 : 0000000020022F08
r30 : F1000000C05874D8 r31 : F1000000C0597EC8
prev 0000000000000000 stackfix 0000000000000000 int_ticks 0000
cfar 00000000001FCC6C
kjmpbuf 0000000000000000 excbranch 0000000000000000 no_pfault 00
intpri 0B backt 00 flags 00
hw_fru_id 00000000 hw_cpu_id 00000000
fpscr 0000000000000000 fpscrx 00000000 fpowner 01
fpeu 01 fpinfo 00 alloc F000
o_iar 000000000000E864 o_toc F1000000C05945A0
o_arg1 00000000DEADBEEF o_vaddr 00000000DEADBEEF
krlockp 0000000000000000 rmgrwa F1000816B0035E20
amrstackhigh F00000002FFCCFF0 amrstacklow F00000002FFCC000
amrstackcur F00000002FFCCFE0 amrstackfix 0000000000000000
kstackhigh 0000000000000000 kstacksize 00000000
frrstart 700DFEED00000000 frrend 700DFEED00000000
frrcur 700DFEED00000000 frrstatic 0000 kjmpfrroff 0000
frrovcnt 0000 frrbarrcnt 0000 frrmask 00 callrmgr 00
Except :
excp_type 00000086 EXCEPT_PROT
orgea 00000000DEADBEEF dsisr 000000000A000000 bit set: DSISR_PROT DSISR_S=
T
vmh 0000000009000D90 curea 00000000DEADBEEF pftyp 0000000000000106
I haven't done a ton of kernel debugging, but DSISR_PROT DSISR_ST seems to =
mean that there was a protection exception.
Does anything jump out to anyone? I will try re-compiling with --enable-de=
bug and --enable-debug-kernel and see if that provides any more info.
So would this be more of a problem with the kernel module, or with afsd?
Thanks!
-Ben
--_000_MWHPR0701MB367403D583B4D648B36695EDA76B9MWHPR0701MB3674_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;margin-bo=
ttom:0;} </style>
</head>
<body dir=3D"ltr">
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
Hi guys-</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
Ok, I got the build done, but as predicted, it kernel panics t=
he system:</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
# /opt/openafs/sbin/afsd -stat 300 -dcache 100 -daemons 2 -volumes 50
<div>afs: Binding rx to 0.0.0.0:7001</div>
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
... and then it panics. Before running afsd, I loaded the export and =
kernel module, and those went just fine. The panic only occurs when s=
tarting afsd.</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
I configured a dump device and did some basic looking. Here's the sta=
ck trace from the dump:</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
CRASH INFORMATION:
<div>CPU 0 CSA F00000002FF47600 at time of crash, error code for LEDs: 3000=
0000</div>
<div>pvthread+01DB00 STACK:</div>
<div>[0000E864]___memset64+00005C ()</div>
<div>[F1000000C04C1AF4]rxevent_alloc+000094 ()</div>
<div>[F1000000C04C2058]rxevent_Post+000038 (F00000002FF46D28, F00000002FF46=
D20,</div>
<div> F1000000C0592ED8, 0000000000000000, 0000000000000000, 000=
0000000000000)</div>
<div>[F1000000C04AE890]rxi_ReapConnections+000630 (0000000000000000, 000000=
0000000000,</div>
<div> 0000000000000000, 0000000000000000)</div>
<div>[F1000000C04AE9E0]rx_StartServer+0000E0 (0000000000000000)</div>
<div>[F1000000C0536E78]afs_ResourceInit+0002D8 (??)</div>
<div>[F1000000C0542A30]afs_InitSetup+0001B0 (0000019000000190)</div>
<div>[F1000000C0543160]afs_syscall_call+000120 (0000000000000000, 000000000=
0000190,</div>
<div> 0000000000000000, 0000000000000000, 0000000000000000, 000=
0000000000000)</div>
<div>[F1000000C0557D5C]syscall+0000DC (0000001C0000001C, 0000000000000000,<=
/div>
<div> 0000019000000190, 0000000000000000, 0000000000000000, 000=
0000000000000,</div>
<div> 0000000000000000)</div>
<div>[00014D70].hkey_legacy_gate+00004C ()</div>
<div>[00003888]mfspurr_sc_flih01+0000E4 ()</div>
<div>[kdb_get_virtual_memory] no real storage @ FFFFFFFF3FFFE60</div>
<div>[kdb_read_mem] no real storage @ FFFFFFFFFFF6150</div>
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
And here's the mst:</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
Machine State Save Area
<div>iar : 000000000000E864 msr : 8000000000009032 &nbs=
p;cr : 24224244</div>
<div>lr : F1000000C04C1AF8 ctr : 0000000000000000=
xer : 00000008</div>
<div>mq : DEADBEEF asr : FFFFFFFFFFFFFFFF a=
mr : 0008C00000000000</div>
<div>r0 : FFFFFFFFFFFFFFFF r1 : F00000002FF46B70 r2=
: F1000000C05945A0</div>
<div>r3 : 00000000DEADBEEF r4 : 0000000000000068 r5=
: 0000000000000000</div>
<div>r6 : 0000000000000000 r7 : 0000000000000000 r8=
: 0000000000000000</div>
<div>r9 : 0000000000000000 r10 : 00000000DEADBEEF r11 : 0=
0000000DEADBF57</div>
<div>r12 : 0000000000000007 r13 : F1000A03E05B7000 r14 : 000000=
0000000009</div>
<div>r15 : 000000002FF22CAC r16 : 000000002FF22CD4 r17 : 000000=
00DEADBEEF</div>
<div>r18 : 00000000DEADBEEF r19 : 00000000F135E1D8 r20 : 000000=
00DEADBEEF</div>
<div>r21 : 00000000DEADBEEF r22 : 0000000020000444 r23 : 000000=
00DEADBEEF</div>
<div>r24 : 00000000DEADBEEF r25 : 0000000020020CD8 r26 : 000000=
0020022F08</div>
<div>r27 : 000000002FF1F5A0 r28 : 0000000000000000 r29 : 000000=
0020022F08</div>
<div>r30 : F1000000C05874D8 r31 : F1000000C0597EC8</div>
<div><br>
</div>
<div>prev 0000000000000000 stackfix 0000000000000=
000 int_ticks 0000</div>
<div>cfar 00000000001FCC6C</div>
<div>kjmpbuf 0000000000000000 excbranch 0000000000000000 no_pfault 0=
0</div>
<div>intpri 0B  =
; backt 00 f=
lags 00</div>
<div>hw_fru_id 00000000 hw_cpu_id 00000000</div=
>
<div>fpscr 0000000000000000 fpscrx 00000000 &nbs=
p; fpowner 01</div>
<div>fpeu 01 =
fpinfo 00 &nb=
sp; alloc F000</div>
<div>o_iar 000000000000E864 o_toc F1000000C0594=
5A0</div>
<div>o_arg1 00000000DEADBEEF o_vaddr 00000000DEADBEEF</=
div>
<div>krlockp 0000000000000000 rmgrwa F1000816B0035E20</=
div>
<div>amrstackhigh F00000002FFCCFF0 amrstacklow F00000002FFCC00=
0</div>
<div>amrstackcur F00000002FFCCFE0 amrstackfix 000000000000000=
0</div>
<div>kstackhigh 0000000000000000 kstacksize 00000=
000</div>
<div>frrstart 700DFEED00000000 frrend 700DFEED00000000</=
div>
<div>frrcur 700DFEED00000000 frrstatic 0000 kjmpfrroff 0000</d=
iv>
<div>frrovcnt 0000 frrbarrcnt 0000 frrmask 00 callrmgr 00</div>
<div>Except :</div>
<div>excp_type 00000086 EXCEPT_PROT</div>
<div> orgea 00000000DEADBEEF dsisr 000000000A000000 bit set: DSI=
SR_PROT DSISR_ST</div>
<div> vmh 0000000009000D90 curea 00000000DEADBEEF pftyp 0000000=
000000106</div>
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
I haven't done a ton of kernel debugging, but DSISR_PROT DSISR_ST seems to =
mean that there was a protection exception.</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
Does anything jump out to anyone? I will try re-compiling with --enab=
le-debug and --enable-debug-kernel and see if that provides any more info.<=
/div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
So would this be more of a problem with the kernel module, or with afsd?</d=
iv>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
Thanks!</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
<br>
</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
-Ben</div>
<div style=3D"font-family: Calibri, Arial, Helvetica, sans-serif; font-size=
: 12pt; color: rgb(0, 0, 0);" class=3D"elementToProof">
<br>
</div>
</body>
</html>
--_000_MWHPR0701MB367403D583B4D648B36695EDA76B9MWHPR0701MB3674_--