[OpenAFS] Kernel NULL pointer dereference
Ken Elkabany
Ken@Elkabany.com
Thu, 19 Apr 2012 18:55:08 -0700
--f46d043be1d083e47704be12955f
Content-Type: text/plain; charset=ISO-8859-1
We have 2 OpenAFS servers running 1.4.14. We have many clients that we just
switched over to 1.6.1pre1. Starting earlier today, we started getting NULL
pointer dereferences, which has been completely hosing the clients. The
client machines hang on any call that deals with AFS, whether it's "ls /",
"ls /afs", "klist", etc... A "vos changeaddr" was done earlier today,
whereby a large collection (4000) of volumes were mistakenly assigned to
another server. These were corrected with "vos syncvldb" followed by "vos
syncserv". I mention it here, as it's the only thing we've done to the AFS
cluster today.
Here's what we found in the syslog:
Apr 20 01:30:43 SERVER kernel: [12861236.027818] BUG: unable to handle
kernel NULL pointer dereference at 0000000000000028
Apr 20 01:30:43 SERVER kernel: [12861236.027836] IP: [<ffffffffa0048087>]
afs_Conn+0x1e7/0x260 [openafs]
Apr 20 01:30:43 SERVER kernel: [12861236.027868] PGD 0
Apr 20 01:30:43 SERVER kernel: [12861236.027874] Oops: 0000 [#1] SMP
Apr 20 01:30:43 SERVER kernel: [12861236.027882] CPU 6
Apr 20 01:30:43 SERVER kernel: [12861236.027885] Modules linked in:
openafs(P) isofs acpiphp
Apr 20 01:30:43 SERVER kernel: [12861236.027897]
Apr 20 01:30:43 SERVER kernel: [12861236.027902] Pid: 1568, comm: apache2
Tainted: P O 3.2.0-23-virtual #36-Ubuntu
Apr 20 01:30:43 SERVER kernel: [12861236.027912] RIP:
e030:[<ffffffffa0048087>] [<ffffffffa0048087>] afs_Conn+0x1e7/0x260
[openafs]
Apr 20 01:30:43 SERVER kernel: [12861236.027936] RSP: e02b:ffff88017f417808
EFLAGS: 00010282
Apr 20 01:30:43 SERVER kernel: [12861236.027942] RAX: ffffc9000188dbe0 RBX:
0000000000000000 RCX: 000000000000581b
Apr 20 01:30:43 SERVER kernel: [12861236.027950] RDX: ffff8801b112a000 RSI:
0000000000000001 RDI: ffff88017f761680
Apr 20 01:30:43 SERVER kernel: [12861236.027957] RBP: ffff88017f417858 R08:
0000000000000000 R09: 0000000000000000
Apr 20 01:30:43 SERVER kernel: [12861236.027964] R10: 0000000000000002 R11:
0000000000000000 R12: ffff880184756f48
Apr 20 01:30:43 SERVER kernel: [12861236.027971] R13: ffff88017f417a20 R14:
0000000000000004 R15: ffff88017f4178f0
Apr 20 01:30:43 SERVER kernel: [12861236.027983] FS:
00007f1f6ae2f700(0000) GS:ffff8801bff73000(0000) knlGS:0000000000000000
Apr 20 01:30:43 SERVER kernel: [12861236.027991] CS: e033 DS: 0000 ES:
0000 CR0: 000000008005003b
Apr 20 01:30:43 SERVER kernel: [12861236.027998] CR2: 0000000000000028 CR3:
0000000181465000 CR4: 0000000000002660
Apr 20 01:30:43 SERVER kernel: [12861236.028006] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Apr 20 01:30:43 SERVER kernel: [12861236.028013] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Apr 20 01:30:43 SERVER kernel: [12861236.028021] Process apache2 (pid:
1568, threadinfo ffff88017f416000, task ffff88017f41adc0)
Apr 20 01:30:43 SERVER kernel: [12861236.028028] Stack:
Apr 20 01:30:43 SERVER kernel: [12861236.028032] 000000004e2a6741
0000000000000000 0000000000000000 000000004f90bc43
Apr 20 01:30:43 SERVER kernel: [12861236.028046] 000000000001584a
ffff880184756cc0 ffff88017f41adc0 ffff88017f417a20
Apr 20 01:30:43 SERVER kernel: [12861236.028059] ffff880184756f48
ffff880184756cc0 ffff88017f417928 ffffffffa0068658
Apr 20 01:30:43 SERVER kernel: [12861236.028072] Call Trace:
Apr 20 01:30:43 SERVER kernel: [12861236.028092] [<ffffffffa0068658>]
afs_FetchStatus+0x58/0x450 [openafs]
Apr 20 01:30:43 SERVER kernel: [12861236.028113] [<ffffffffa004672b>] ?
afs_GetCellStale+0x3b/0x60 [openafs]
Apr 20 01:30:43 SERVER kernel: [12861236.028134] [<ffffffffa0046a25>] ?
afs_IsPrimaryCell+0x25/0x40 [openafs]
Apr 20 01:30:43 SERVER kernel: [12861236.028157] [<ffffffffa0082b80>] ?
afs_GetVolume+0x40/0x1d0 [openafs]
Apr 20 01:30:43 SERVER kernel: [12861236.028179] [<ffffffffa006ae8d>]
afs_GetVCache+0x26d/0x5d0 [openafs]
Apr 20 01:30:43 SERVER kernel: [12861236.028200] [<ffffffffa006b343>]
afs_VerifyVCache2+0x153/0x200 [openafs]
Apr 20 01:30:43 SERVER kernel: [12861236.028222] [<ffffffffa006ccec>]
afs_getattr+0x29c/0x350 [openafs]
Apr 20 01:30:43 SERVER kernel: [12861236.028242] [<ffffffffa009340f>]
afs_linux_dentry_revalidate+0x39f/0x470 [openafs]
Apr 20 01:30:43 SERVER kernel: [12861236.028265] [<ffffffffa006bf43>] ?
afs_AccessOK+0x113/0x1e0 [openafs]
Apr 20 01:30:43 SERVER kernel: [12861236.028279] [<ffffffff816552de>] ?
_raw_spin_lock+0xe/0x20
Apr 20 01:30:43 SERVER kernel: [12861236.028290] [<ffffffff811818eb>]
do_lookup+0x18b/0x310
Apr 20 01:30:43 SERVER kernel: [12861236.028298] [<ffffffff8129885c>] ?
security_inode_permission+0x1c/0x30
Apr 20 01:30:43 SERVER kernel: [12861236.028306] [<ffffffff81182268>]
link_path_walk+0x138/0x870
Apr 20 01:30:43 SERVER kernel: [12861236.028313] [<ffffffff811834ad>] ?
path_init+0x2ed/0x3c0
Apr 20 01:30:43 SERVER kernel: [12861236.028319] [<ffffffff811835d8>]
path_lookupat+0x58/0x750
Apr 20 01:30:43 SERVER kernel: [12861236.028339] [<ffffffffa006cb3c>] ?
afs_getattr+0xec/0x350 [openafs]
Apr 20 01:30:43 SERVER kernel: [12861236.028348] [<ffffffff810067be>] ?
xen_pmd_val+0xe/0x10
Apr 20 01:30:43 SERVER kernel: [12861236.028355] [<ffffffff81183d01>]
do_path_lookup+0x31/0xc0
Apr 20 01:30:43 SERVER kernel: [12861236.028362] [<ffffffff81184809>]
user_path_at_empty+0x59/0xa0
Apr 20 01:30:43 SERVER kernel: [12861236.028369] [<ffffffff8100aa32>] ?
check_events+0x12/0x20
Apr 20 01:30:43 SERVER kernel: [12861236.028377] [<ffffffff8100a25d>] ?
xen_force_evtchn_callback+0xd/0x10
Apr 20 01:30:43 SERVER kernel: [12861236.028384] [<ffffffff81184861>]
user_path_at+0x11/0x20
Apr 20 01:30:43 SERVER kernel: [12861236.028391] [<ffffffff8117995a>]
vfs_fstatat+0x3a/0x70
Apr 20 01:30:43 SERVER kernel: [12861236.028398] [<ffffffff8100aa1f>] ?
xen_restore_fl_direct_reloc+0x4/0x4
Apr 20 01:30:43 SERVER kernel: [12861236.028405] [<ffffffff8100465d>] ?
xen_clts+0x8d/0x190
Apr 20 01:30:43 SERVER kernel: [12861236.028412] [<ffffffff811799ae>]
vfs_lstat+0x1e/0x20
Apr 20 01:30:43 SERVER kernel: [12861236.028418] [<ffffffff81179b4a>]
sys_newlstat+0x1a/0x40
Apr 20 01:30:43 SERVER kernel: [12861236.028427] [<ffffffff810146e1>] ?
math_state_restore+0x51/0x80
Apr 20 01:30:43 SERVER kernel: [12861236.028435] [<ffffffff816562fe>] ?
do_device_not_available+0xe/0x10
Apr 20 01:30:43 SERVER kernel: [12861236.028445] [<ffffffff8165f8cb>] ?
device_not_available+0x1b/0x20
Apr 20 01:30:43 SERVER kernel: [12861236.028452] [<ffffffff8165d8c2>]
system_call_fastpath+0x16/0x1b
Apr 20 01:30:43 SERVER kernel: [12861236.028458] Code: 89 ef 48 89 45 c8 e8
39 c4 01 00 48 8b 45 c8 48 83 c4 28 5b 41 5c 41 5d 41 5e 41 5f 5d c3 48 85
ff 0f 84 95 fe ff ff 48 8b 5f 58 <f6> 43 28 20 0f 85 87 fe ff ff
41 80 7d 12 00 7e 29 41 80 7d 13
Apr 20 01:30:43 SERVER kernel: [12861236.028543] RIP [<ffffffffa0048087>]
afs_Conn+0x1e7/0x260 [openafs]
Apr 20 01:30:43 SERVER kernel: [12861236.028563] RSP <ffff88017f417808>
Apr 20 01:30:43 SERVER kernel: [12861236.028568] CR2: 0000000000000028
--f46d043be1d083e47704be12955f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div style><div><font color=3D"#222222" face=3D"arial, sans-serif">We have =
2 OpenAFS servers running 1.4.14. We have many clients that we just switche=
d over to 1.6.1pre1. Starting earlier today, we started getting NULL pointe=
r dereferences, which has been completely hosing the clients. The client ma=
chines hang on any call that deals with AFS, whether it's "ls /&qu=
ot;, "ls /afs", "klist", etc... A "vos changeaddr&=
quot; was done earlier today, whereby a large collection (4000) of volumes =
were mistakenly assigned to another server. These were corrected with "=
;vos syncvldb" followed by "vos syncserv". I mention it here=
, as it's the only thing we've done to the AFS cluster today.</font=
></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif"><br></font></div><d=
iv><span style=3D"color:rgb(34,34,34);font-family:arial,sans-serif">Here=
9;s what we found in the syslog:</span></div><div><font color=3D"#222222" f=
ace=3D"arial, sans-serif"><br>
</font></div><div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20=
01:30:43 SERVER kernel: [12861236.027818] BUG: unable to handle kernel NUL=
L pointer dereference at 0000000000000028</font></div><div><font color=3D"#=
222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236=
.027836] IP: [<ffffffffa0048087>] afs_Conn+0x1e7/0x260 [openafs]</fon=
t></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.027868] PGD 0=A0</font></div><div><font color=3D"#222=
222" face=3D"arial, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.02=
7874] Oops: 0000 [#1] SMP=A0</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.027882] CPU 6=A0</font></div><div><font color=3D"#222=
222" face=3D"arial, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.02=
7885] Modules linked in: openafs(P) isofs acpiphp</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.027897]=A0</font></div><div><font color=3D"#222222" f=
ace=3D"arial, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.027902] =
Pid: 1568, comm: apache2 Tainted: P =A0 =A0 =A0 =A0 =A0 O 3.2.0-23-virtual =
#36-Ubuntu =A0</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.027912] RIP: e030:[<ffffffffa0048087>] =A0[<=
ffffffffa0048087>] afs_Conn+0x1e7/0x260 [openafs]</font></div><div><font=
color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SERVER kernel=
: [12861236.027936] RSP: e02b:ffff88017f417808 =A0EFLAGS: 00010282</font></=
div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.027942] RAX: ffffc9000188dbe0 RBX: 0000000000000000 R=
CX: 000000000000581b</font></div><div><font color=3D"#222222" face=3D"arial=
, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.027950] RDX: ffff880=
1b112a000 RSI: 0000000000000001 RDI: ffff88017f761680</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.027957] RBP: ffff88017f417858 R08: 0000000000000000 R=
09: 0000000000000000</font></div><div><font color=3D"#222222" face=3D"arial=
, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.027964] R10: 0000000=
000000002 R11: 0000000000000000 R12: ffff880184756f48</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.027971] R13: ffff88017f417a20 R14: 0000000000000004 R=
15: ffff88017f4178f0</font></div><div><font color=3D"#222222" face=3D"arial=
, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.027983] FS: =A000007=
f1f6ae2f700(0000) GS:ffff8801bff73000(0000) knlGS:0000000000000000</font></=
div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.027991] CS: =A0e033 DS: 0000 ES: 0000 CR0: 0000000080=
05003b</font></div><div><font color=3D"#222222" face=3D"arial, sans-serif">=
Apr 20 01:30:43 SERVER kernel: [12861236.027998] CR2: 0000000000000028 CR3:=
0000000181465000 CR4: 0000000000002660</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028006] DR0: 0000000000000000 DR1: 0000000000000000 D=
R2: 0000000000000000</font></div><div><font color=3D"#222222" face=3D"arial=
, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.028013] DR3: 0000000=
000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028021] Process apache2 (pid: 1568, threadinfo ffff88=
017f416000, task ffff88017f41adc0)</font></div><div><font color=3D"#222222"=
face=3D"arial, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.028028=
] Stack:</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028032] =A0000000004e2a6741 0000000000000000 00000000=
00000000 000000004f90bc43</font></div><div><font color=3D"#222222" face=3D"=
arial, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.028046] =A00000=
00000001584a ffff880184756cc0 ffff88017f41adc0 ffff88017f417a20</font></div=
>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028059] =A0ffff880184756f48 ffff880184756cc0 ffff8801=
7f417928 ffffffffa0068658</font></div><div><font color=3D"#222222" face=3D"=
arial, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.028072] Call Tr=
ace:</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028092] =A0[<ffffffffa0068658>] afs_FetchStatus=
+0x58/0x450 [openafs]</font></div><div><font color=3D"#222222" face=3D"aria=
l, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.028113] =A0[<fff=
fffffa004672b>] ? afs_GetCellStale+0x3b/0x60 [openafs]</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028134] =A0[<ffffffffa0046a25>] ? afs_IsPrimary=
Cell+0x25/0x40 [openafs]</font></div><div><font color=3D"#222222" face=3D"a=
rial, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.028157] =A0[<=
ffffffffa0082b80>] ? afs_GetVolume+0x40/0x1d0 [openafs]</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028179] =A0[<ffffffffa006ae8d>] afs_GetVCache+0=
x26d/0x5d0 [openafs]</font></div><div><font color=3D"#222222" face=3D"arial=
, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.028200] =A0[<ffff=
ffffa006b343>] afs_VerifyVCache2+0x153/0x200 [openafs]</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028222] =A0[<ffffffffa006ccec>] afs_getattr+0x2=
9c/0x350 [openafs]</font></div><div><font color=3D"#222222" face=3D"arial, =
sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.028242] =A0[<ffffff=
ffa009340f>] afs_linux_dentry_revalidate+0x39f/0x470 [openafs]</font></d=
iv>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028265] =A0[<ffffffffa006bf43>] ? afs_AccessOK+=
0x113/0x1e0 [openafs]</font></div><div><font color=3D"#222222" face=3D"aria=
l, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.028279] =A0[<fff=
fffff816552de>] ? _raw_spin_lock+0xe/0x20</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028290] =A0[<ffffffff811818eb>] do_lookup+0x18b=
/0x310</font></div><div><font color=3D"#222222" face=3D"arial, sans-serif">=
Apr 20 01:30:43 SERVER kernel: [12861236.028298] =A0[<ffffffff8129885c&g=
t;] ? security_inode_permission+0x1c/0x30</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028306] =A0[<ffffffff81182268>] link_path_walk+=
0x138/0x870</font></div><div><font color=3D"#222222" face=3D"arial, sans-se=
rif">Apr 20 01:30:43 SERVER kernel: [12861236.028313] =A0[<ffffffff81183=
4ad>] ? path_init+0x2ed/0x3c0</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028319] =A0[<ffffffff811835d8>] path_lookupat+0=
x58/0x750</font></div><div><font color=3D"#222222" face=3D"arial, sans-seri=
f">Apr 20 01:30:43 SERVER kernel: [12861236.028339] =A0[<ffffffffa006cb3=
c>] ? afs_getattr+0xec/0x350 [openafs]</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028348] =A0[<ffffffff810067be>] ? xen_pmd_val+0=
xe/0x10</font></div><div><font color=3D"#222222" face=3D"arial, sans-serif"=
>Apr 20 01:30:43 SERVER kernel: [12861236.028355] =A0[<ffffffff81183d01&=
gt;] do_path_lookup+0x31/0xc0</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028362] =A0[<ffffffff81184809>] user_path_at_em=
pty+0x59/0xa0</font></div><div><font color=3D"#222222" face=3D"arial, sans-=
serif">Apr 20 01:30:43 SERVER kernel: [12861236.028369] =A0[<ffffffff810=
0aa32>] ? check_events+0x12/0x20</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028377] =A0[<ffffffff8100a25d>] ? xen_force_evt=
chn_callback+0xd/0x10</font></div><div><font color=3D"#222222" face=3D"aria=
l, sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.028384] =A0[<fff=
fffff81184861>] user_path_at+0x11/0x20</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028391] =A0[<ffffffff8117995a>] vfs_fstatat+0x3=
a/0x70</font></div><div><font color=3D"#222222" face=3D"arial, sans-serif">=
Apr 20 01:30:43 SERVER kernel: [12861236.028398] =A0[<ffffffff8100aa1f&g=
t;] ? xen_restore_fl_direct_reloc+0x4/0x4</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028405] =A0[<ffffffff8100465d>] ? xen_clts+0x8d=
/0x190</font></div><div><font color=3D"#222222" face=3D"arial, sans-serif">=
Apr 20 01:30:43 SERVER kernel: [12861236.028412] =A0[<ffffffff811799ae&g=
t;] vfs_lstat+0x1e/0x20</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028418] =A0[<ffffffff81179b4a>] sys_newlstat+0x=
1a/0x40</font></div><div><font color=3D"#222222" face=3D"arial, sans-serif"=
>Apr 20 01:30:43 SERVER kernel: [12861236.028427] =A0[<ffffffff810146e1&=
gt;] ? math_state_restore+0x51/0x80</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028435] =A0[<ffffffff816562fe>] ? do_device_not=
_available+0xe/0x10</font></div><div><font color=3D"#222222" face=3D"arial,=
sans-serif">Apr 20 01:30:43 SERVER kernel: [12861236.028445] =A0[<fffff=
fff8165f8cb>] ? device_not_available+0x1b/0x20</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028452] =A0[<ffffffff8165d8c2>] system_call_fas=
tpath+0x16/0x1b</font></div><div><font color=3D"#222222" face=3D"arial, san=
s-serif">Apr 20 01:30:43 SERVER kernel: [12861236.028458] Code: 89 ef 48 89=
45 c8 e8 39 c4 01 00 48 8b 45 c8 48 83 c4 28 5b 41 5c 41 5d 41 5e 41 5f 5d=
c3 48 85 ff 0f 84 95 fe ff ff 48 8b 5f 58 <f6> 43 28 20 0f 85 87 fe =
ff ff=A0</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">41 80 7d 12 00 7e 2=
9 41 80 7d 13=A0</font></div><div><font color=3D"#222222" face=3D"arial, sa=
ns-serif">Apr 20 01:30:43 SERVER kernel: [12861236.028543] RIP =A0[<ffff=
ffffa0048087>] afs_Conn+0x1e7/0x260 [openafs]</font></div>
<div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SER=
VER kernel: [12861236.028563] =A0RSP <ffff88017f417808></font></div><=
div><font color=3D"#222222" face=3D"arial, sans-serif">Apr 20 01:30:43 SERV=
ER kernel: [12861236.028568] CR2: 0000000000000028</font></div>
</div>
--f46d043be1d083e47704be12955f--