Fwd: [OpenAFS] /afs area is hanging
Ted Creedon
tcreedon@easystreet.net
Thu, 7 May 2009 14:58:10 -0700
--001636e909a39760230469599b58
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
---------- Forwarded message ----------
From: Ted Creedon <tcreedon@easystreet.net>
Date: Thu, May 7, 2009 at 2:57 PM
Subject: Re: [OpenAFS] /afs area is hanging
To: Mark Henry <mark.henry@infoprint.com>
Use select cut and paste typical portions of dmesg and /var/log/messages
I fixed my problem by recreating all the principals using "kadmin.local -e
des-cbc-crc:normal"
which forces all the keys to be single des.
Typical for 3 servers all of which hung after "ls /afs"
I niced affsd to get enough keyboard control to see that klogd and syslog-ng
are using all the cycles filling /var/log/messages with rx carps.
I think that a massive error caused by mis-keying should not hang windows
and linux clients...
A key check diagnostic would certainly help.
1.5.59 win and 1.4.10 linux on 3 suse 10.2 and 11.1 server boxes, one is
dual homed
Seems to work fine now.
On Thu, May 7, 2009 at 1:51 PM, Mark Henry <mark.henry@infoprint.com> wrote:
>
> I ran the 'echo t' command recommended below once afs hung again. It
> definitely put some output in dmesg. The only D states that were listed
> were bash sessions (I think). It looks like they were sessions that I
> opened after the user told me that afs was hung again. I tried to attach
> the dmesg output and cmdebug output but the email was rejected because the
> log files were way too big. Any ideas of what to try next? Or is there
> anything in particular that I should look at in the cmdebug or dmesg output.
> Thanks,
>
> Mark Henry
>
>
>
> *Felix Frank <Felix.Frank@Desy.de>*
>
> 05/04/2009 11:46 PM
> To
> Mark Henry <mark.henry@infoprint.com> cc
> openafs-info@openafs.org Subject
> Re: [OpenAFS] /afs area is hanging
>
>
>
>
> On Mon, 4 May 2009, Mark Henry wrote:
>
> > I tried the -fakestat-all option and it did not work.
>
> Weird.
>
> > I have searched /var/log/messages. I have checked the config files.
> User
> > authentication works fine (even when the system is hanging). If I run
> the
> > command 'ls -l /afs' that window is hung (or any other command that
> > references afs). If an afs user logs in when the system is in a bad
> state
> > the session immediately hangs because it can't cd to the afs home dir. I
> > don't know what to do other than reboot. Can someone tell me what else
> to
> > try to find out why this system is hanging? Thanks,
>
> You can find out just which call gets stuck by issuing an
> 'echo t >>/proc/sysrq-trigger'. Call traces for all processes can then be
> found in dmesg. The broken processes are probably in a D state.
>
> This will probably not identify the root cause, but may give a clue about
> what's going on.
>
> HTH
> - Felix
>
>
>
> _____________________________________________________________________________
> "This message and any attachments are solely for the intended recipient and
> may contain confidential or privileged information. If you are not the
> intended recipient, any disclosure, copying, use, or distribution of the
> information included in this message and any attachments is prohibited. If
> you have received this communication in error, please notify us by reply
> e-mail and immediately and permanently delete this message and any
> attachments. Thank you."
> _____________________________________________________________________________
>
--001636e909a39760230469599b58
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<br><br><div class=3D"gmail_quote">---------- Forwarded message ----------<=
br>From: <b class=3D"gmail_sendername">Ted Creedon</b> <span dir=3D"ltr">&l=
t;<a href=3D"mailto:tcreedon@easystreet.net">tcreedon@easystreet.net</a>>=
;</span><br>
Date: Thu, May 7, 2009 at 2:57 PM<br>Subject: Re: [OpenAFS] /afs area is ha=
nging<br>To: Mark Henry <<a href=3D"mailto:mark.henry@infoprint.com">mar=
k.henry@infoprint.com</a>><br><br><br>Use select cut and paste typical p=
ortions=A0 of dmesg and /var/log/messages<br>
<br>I fixed my problem by recreating all the principals using "kadmin.=
local -e des-cbc-crc:normal"
<br>which forces all the keys to be single des.<br><br>Typical for 3 server=
s all of which hung after "ls /afs"<br><br><div class=3D"gmail_qu=
ote">I niced affsd to get enough keyboard control to see that klogd and sys=
log-ng are using all the cycles filling /var/log/messages with rx carps.<br=
>
<br>I think that a massive error caused by mis-keying should not hang windo=
ws and linux clients...<br><br>A key check diagnostic would certainly help.=
<br><br>1.5.59 win and 1.4.10 linux on 3 suse 10.2 and 11.1 server boxes, o=
ne is dual homed<br>
<br>Seems to work fine now.<div><div></div><div class=3D"h5"><br><br>On Thu=
, May 7, 2009 at 1:51 PM, Mark Henry <span dir=3D"ltr"><<a href=3D"mailt=
o:mark.henry@infoprint.com" target=3D"_blank">mark.henry@infoprint.com</a>&=
gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br><font face=3D"sans-serif" size=3D"2">I ran the 'echo t' command=
recommended
below once afs hung again. =A0It definitely put some output in dmesg.
=A0The only D states that were listed were bash sessions (I think).
=A0It looks like they were sessions that I opened after the user told
me that afs was hung again. =A0I tried to attach the dmesg output and
cmdebug output but the email was rejected because the log files were way
too big. =A0Any ideas of what to try next? =A0Or is there anything
in particular that I should look at in the cmdebug or dmesg output. =A0Than=
ks,</font>
<br><div><font face=3D"sans-serif" size=3D"2"><br>
Mark Henry<br>
</font>
<br>
<br>
<br>
<table width=3D"100%">
<tbody><tr valign=3D"top">
<td width=3D"40%"><font face=3D"sans-serif" size=3D"1"><b>Felix Frank <F=
elix.Frank@Desy.de></b>
</font>
<p><font face=3D"sans-serif" size=3D"1">05/04/2009 11:46 PM</font>
</p></td><td width=3D"59%">
<table width=3D"100%">
<tbody><tr valign=3D"top">
<td>
<div align=3D"right"><font face=3D"sans-serif" size=3D"1">To</font></div>
</td><td><font face=3D"sans-serif" size=3D"1">Mark Henry <<a href=3D"mai=
lto:mark.henry@infoprint.com" target=3D"_blank">mark.henry@infoprint.com</a=
>></font>
</td></tr><tr valign=3D"top">
<td>
<div align=3D"right"><font face=3D"sans-serif" size=3D"1">cc</font></div>
</td><td><font face=3D"sans-serif" size=3D"1"><a href=3D"mailto:openafs-inf=
o@openafs.org" target=3D"_blank">openafs-info@openafs.org</a></font>
</td></tr><tr valign=3D"top">
<td>
<div align=3D"right"><font face=3D"sans-serif" size=3D"1">Subject</font></d=
iv>
</td><td><font face=3D"sans-serif" size=3D"1">Re: [OpenAFS] /afs area is ha=
nging</font></td></tr></tbody></table>
<br>
<table>
<tbody><tr valign=3D"top">
<td>
</td><td></td></tr></tbody></table>
<br></td></tr></tbody></table>
<br>
<br>
<br></div><div><div></div><div><tt><font size=3D"2">On Mon, 4 May 2009, Mar=
k Henry wrote:<br>
<br>
> I tried the -fakestat-all option and it did not work.<br>
<br>
Weird.<br>
<br>
> I have searched /var/log/messages. =A0I have checked the config
files. =A0User<br>
> authentication works fine (even when the system is hanging). =A0If
I run the<br>
> command 'ls -l /afs' that window is hung (or any other command=
that<br>
> references afs). =A0If an afs user logs in when the system is in
a bad state<br>
> the session immediately hangs because it can't cd to the afs home
dir. =A0I<br>
> don't know what to do other than reboot. =A0Can someone tell me
what else to<br>
> try to find out why this system is hanging? =A0Thanks,<br>
<br>
You can find out just which call gets stuck by issuing an<br>
'echo t >>/proc/sysrq-trigger'. Call traces for all processes=
can
then be<br>
found in dmesg. The broken processes are probably in a D state.<br>
<br>
This will probably not identify the root cause, but may give a clue about<b=
r>
what's going on.<br>
<br>
HTH<br>
=A0- Felix<br>
</font></tt>
<br>
<br></div></div><div><div></div><div>
___________________________________________________________________________=
__<br>
"This message and any attachments are solely for the intended recipien=
t and may contain confidential or privileged information. If you are not th=
e intended recipient, any disclosure, copying, use, or distribution of the =
information included in this message and any attachments is prohibited. If =
you have received this communication in error, please notify us by reply e-=
mail and immediately and permanently delete this message and any attachment=
s. Thank you." _______________________________________________________=
______________________<br>
</div></div></blockquote></div></div></div><br>
</div><br>
--001636e909a39760230469599b58--