[OpenAFS] 1.4.2 fileserver keep getting large number of blocked connections

Matthew Cocker cockerm@gmail.com
Wed, 1 Aug 2007 13:51:27 +1200


------=_Part_52381_8285141.1185933087481
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Forgot reply-all again

On 8/1/07, Matthew Cocker <cockerm@gmail.com> wrote:
>
> The -p 128 flag alone has not solved the problem. I have a Filelog (from
> one locked up server) and rxdebug output from all servers at the time when
> lots locked up last night). They can be accessed via
>
>
> https://webdropoff.auckland.ac.nz/cgi-bin/pickup/6f25589961a37ad4d4a144bbd868058a/516290
> Interesting I started running cmdebug against all the clients that connect
> to our servers and I can see a lot of machines with the same UUID. I tried
> to apply the patch but it failed to add two of the chunks against 1.4.3source in the rpm. The rejected file output was
>
> patch -p0 < ../../../UUID.patch
> patching file audit.c
> Reversed (or previously applied) patch detected!  Assume -R? [n] y
> Hunk #1 succeeded at 66 (offset 10 lines).
> Hunk #3 FAILED at 88.
> Hunk #4 succeeded at 114 (offset 6 lines).
> Hunk #5 succeeded at 144 (offset 1 line).
> Hunk #6 succeeded at 181 (offset 10 lines).
> Hunk #7 FAILED at 223.
> Hunk #8 succeeded at 222 (offset -3 lines).
> 2 out of 8 hunks FAILED -- saving rejects to file audit.c.rej
> less audit.c.rej
> ***************
> *** 88,98 ****
>             bufferPtr += sizeof(vaLong);
>             break;
>         case AUD_LST:           /* Ptr to another list */
> -           vaLst = (va_list)va_arg(vaList, va_list);
>             audmakebuf(audEvent, vaLst);
>             break;
>         case AUD_FID:           /* AFSFid - contains 3 entries */
> -           vaFid = (struct AFSFid *)va_arg(vaList, struct AFSFid *);
>             if (vaFid) {
> ***************
> *** 223,233 ****
>             fprintf(out, "LONG %d ", vaLong);
>             break;
>         case AUD_LST:           /* Ptr to another list */
> -           vaLst = va_arg(vaList, va_list);
>             printbuf(out, 1, "VALST", 0, vaLst);
>             break;
>         case AUD_FID:           /* AFSFid - contains 3 entries */
> -           vaFid = va_arg(vaList, struct AFSFid *);
>             if (vaFid)
>                 fprintf(out, "FID %u:%u:%u ", vaFid->Volume, vaFid->Vnode,
>                        vaFid->Unique);
> --- 223,233 ----
>             fprintf(out, "LONG %d ", vaLong);
>             break;
>         case AUD_LST:           /* Ptr to another list */
> +           vaLst = (char *)va_arg(vaList, int);
>             printbuf(out, 1, "VALST", 0, vaLst);
>             break;
>         case AUD_FID:           /* AFSFid - contains 3 entries */
> +           vaFid = (struct AFSFid *)va_arg(vaList, int);
>             if (vaFid)
>                 fprintf(out, "FID %u:%u:%u ", vaFid->Volume, vaFid->Vnode,
>                        vaFid->Unique);
>
>
>
> On 7/31/07, Derrick Brashear <shadow@gmail.com> wrote:
> >
> >
> > It's not really particularly useful.
> >
> > I do, however, have a suggestion.
> >
> > 1) up the number of threads to 128 (fileserver -p 128)
> > 2) apply the diff in RT ticket #19461, which i added to the ticket on
> > Mon Aug 08 14:49:12 2005
> >
> >
>

------=_Part_52381_8285141.1185933087481
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Forgot reply-all again<br><br><div><span class="gmail_quote">On 8/1/07, <b class="gmail_sendername">Matthew Cocker</b> &lt;<a href="mailto:cockerm@gmail.com">cockerm@gmail.com</a>&gt; wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
The -p 128 flag alone has not solved the problem. I have a Filelog (from one locked up server) and rxdebug output from all servers at the time when lots locked up last night). They can be accessed via 

<p><a href="https://webdropoff.auckland.ac.nz/cgi-bin/pickup/6f25589961a37ad4d4a144bbd868058a/516290" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">https://webdropoff.auckland.ac.nz/cgi-bin/pickup/6f25589961a37ad4d4a144bbd868058a/516290
</a></p>

Interesting I started running cmdebug against all the clients that connect to our servers and I can see a lot of machines with the same UUID. I tried to apply the patch but it failed to add two of the chunks against 1.4.3

 source in the rpm. The rejected file output was<br><br>patch -p0 &lt; ../../../UUID.patch <br>patching file audit.c<br>Reversed (or previously applied) patch detected!&nbsp; Assume -R? [n] y<br>Hunk #1 succeeded at 66 (offset 10 lines).
<br>Hunk #3 FAILED at 88.<br>Hunk #4 succeeded at 114 (offset 6 lines).<br>Hunk #5 succeeded at 144 (offset 1 line).<br>Hunk #6 succeeded at 181 (offset 10 lines).<br>Hunk #7 FAILED at 223.<br>Hunk #8 succeeded at 222 (offset -3 lines).
<br>2 out of 8 hunks FAILED -- saving rejects to file audit.c.rej<br>less audit.c.rej <br>***************<br>*** 88,98 ****<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; bufferPtr += sizeof(vaLong);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; break;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; case AUD_LST:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /* Ptr to another list */
<br>-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; vaLst = (va_list)va_arg(vaList, va_list);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; audmakebuf(audEvent, vaLst);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; break;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; case AUD_FID:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /* AFSFid - contains 3 entries */<br>-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; vaFid = (struct AFSFid *)va_arg(vaList, struct AFSFid *);
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (vaFid) {<br>***************<br>*** 223,233 ****<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fprintf(out, &quot;LONG %d &quot;, vaLong);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; break;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; case AUD_LST:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /* Ptr to another list */<br>-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; vaLst = va_arg(vaList, va_list);
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; printbuf(out, 1, &quot;VALST&quot;, 0, vaLst);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; break;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; case AUD_FID:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /* AFSFid - contains 3 entries */<br>-&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; vaFid = va_arg(vaList, struct AFSFid *);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (vaFid)
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fprintf(out, &quot;FID %u:%u:%u &quot;, vaFid-&gt;Volume, vaFid-&gt;Vnode,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; vaFid-&gt;Unique);<br>--- 223,233 ----<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fprintf(out, &quot;LONG %d &quot;, vaLong);<br>

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; break;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; case AUD_LST:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /* Ptr to another list */<br>+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; vaLst = (char *)va_arg(vaList, int);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; printbuf(out, 1, &quot;VALST&quot;, 0, vaLst);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; break;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; case AUD_FID:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /* AFSFid - contains 3 entries */
<br>+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; vaFid = (struct AFSFid *)va_arg(vaList, int);<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (vaFid)<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fprintf(out, &quot;FID %u:%u:%u &quot;, vaFid-&gt;Volume, vaFid-&gt;Vnode,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; vaFid-&gt;Unique);
<div><span class="e" id="q_1141f1b34c074f0f_1"><br><br><br><br><div><span class="gmail_quote">On 7/31/07, <b class="gmail_sendername">Derrick Brashear</b> &lt;<a href="mailto:shadow@gmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
shadow@gmail.com</a>&gt; wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>It&#39;s not really particularly useful.<br><br>I do, however, have a suggestion.<br><br>1) up the number of threads to 128 (fileserver -p 128)<br>2) apply the diff in RT ticket #19461, which i added to the ticket on 
<font size="-2">
Mon&nbsp;Aug&nbsp;08&nbsp;14:49:12&nbsp;2005</font> <div><span><br><br></span></div></blockquote></div><br>
</span></div></blockquote></div><br>

------=_Part_52381_8285141.1185933087481--