[OpenAFS] Debugging a network performance problem that affects AFS

Dale Pontius pontius@btv.ibm.com
Fri, 14 Jan 2011 08:33:49 -0500


This is a multi-part message in MIME format.
--------------000108000805090703010804
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Now that you mention it, there *may* be a firewall involved - I'm not 
sure.  Some time back they began putting putting some server pools 
behind firewalls, "protecting" them from the campus LAN.  I don't know 
if the AFS servers are one of those pools.

As for client software, this is more of a network-wide event.  Many 
client machines, several levels of client software, but most of the 
machines are not user-installed.  (mine being one of few exceptions)  
I'm relatively certain that none of these machines have multiple clients 
installed.  Well, my machine does, because it's multi-boot and the 
various OS installs have different levels of AFS client software.  But 
at any one time only one level of client is active.

As for cache, it's almost entirely on its own partition.  The standard 
install appears to allocate an 8G partition, and my own installs use a 
4G partition.  (The cache partition is shared between bootable OS installs.)

Dale
On 01/13/2011 03:56 PM, Anne Salemme wrote:
> hi dale,
>    not knowing anything about your setup...let me tell you two things 
> that took me _weeks_ to figure out at one site....might be useful for you:
>
> 1) a firewall between client and server....this one had a throttle, so 
> it would only cut off if traffic got "high enough"...
>
> 2) client system had two different versions of afs client installed 
> (obviously, should not have happened)...anyway, two afs clients on 
> same system, fighting it out, looked like "afs is slow"
>
> one other thing to look at: the client cache setup..make sure it's not 
> competing for space...
> best of luck!
>
> anne
>
> --- On *Thu, 1/13/11, Dale Pontius /<pontius@btv.ibm.com>/* wrote:
>
>
>     From: Dale Pontius <pontius@btv.ibm.com>
>     Subject: [OpenAFS] Debugging a network performance problem that
>     affects AFS
>     To: openafs-info@openafs.org
>     Date: Thursday, January 13, 2011, 3:24 PM
>
>     We're having intermittent network performance problems, and the
>     primary manifestation to us as users looks like "an OpenAFS
>     slowdown."  I'm wondering if it's possible to collect access time
>     statistics out of an OpenAFS Linux client.  A little time with
>     google and I see the "-enable_peer_stats" and
>     "-enable_process_stats" options when starting the client daemon,
>     and this very well may furnish the information that I need.  A
>     subsequent search gets me to the "rxdebug" document, though that
>     document appears to be server-centric as opposed to querying the
>     client.  Nor does it tell me what information I can collect or if
>     access time is part of that information - only mentioning serveral
>     parameters that it does collect.
>
>     Can someone toss me a bone here - or a link?
>
>     Thanks
>     Dale Pontius
>
>     -- Dale Pontius
>     Senior Engineer
>     IBM Corporation
>     Phone: (802) 769-6850
>     Tie-Line: 446-6850
>     email: pontius@us.ibm.com </mc/compose?to=pontius@us.ibm.com>
>
>     This e-mail and its attachments, if any, may contain confidential
>     and privileged material for the sole use of the intended
>     recipient. Any review, use, distribution or disclosure by others
>     is strictly prohibited. If you are not the intended recipient (or
>     authorized to receive for the recipient), please contact the
>     sender by reply e-mail and delete all copies of this message from
>     your system without copying it and notify sender of the
>     misdirection by reply e-mail.
>
>     _______________________________________________
>     OpenAFS-info mailing list
>     OpenAFS-info@openafs.org </mc/compose?to=OpenAFS-info@openafs.org>
>     https://lists.openafs.org/mailman/listinfo/openafs-info
>

-- 
Dale Pontius
Senior Engineer
IBM Corporation
Phone: (802) 769-6850
Tie-Line: 446-6850
email: pontius@us.ibm.com

This e-mail and its attachments, if any, may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply e-mail and delete all copies of this message from your system without copying it and notify sender of the misdirection by reply e-mail.


--------------000108000805090703010804
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#ffffff" text="#000000">
    Now that you mention it, there *may* be a firewall involved - I'm
    not sure.&nbsp; Some time back they began putting putting some server
    pools behind firewalls, "protecting" them from the campus LAN.&nbsp; I
    don't know if the AFS servers are one of those pools.<br>
    <br>
    As for client software, this is more of a network-wide event.&nbsp; Many
    client machines, several levels of client software, but most of the
    machines are not user-installed.&nbsp; (mine being one of few
    exceptions)&nbsp; I'm relatively certain that none of these machines have
    multiple clients installed.&nbsp; Well, my machine does, because it's
    multi-boot and the various OS installs have different levels of AFS
    client software.&nbsp; But at any one time only one level of client is
    active.<br>
    <br>
    As for cache, it's almost entirely on its own partition.&nbsp; The
    standard install appears to allocate an 8G partition, and my own
    installs use a 4G partition.&nbsp; (The cache partition is shared between
    bootable OS installs.)<br>
    <br>
    Dale<br>
    On 01/13/2011 03:56 PM, Anne Salemme wrote:
    <blockquote cite="mid:153454.97340.qm@web305.biz.mail.mud.yahoo.com"
      type="cite">
      <table border="0" cellpadding="0" cellspacing="0">
        <tbody>
          <tr>
            <td style="font: inherit;" valign="top">hi dale,<br>
              &nbsp;&nbsp; not knowing anything about your setup...let me tell you
              two things that took me _weeks_ to figure out at one
              site....might be useful for you:<br>
              <br>
              1) a firewall between client and server....this one had a
              throttle, so it would only cut off if traffic got "high
              enough"...<br>
              <br>
              2) client system had two different versions of afs client
              installed (obviously, should not have happened)...anyway,
              two afs clients on same system, fighting it out, looked
              like "afs is slow"<br>
              <br>
              one other thing to look at: the client cache setup..make
              sure it's not competing for space...<br>
              best of luck!<br>
              <br>
              anne<br>
              <br>
              --- On <b>Thu, 1/13/11, Dale Pontius <i><a class="moz-txt-link-rfc2396E" href="mailto:pontius@btv.ibm.com">&lt;pontius@btv.ibm.com&gt;</a></i></b>
              wrote:<br>
              <blockquote style="border-left: 2px solid rgb(16, 16,
                255); margin-left: 5px; padding-left: 5px;"><br>
                From: Dale Pontius <a class="moz-txt-link-rfc2396E" href="mailto:pontius@btv.ibm.com">&lt;pontius@btv.ibm.com&gt;</a><br>
                Subject: [OpenAFS] Debugging a network performance
                problem that affects AFS<br>
                To: <a class="moz-txt-link-abbreviated" href="mailto:openafs-info@openafs.org">openafs-info@openafs.org</a><br>
                Date: Thursday, January 13, 2011, 3:24 PM<br>
                <br>
                <div class="plainMail">We're having intermittent network
                  performance problems, and the primary manifestation to
                  us as users looks like "an OpenAFS slowdown."&nbsp; I'm
                  wondering if it's possible to collect access time
                  statistics out of an OpenAFS Linux client.&nbsp; A little
                  time with google and I see the "-enable_peer_stats"
                  and "-enable_process_stats" options when starting the
                  client daemon, and this very well may furnish the
                  information that I need.&nbsp; A subsequent search gets me
                  to the "rxdebug" document, though that document
                  appears to be server-centric as opposed to querying
                  the client.&nbsp; Nor does it tell me what information I
                  can collect or if access time is part of that
                  information - only mentioning serveral parameters that
                  it does collect.<br>
                  <br>
                  Can someone toss me a bone here - or a link?<br>
                  <br>
                  Thanks<br>
                  Dale Pontius<br>
                  <br>
                  -- Dale Pontius<br>
                  Senior Engineer<br>
                  IBM Corporation<br>
                  Phone: (802) 769-6850<br>
                  Tie-Line: 446-6850<br>
                  email: <a moz-do-not-send="true"
                    ymailto="mailto:pontius@us.ibm.com"
                    href="/mc/compose?to=pontius@us.ibm.com">pontius@us.ibm.com</a><br>
                  <br>
                  This e-mail and its attachments, if any, may contain
                  confidential and privileged material for the sole use
                  of the intended recipient. Any review, use,
                  distribution or disclosure by others is strictly
                  prohibited. If you are not the intended recipient (or
                  authorized to receive for the recipient), please
                  contact the sender by reply e-mail and delete all
                  copies of this message from your system without
                  copying it and notify sender of the misdirection by
                  reply e-mail.<br>
                  <br>
                  _______________________________________________<br>
                  OpenAFS-info mailing list<br>
                  <a moz-do-not-send="true"
                    ymailto="mailto:OpenAFS-info@openafs.org"
                    href="/mc/compose?to=OpenAFS-info@openafs.org">OpenAFS-info@openafs.org</a><br>
                  <a moz-do-not-send="true"
                    href="https://lists.openafs.org/mailman/listinfo/openafs-info"
                    target="_blank">https://lists.openafs.org/mailman/listinfo/openafs-info</a><br>
                </div>
              </blockquote>
            </td>
          </tr>
        </tbody>
      </table>
    </blockquote>
    <br>
    <pre class="moz-signature" cols="72">-- 
Dale Pontius
Senior Engineer
IBM Corporation
Phone: (802) 769-6850
Tie-Line: 446-6850
email: <a class="moz-txt-link-abbreviated" href="mailto:pontius@us.ibm.com">pontius@us.ibm.com</a>

This e-mail and its attachments, if any, may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply e-mail and delete all copies of this message from your system without copying it and notify sender of the misdirection by reply e-mail.
</pre>
  </body>
</html>

--------------000108000805090703010804--