[OpenAFS] broken callbacks

[Andreas Hirczy]

Hi!

To hunt down some performance problems on our AFS servers I started to
integrate some metrics to our Ganglia monitoring system via a dirty
hack; see for example
<https://itp.tugraz.at/ganglia/?c=Theoretische%20Physik%20-%20Serverraum%20Chemie&h=faepsv00.tu-graz.ac.at>.

In that context 3 questions regarding nFEs and nCBs entered my mind:

 - What's the meaning of those?
 
 - I found some information about nFEs and nCBs in
   <https://www.openafs.org/pages/newsletter/newsletter-2013-03-volume004-issue05.html#openafs_tuning__part_i__fileservers__general>:
   "If nFEs or nCBs ever exceeds nblks, that is when the fileserver runs
   out of callbacks." I found that those metrices have a similar
   behaviour, but are usually not the same.

   Should I consider to store just "max(nFEs, nCBs)" or can I learn
   something from this difference?

 - Sometimes I see a spike in the usage of those values, e.g.
   <https://itp.tugraz.at/~ahi/privat/OpenAFS/graph_nCBs_2016-04-20.png>
   I can find out about volumes if I turn the debug level of the
   fileserver processes up 3 times, but this uses quite a bit of space
   to leave turned on permanently. Is there some easy accessible data on
   the (historic) distribution of callback on volumes?

   I'd rather prefer not to increase -cb, since this seems to be not our
   usual usage pattern.
   ----
   Wed Apr 20 12:59:57 2016 We have run out of callback space; forcing
                            callback revocation. This suggests the
                            fileserver is configured with insufficient
                            callbacks; you probably want to increase the
                            -cb fileserver parameter (current setting:
                            1048576). The fileserver will continue to
                            operate, but this may indicate a severe
                            performance problem
   ----

You can find a bit more of my thougts in
<https://itp.tugraz.at/~ahi/admin/AndrewFileSystem.html#org1c1731f>, but
be aware - most of it is in german..

Best regards
Andreas
-- 
Andreas Hirczy <ahi@itp.tugraz.at>                  https://itp.tugraz.at/~ahi/
Graz University of Technology                       phone: +43/316/873-   8190
Institute of Theoretical and Computational Physics    fax: +43/316/873-10 8190
Petersgasse 16, A-8010 Graz                        mobile: +43/664/859 23 57