[OpenAFS-devel] still seeing iput errors on 2.2.19...

[Nathan Neulinger]

I just checked the same thing on another couple servers I've got that
experiences a similar problem. It has another 'volume root' cache entry
with a large refCount, but not nearly as high, and not growing as
rapidly. As with htdig, it's a afs volume that a web server runs out of.

I even saw this oddball:

** Cache entry @ 0xd0c36fa0 for 1.536925706.1.1
    2048 bytes  DV 21 refcnt -27651
    callback cf04a320   expires 987300044
    0 opens     0 writers
    volume root
    states (0x1), stat'd

That's a volume for a heavily accessed cgi script. That one appears to
be the opposite problem though.

It appears pretty consistent though - all the entries with rediculous
refCounts are volume roots. 

I'm doing what I can to instrument afs_dcache.c to do some cache
tracing. Hopefully can track down what is going on here, seems to me
that there are mismatched refcount manipulations somewhere.

-- Nathan

Nathan Neulinger wrote:
> 
> Already been rebooted, but I will use this info the next time I get that
> same panic. In fact, I think I might just start watching cmdebug (never
> knew about that cmd) on this machine.
> 
> I just ran it on the machine, and note this:
> 
> ** Cache entry @ 0xd0340a28 for 1.536994608.1.1
>     2048 bytes  DV 65 refcnt 18375
>     callback cc00e380   expires 987298636
>     0 opens     0 writers
>     volume root
>     states (0x1), stat'd
> 
> lsof says nothing has that inode opened.
> 
> I ran it a little bit later:
>     2048 bytes  DV 65 refcnt 18387
> and a little later:
>     2048 bytes  DV 65 refcnt 18439
> 
> I'm assuming that the second number on there is the volume:
> 
> sw.htdig                          536994608 RW     266915 K  On-line
>     afs4.cc.umr.edu /vicepc
>     RWrite  536994608 ROnly          0 Backup  536994610
>     MaxQuota     350000 K
>     Creation    Tue Sep 12 16:16:07 2000
>     Last Update Mon Apr  2 08:44:02 2001
>     167 accesses in the past day (i.e., vnode references)
> 
>     RWrite: 536994608     Backup: 536994610
>     number of sites -> 1
>        server afs4.cc.umr.edu partition /vicepc RW Site
> 
> There are cgi scripts that run out of that dir, and the web server
> itself, but it's data files and logging are all on local disk.
> Everything on afs on that volume is read-only. (Server is not
> authenticated.)
> 
> I did a find -ls over that entire volume, and that inode number doesn't
> exist. I'm guessing that this is because it's not actually a file - it
> said "volume root".
> 
> So, the ultimate question - why is the refcount increasing?
> 
> Running an fstrace on the machine shows no continual accesses to that
> volume other than "lookup adp" on the .htaccess file, which don't exist
> on that volume. (There are probably others when the cgi is hit, but that
> isn't very often, certainly not often enough to account for above refcnt
> increases.)
> 
> -- Nathan
> 
> Chaskiel M Grundman wrote:
> >
> > Excerpts from mail: 14-Apr-101 Re: [OpenAFS-devel] still s.. by Nathan
> > Neulinger@umr.edu
> > > Interesting... this just happened again on the machine it usually
> > > happens on, but I happened to notice something about it:
> > Assuming that you haven't rebooted yet, could you try the following:
> >
> > % cmdebug localhost -long | grep -A4 -B1 'refcnt .....'
> >
> > Which will produce something like
> >
> > ** Cache entry @ 0xc89b3a00 for 1.1970723513.3374.152806
> >     287 bytes   DV 4 refcnt 48
> >     callback 00000000   expires 986966221
> >     0 opens     0 writers
> >     normal file
> >
> > (except that the refcount will be big) Take the second and third decimal
> > numbers on the first line and run them through this perl one-liner:
> >
> > % perl -e 'print ((($ARGV[0]&0x7fff) << 16) | $ARGV[1]); print "\n"'
> > 1970723513 3374
> >
> > This will produce a single decimal number (that being the fake inode
> > number AFS gives the kernel.) now, use lsof to get all references to
> > that inode
> >
> > # lsof |grep 1522076974
> >
> > (you might want to just send me this stuff....)
> >
> > _______________________________________________
> > OpenAFS-devel mailing list
> > OpenAFS-devel@openafs.org
> > https://lists.openafs.org/mailman/listinfo.cgi/openafs-devel
> 
> --
> 
> ------------------------------------------------------------
> Nathan Neulinger                       EMail:  nneul@umr.edu
> University of Missouri - Rolla         Phone: (573) 341-4841
> CIS - Systems Programming                Fax: (573) 341-4216
> _______________________________________________
> OpenAFS-devel mailing list
> OpenAFS-devel@openafs.org
> https://lists.openafs.org/mailman/listinfo.cgi/openafs-devel

-- 


------------------------------------------------------------
Nathan Neulinger                       EMail:  nneul@umr.edu
University of Missouri - Rolla         Phone: (573) 341-4841
CIS - Systems Programming                Fax: (573) 341-4216