[OpenAFS] AFS version of du
Sun, 02 May 2010 10:36:49 -0400
Douglas E. Engert wrote:
> Richard Brittain wrote:
>> On Fri, 30 Apr 2010, Steve Simmons wrote:
>>> On Apr 30, 2010, at 4:15 AM, Staffan H?m?l? wrote:
>>>> Is there a version of du that does not follow AFS mountpoints?
>>>> If I try to do a 'du -sh *' in a directory that has some AFS
>>>> mountpoints it inevitably fails after some time. It also takes a
>>>> lot of time when it has to look through things in mounted volumes
>>>> (e.g. the backup volume that I have mounted in many places).
>>>> I've tried -P and -x to make it skip mount points, but it doesn't
>>>> work (at least on CentOS 5 Linux).
>>> The short answer is that it might be coming coming, but it's going
>>> to be a while. Here's the detailed scoop as I understand it, based
>>> on some old mail exchanges I had with the findutils maintainer James
>>> Youngman and discussion on the findutils mailing list.
>> This is very promising news. I was about to start a new thread about
>> problems with 'find', but since it is so closely linked with this
>> I'll just jump in here.
>> Yesterday I was running 'find' and 'fs lsmount' across our cell to
>> explicitly look for volume mountpoints, and got some very strange
>> errors. On directories which exist and I had full access to, find
>> would sometimes throw a big pile of errors like:
>> find: ./users/a/: No such file or directory
> I have been working on an AFS audit tool written in Perl since find has
> too many issues. The heart of this uses File::Find and if a directory is
> found AFS::FS::lsmount( $rname ) is used to test if its a mount point
> and set File::Find::prune = 1; The Perl module AFS-2.6.2 was released
> on 4/1, and is working well!
> This same combination could be used to write a Perl du command that
> would not cross mount points.
> A list of volumes is input to the tool, and each volume is mounted
> temporarily. Using the AFS::ACL module ACLs are examined and if a sub
> directory has the same ACL as its parent, it is reported with the parent.
> (the ACL entries are sorted before the compare.) So our 2000 volumes
> with 600,000 directories get reduced down to 8000 entries to look at.
> These are further examined based on the type of volume and the uses and
> rights listed in the ACLs.
> If all the volumes in the cell are scanned, one can find volumes that
> have no mount points as well as volumes with multiple mount points.
>> - and obviously did not recurse into that directory. It was acting
>> like it read the directory entries but failed to later stat() them.
>> I blamed GNU find for a while, but then observed the same behaviour
>> from Solaris 9 and OSX.
>> This is definitely not a permission issue (l ACL but not r), but it
>> does seem to be related to crossing mount points, since I have never
>> seen an error running find inside a single volume. I finally got a
>> RHEL5 linux box to recurse my cell to the desired depth with no
>> errors, but only as long as it was doing nothing else. Running two
>> 'find's immediately generated errors (RHEL5 current, OpenAFS 1.4.12).
>> My servers are a mix of 1.4.11 and 1.4.12 currently.
>> I discovered that one user had made a mount point for a different
>> cell's root inside his home. RHEL5/openafs 1.4.12 happily crossed
>> over and starting searching the other cell, but I noticed that my
>> other test system (RHEL4/openafs 1.4.11) didn't follow. It would be
>> _really_ nice if tools didn't follow mount points to other cells, if
>> nothing else!
Is there a third-party du-like program available that I can run? We're
looking at adding folder totals to filedrawers. Yes, we're looking at it
even though it might cause issues.