[OpenAFS] Cache Consistency problem

[Paul Blackburn]

This is a multi-part message in MIME format.
--------------070700050307010708060200
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

Kağan Coşkun wrote:

> Hi All,
>
> We are having some cache consistency problems. Sometimes the afs 
> client on our web server does not represent the actual files on the 
> some volumes. That is when someone uploads a new file to his directory 
> the file is not noticed on the client side. The volume is a read/write 
> volume. And the problem can be overcomed by creating a file on the 
> volume using the client and also flush commands work too.
>
> And another point actual cache size is bigger then the defined cache 
> size in /usr/vice/etc/cacheinfo. It is 100MB.
>
> [root@cs kaan]# du /usr/vice/cache/ -h
> 32k     /usr/vice/cache/D0
> 25M     /usr/vice/cache/D1
> 54M     /usr/vice/cache/D2
> 53M     /usr/vice/cache/D3
> 44M     /usr/vice/cache/D4
> 175M    /usr/vice/cache
>
>
> Our client is openafs-client-1.2.3-rh7.2.2.
>
> Any ideas to track the problem?
>
> Kaan
>
> _________________________________________________________________
> Protect your PC - get McAfee.com VirusScan Online 
> http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info

Hello Kağan,

The method that AFS uses to maintain cache consistency is called "Callback".

"Callback" is a promise from an AFS fileserver to an AFS client that
if the server-side copy of a file is updated then the callback will be 
"broken".

The way that a client notices a broken Callback is when the file is closed
and then the same file is read again.

The AFS Cache Manager (afsd) will then "see" the Callback is broken
and re-fetch the file from the file server (instead of from the local 
cache).

Therefore, it is possible for a server-side copy of a file to be different
from a client-side copy if the client-side has not re-read the changed file.

Normal behaviour on the client-side is for files to be opened, read and 
closed
so that when they are again opened and read the new server copy will be 
seen.


Regarding the size of the AFS cache: it is recommended that the value
for cache size defined in the cacheinfo file is no larger than about 84%
of the size of the actual cache size. So, in your 100mb cacheinfo size
example, the optimum size for the cache would be 125mb. However,
you can safely have less than 84% and your figures of 80/175 give 45%.

On the subject of disk cache, it is a useful to make a separate local
filesystem (typically "/usr/vice/cache"). Then the optimum size in
cacheinfo will be 84% the size of this filesystem.

Example:
$ cat /usr/vice/etc/cacheinfo
/afs:/usr/vice/cache:55969

$ df /usr/vice/cache
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/hda8                65846     50742     11704  82% /usr/vice/cache

$ echo "$(compute 100*55969/65846)% of $(compute 65846*k/m) mb"
84.99% of 64.30 mb

$ # Utilization of cache:
$ echo "$(compute 100*50742/55969)%"
90.66%

The advantages of having a separate filesystem for /usr/vice/cache/ are:
a) Only the AFS Cache Manager will use the disk space
    in /usr/vice/cache/ this eliminates the problem of not
    having enough space in the /usr/ filesystem if some other
    process floods /usr/.
b) You can measure the utilization of the cache as a percentage
     of disk blocks used versus cacheinfo defined size. (also, use "df")
c) You can specify a different filesystem type for /usr/vice/cache/
     than is used for /usr/.
     Earlier discussions here on the openafs-info mailing list
     suggest it is better to use ext2 for the AFS disk cache.
     You may choose to use ext3 for /usr/.
     FWIW, I have found no problems using ext3 for /usr/vice/cache.

The other point to make is that you appear to be using AFS 1.2.3
and you may like to consider upgrading to the current 1.2.7 version
(I understand 1.2.8 is being worked on now).
--
cheers
paul                                     http://acm.org/~mpb




--------------070700050307010708060200
Content-Type: text/plain;
 name="compute"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="compute"

#!/bin/sh -
#
# $Header:
# $Locker:
# ---------------------------------------------------------------------------
#
# NAME		compute
# AUTHOR	Paul Blackburn, AIXSSC IBM UK    http://acm.org/~mpb
# DATE		Thu Jul 04 15:06:24 BST 1991
# PURPOSE       compute using 'bc' with predefined values of:
#               k(1024), b(512), m(k*k), p(4*m), and scale=2
# BENEFIT
#               Why is "compute" useful. It is very handy for command-line
#               calculations. For example: how many megabytes is 3200
#               512-byte blocks?  answer: compute 3200*b/m      -> 1.56
#
# ---------------------------------------------------------------------------

CMD=`basename ${0}`

usage() {
	cat <<eeooff >&2
Usage: ${CMD} "calculation"
compute "calculation" using 'bc' with predefined values of:
k(1024), b(512), m(k*k), p(4*m), and scale=2

for example, how many (512 byte) blocks in 20 (4*m) physical partions?
$  compute "20*p/b"
163840.00

NB: "calculation" should be in double-quotes to avoid shell evaluation (of *).
(exception: where "calculation" is not matched as a filename by the shell)
eeooff
}

error() {
	echo "${CMD} error: ${1}" >&2
}


if [ -z "${1}" ]; then
	usage
	exit 1
fi

echo "k=1024;b=512;m=k*k;p=4*m;scale=2;" "$*" | bc

--------------070700050307010708060200--