[AFS3-std] File Server/Cache Manager Extended Status Information

Sun, 06 Apr 2008 17:39:09 -0400

This is a cryptographically signed message in MIME format.

--------------ms080402060609020402010803
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Matt Benjamin wrote:
> File Server/Cache Manager Extended Status Information
>
> Motivation
>
> The FetchStatus/BulkStatus and Callback mechanisms provided for by the
> current AFS-3 protocol are not sufficiently rich to support
> server-coordinated byte-range locking [1].
>
> Our analysis is that implementation of, in particular, granular and
> asynchronous file change notifications in the AFS protocol would
> provide an important general improvement independent of new features.
> Specifically, the current file status notification mechanism requires
> clients to discard an entire file when a small portion of the file may
> have changed.
>
> Changes to file access permissions are added, because they were
> enumerated by Jeff Altman as of interest.  Some details in this area are
> not finalized, and feedback/discussion is requested.  Locking specific
> change notifications are omitted (just the NotificationType flag is
> included) to simplify discussion.  We expect to send a separate proposal
> on locking interfaces.
You misunderstood our discussion.  ACL changes are not an area of interest.
They are one of the reasons that BreakCallback() is executed in the file
server.

The full list includes:

    * StoreACL
    * StoreData
    * StoreStatus
    * RemoveFile
    * CreateFile
    * RenameFile
    * Symlink
    * Link
    * MakeDir
    * RemoveDir
    * ReleaseLock
    * when the callback is dropped by the file server due to lack of space

> Proposed Change
>
> This proposal would supplement the current CallBack interface and
> protocol contract with a granular chunk or range invalidate mechanism,
> reducing network traffic and improving cache coherence.
>
> The specifics in this protocol draft are submitted for discussion.  We
> feel that the following features of the proposal are key:
>
> 1. granular notifications from the file server to cache managers,
> specifically, invalidation of byte ranges in a file for which it owns
> a CallBack
>
> 2. backward compatiblity with existing clients and servers (this is
> provided for using the *_GetCapabilties RPC mechansims)
>
> 3. asynchronous notification from the file server to cache managers,
> where cache manager and file server agree to support it.
>
> Protocol Negotiation
>
> Prior to issuing any FetchStatus RPCs, clients using this feature will
> issue a GetCapabilities() RPC on the file server of interest.  A new
> capability flag, cacheable by the client, will indicate the file
> server is capable of delivering extended status notifications.
I'm not sure that this is required.
>
> Should the client wish to receive extended status notifications, it
> will so notify the file server by issuing an extended version of the
> FS FetchStatus RPC, containing a filter mask parameter specifying
> extended status notifications the client is willing to receive
> (cf. Data Types).
Why should the client have to request an extended callback notification?

The client already sends the file server a capabilities list as part of the
TellMeAboutYourself response.  If the client supports the extended
callbacks it can notify the file server in the TellMeAboutYourself response
and the file server can simply send the extended callbacks.
>
> Hypothetically, this call might look like:
>
> FetchStatusEx(
> ~  IN AFSFid *Fid,
> ~  IN afs_int32 NotificationTypes,
> ~  OUT AFSFetchStatus *OutStatus,
> ~  AFSCallBack *CallBack,
> ~  AFSVolSync *Sync
> ) = 132;
>
> Data Types
>
> The following data types support the extended FetchStatus RPC in the
> file server interface, and a set of extended status notification
> callback RPCs in the cache manager interface.  They are presented in
> rxgen format.
>
>
> ***
>
> /* callback notification classes, used by a client to specify a set
> ~ * of notifications it is interested in, and by the file server to
> ~ * specify notification type it knows how to provide */
>
> %#define AFSCB_IND_FileInvalidate      0x01    /* traditional callback */
> %#define AFSCB_IND_RangeInvalidate      0x02    /* range in file has 
> changed */
> %#define AFSCB_IND_AccessChange          0x04    /* ACL or mode has 
> changed */
>
> /* lock management notification interfaces -- notification structures
> ~ * and suggested prototypes for these calls are not included here to
> ~ * reduce scope of discussion, at GK request */
>
> %#define AFSCB_IND_LockAsyncResult      0x08    /* lock request 
> completions */
> %#define AFSCB_IND_LockDelegation      0x16    /* lock delegation offers
>                                               * and revocations are 
> requested */
>
>
> /* new file-change notification types */
>
> %#define AFSCB_FChange            0x01  /* range is invalidated */
> %#define AFSCB_FTrunc            0x02  /* file reset to [0..Length] */
>
> struct AFSCBRange {
> ~    afs_int64 Offset;
> ~    afs_int64 Length;
> };
>
> struct AFSRangeInvalidateCallback {
> ~    afs_uint32 NotificationType;
> ~    afs_uint32 NewDataVersion;
> ~    AFSCBRange Range;
> };
>
> typedef    AFSRangeInvalidateCallBack AFSRangeCallBackSeq<AFSCBMAX>;
>
> ***
I do not understand this level of complexity.  A callback is issued 
because a
change has occurred on the file server.  The problem that the cache managers
have with the existing callback mechanism is that the cache manager has no
idea what the trigger was.  From my perspective, the cache manager should
simply be told the reason for the trigger.

For example:

    *     StoreData (newDV, offset, length)
    *     StoreACL (newDV)
    *     StoreStatus (newDV, status data)
    *     CreateFile (newDV, filename, fid)
    *     RemoveFile (newDV, filename, fid)
    *     RenameFile (newDV, oldname, newname, fid)
    *     Symlink (newDV, name, fid)
    *     Link (newDV, name, fid)
    *     MakeDir (newDV, name, fid)
    *     RemoveDir (newDV, name, fid)
    *     ReleaseLock (newDV, type, fid)
    *     Cancelled

By doing so the cache managers obtain the data they need without requiring
any architectural changes to the file server.  These changes could be 
deployed
today.
>
> Remote Procedure Calls
>
> The following RPCs are added to the cache manager call back
> interface.  They are issued only when
>
> 1. a valid CallBack promise exists on a file for the called-back
> client
>
> 2. the client has registered an interest in receiving extended
> notifications
>
> Proc RangeInvalidateCallBack
>
> This RPC communicates a vector of file changes, of byte-range
> granularity, for a corresponding vector of CallBacks.
>
> proc RangeInvalidateCallBack {
> ~  IN  AFSCBFids *Fids_Array,
> ~  AFSRangeCallbackSeq *CallBacks_Array
> } multi = 65539;
>
Why is there a range of byte range changes?  A StoreData operation
does not take a list of independent commits.  Each StoreData will
result in a BreakCallBack(). 
> Proc AccessChangeCallBack
>
> This RPC communicates a vector of file/directory access changes.
> These changes may be to ACLs or Unix mode bits on the object.
>
> NB., we are aware that currently, the OpenAFS cache manager tracks
> user access right on files, not ACLs.  This formated is used to
> stimulate discussion--the cache manager and file server may not
> currently share sufficient information (by design) to permit trivially
> sending changes to the access structures cached by clients.
>
> proc AccessChangeCallBack {
> ~  IN  AFSCBFids *Fids_Array,
> ~  AFSAccessCallbackSeq *CallBacks_Array
> } multi = 65540;
The Cache Manager does not track ACLs because it does not have
enforcement responsibility and has no interaction with the Protection
Service.  The purpose of the StoreACL BreakCallBack is to instruct
the cache manager to flush the access rights history for the file.

Jeffrey Altman

--------------ms080402060609020402010803
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJeTCC
AxcwggKAoAMCAQICEALr5BE3U6n+HWCoLbyhohMwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UE
BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT
I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA3MDUzMTA2MTM1N1oX
DTA4MDUzMDA2MTM1N1owczEPMA0GA1UEBBMGQWx0bWFuMRUwEwYDVQQqEwxKZWZmcmV5IEVy
aWMxHDAaBgNVBAMTE0plZmZyZXkgRXJpYyBBbHRtYW4xKzApBgkqhkiG9w0BCQEWHGphbHRt
YW5Ac2VjdXJlLWVuZHBvaW50cy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQCsoz/0+s4Cn65n/3bU3shXw4y5u1uEMEsBOiqNU0PfIKGYQe95b1FKNbNAkctSdQT6GF5c
bhSnJPmb2OOb1frx64dlDgskaG561xa8XPA1aP8Cc+33dgsSLIxGEh97lyUYHEfWBC03KMCF
PKhZfcrGAXoVCrFBadnLAokQbUTFahVg/qQx2IT3wSj1sCIfV5UDuXcEKHCvRtEZIsSzu184
9Cj6I4nY5bt+r94kyDHM94MHYBJi+6tWLFRy2gkIB3HEPmxAiQrKljNpH9bOffiBLIAgmJ6d
1ZXepBXyexQbwOYvftpVlMEFHHQmdiwH3tj69hE78XvM5X9J+SbjbuNpAgMBAAGjOTA3MCcG
A1UdEQQgMB6BHGphbHRtYW5Ac2VjdXJlLWVuZHBvaW50cy5jb20wDAYDVR0TAQH/BAIwADAN
BgkqhkiG9w0BAQUFAAOBgQB8FShDN2Ig034Y5eyadiFDEtOvsIJ3Z2xV9aTL4u8xMlz1gZR1
AZAvCv+ZMMRRKWCsrG5tItV8DFPSfWAGMpInmMarA4f76JRLQEUhkRUg8GpkJM5ryk5EDakk
0oiBQcQD8A+UHwrcmaj3UWxQ9zCjDgU+1mY9nEQxZZyp4eeUfzCCAxcwggKAoAMCAQICEALr
5BE3U6n+HWCoLbyhohMwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoT
HFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25h
bCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA3MDUzMTA2MTM1N1oXDTA4MDUzMDA2MTM1N1ow
czEPMA0GA1UEBBMGQWx0bWFuMRUwEwYDVQQqEwxKZWZmcmV5IEVyaWMxHDAaBgNVBAMTE0pl
ZmZyZXkgRXJpYyBBbHRtYW4xKzApBgkqhkiG9w0BCQEWHGphbHRtYW5Ac2VjdXJlLWVuZHBv
aW50cy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCsoz/0+s4Cn65n/3bU
3shXw4y5u1uEMEsBOiqNU0PfIKGYQe95b1FKNbNAkctSdQT6GF5cbhSnJPmb2OOb1frx64dl
DgskaG561xa8XPA1aP8Cc+33dgsSLIxGEh97lyUYHEfWBC03KMCFPKhZfcrGAXoVCrFBadnL
AokQbUTFahVg/qQx2IT3wSj1sCIfV5UDuXcEKHCvRtEZIsSzu1849Cj6I4nY5bt+r94kyDHM
94MHYBJi+6tWLFRy2gkIB3HEPmxAiQrKljNpH9bOffiBLIAgmJ6d1ZXepBXyexQbwOYvftpV
lMEFHHQmdiwH3tj69hE78XvM5X9J+SbjbuNpAgMBAAGjOTA3MCcGA1UdEQQgMB6BHGphbHRt
YW5Ac2VjdXJlLWVuZHBvaW50cy5jb20wDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQUFAAOB
gQB8FShDN2Ig034Y5eyadiFDEtOvsIJ3Z2xV9aTL4u8xMlz1gZR1AZAvCv+ZMMRRKWCsrG5t
ItV8DFPSfWAGMpInmMarA4f76JRLQEUhkRUg8GpkJM5ryk5EDakk0oiBQcQD8A+UHwrcmaj3
UWxQ9zCjDgU+1mY9nEQxZZyp4eeUfzCCAz8wggKooAMCAQICAQ0wDQYJKoZIhvcNAQEFBQAw
gdExCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUg
VG93bjEaMBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRp
b24gU2VydmljZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFp
bCBDQTErMCkGCSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw0w
MzA3MTcwMDAwMDBaFw0xMzA3MTYyMzU5NTlaMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxU
aGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwg
RnJlZW1haWwgSXNzdWluZyBDQTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAxKY8VXNV
+065yplaHmjAdQRwnd/p/6Me7L3N9VvyGna9fww6YfK/Uc4B1OVQCjDXAmNaLIkVcI7dyfAr
hVqqP3FWy688Cwfn8R+RNiQqE88r1fOCdz0Dviv+uxg+B79AgAJk16emu59l0cUqVIUPSAR/
p7bRPGEEQB5kGXJgt/sCAwEAAaOBlDCBkTASBgNVHRMBAf8ECDAGAQH/AgEAMEMGA1UdHwQ8
MDowOKA2oDSGMmh0dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3dGVQZXJzb25hbEZyZWVtYWls
Q0EuY3JsMAsGA1UdDwQEAwIBBjApBgNVHREEIjAgpB4wHDEaMBgGA1UEAxMRUHJpdmF0ZUxh
YmVsMi0xMzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPqCy7MDaNmrGcPf6+svsIXoUOWlJ1/
TCG4+DYfqi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUaC4J0niVQlGLH2ydxVyWN3amc
OY6MIE9lX5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk13iSx0x1G/11fZU8xggNkMIID
YAIBATB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5
KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQ
AuvkETdTqf4dYKgtvKGiEzAJBgUrDgMCGgUAoIIBwzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN
AQcBMBwGCSqGSIb3DQEJBTEPFw0wODA0MDYyMTM5MDlaMCMGCSqGSIb3DQEJBDEWBBTaERCg
nOZmTJdltkNWhwKccRocdDBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqGSIb3DQMHMA4GCCqGSIb3
DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCBhQYJKwYB
BAGCNxAEMXgwdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcg
KFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3Vpbmcg
Q0ECEALr5BE3U6n+HWCoLbyhohMwgYcGCyqGSIb3DQEJEAILMXigdjBiMQswCQYDVQQGEwJa
QTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhh
d3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEALr5BE3U6n+HWCoLbyhohMwDQYJ
KoZIhvcNAQEBBQAEggEAVQ5jjnNC9AYAQUa+DSOgOSbjAyyQFj2u2lDRPXVV1xGa52kL6AAe
zg8Qtki8iKUZIGubVE7vN/1FL3tqkCP1yM2RpAlOe6QCHyF32vMlILe99D4Oelc2v+LbzOnF
9pnv1cSOrNMuK7IimiMQXIb8IwTjSULboeCRnK4C/N97t6BeJcGoyk0xYdVVcXO52AZ5my0U
me+DTkIRZCWzh2yosq0FG16dMn+wIZNdrTub9+i/WbwZkmHWVeDMlqbChIEzpIW3AOAK6Q8Q
hU1OWiGtCCbbWMc8S3XlFs1to/whL7Ll6RplRfXlLkygOhMMIXwMMhiUxrKhRYEdAboBjMth
rgAAAAAAAA==
--------------ms080402060609020402010803--