[AFS3-std] updated extended callback draft
Matt Benjamin
matt@linuxbox.com
Sun, 26 Oct 2008 14:15:11 -0400
This is a multi-part message in MIME format.
--------------040502080901020400030007
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Attached find an updated extended callback draft, incorporating changes,
and corrections made during the openafs implementation, prior
discussion, and feedback from a number of individuals (to whom, thanks).
Nb., recent reviewers, I've left the names of AFSCB_Cancel_LostMyMind
and AFSCB_Cancel_IHateYou as previously defined, by request of the author.
Thanks,
Matt
- --
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFJBLOvJiSUUSaRdSURCG2QAKCOFGhwpI1h6GisXghwXUEZaoJMfQCfRSG5
UsSFjdxB7A3xk0xzIxVZXrg=
=L2JW
-----END PGP SIGNATURE-----
--------------040502080901020400030007
Content-Type: text/plain;
name="callback_extension_d9.txt"
Content-Disposition: inline;
filename="callback_extension_d9.txt"
Content-Transfer-Encoding: quoted-printable
X-MIME-Autoconverted: from 8bit to quoted-printable by aa.linuxbox.com id
m9QIFT8R024514
AFS Callback Extensions (Draft 9)
Matt Benjamin <matt@linuxbox.com>
10/26/2008
Status of this Memo
This document specifies a standards track protocol extension for=20
the OpenAFS community, and requests discussion and suggestions=20
for improvements. Thanks to Jeffery Altman, Tom Keiser, Jeffrey=20
Hutzelman, Derrick Brashear, and Steven Jenkins for their=20
feedback and suggested improvements from previous drafts.
Key Words
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL=20
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and=20
"OPTIONAL" in this document are to be interpreted as described in=20
Internet Engineering Task Force RFC 2119.
Abstract
AFS cache-control strategy is callback (invalidate) based. The=20
AFS callback design allows a client to know when an object it has=20
cached is no longer consistent, but the callback notification=20
message itself provides no specific information about the=20
triggering event. This is a protocol inefficiency, as in several=20
scenarios it results in unnecessary round-trips to file servers=20
to verify file status information, file access information, or to=20
fetch file data which has not changed. We propose an extension of=20
the callback mechanism to provide information about the event(s)=20
triggering a callback, in the payload of the callback=20
notification message itself. The proposed mechanism eliminates=20
most or all unnecessary round-trips imposed by the current=20
callback mechanism, and simultaneously allows AFS implementations=20
to (efficiently) provide correct semantics in several scenarios=20
involving multiple writers (ie, where AFS currently provides=20
incorrect semantics).
Table of Contents
Status of this Memo
Key Words
Abstract
1 Introduction
2 The AFS Callback Mechanism
2.1 Description
2.2 Analysis
3 Extended Callback Interface
3.1 Backward Compatibility
3.2 Interface Changes
3.2.1 Procedures
3.2.2 Constants
All Sequences
AFSXCBInvocation
AFSExtendedCallBack
AFSExtendedCallBackResult
Sequence Types
3.2.3 Data Types
HostIdentifier
AFSXCBInvocation
Fid
Flags
lowDV
highDV
ExpirationTime
CallBacks_Array
AFSExtendedCallback
Flags and ExtraFlags
Origin
NCoalesced
DataVersion
Data
AFSCBFileStatus
AFSCBDirStatus
AFSCB_NotificationData
3.3 Semantic Changes
3.3.1 DataVersion Rule
3.4 Callback Invocations
3.4.1 AFSXCBInvocation
Origin
ExpirationTime
3.4.2 AFSExtendedCallback
Flags
ExtraFlags
DataVersion
Data
3.4.3 Reasons for Cancellation
AFSCB_Cancel_Shutdown
AFSCB_Cancel_CallbackGC
AFSCB_Cancel_VolumeOffline
AFSCB_Cancel_VolumeMoved
AFSCB_Cancel_LostMyMind
AFSCB_Cancel_IHateYou
3.4.4 ExtendedCallback Procedure
3.4.5 Callback Coalescing
Call Consolidation (Sequences of Notifications)
Coalescing of Equivalent Notifications
3.4.6 Asynchronous Delivery
3.4.7 AFSCB_Event_StoreData
3.4.8 AFSCB_Event_StoreACL
3.4.9 AFSCB_Event_StoreStatus
3.4.10 AFSCB_Event_CreateFile
3.4.11 AFSCB_Event_MakeDir
3.4.12 AFSCB_Data_Symlink
3.4.13 AFSCB_Event_Link
3.4.14 AFSCB_Event_RemoveFile
3.4.15 AFSCB_Event_RemoveDir
3.4.16 AFSCB_Event_Rename
3.4.17 AFSCB_Event_Deleted
3.4.18 AFSCB_Event_ReleaseLock
3.5 Callbacks And Read-Only Volume Replicas
3.5.1 Constants
AFSCB_Flag_Release
AFSCB_Release_WholeVolumeCancel
3.5.2 Semantic Changes
4 Appendix A: XDR Grammar
1 Introduction
The AFS protocol provides a comprehensive framework for scalable,=20
secure, wide-area file sharing over IP networks. The AFS system=20
has historically distinguished itself through its emphasis on=20
scalability, a key source of which is client-side caching[1, 4].=20
File data, file and directory metadata, and access control=20
information may all be cached. Cache consistency is maintained=20
through client registration and an associated asynchronous=20
notification mechanism known as the callback.=20
The current AFS consistency model (which is of larger scope than=20
the callback mechanism, eg, it includes AFS sync-on-close=20
semantics) has allowed AFS to scale to large numbers of clients=20
(tens of thousands today), and to perform well under the=20
workloads for which AFS was originally designed.
However, AFS does not perform efficiently under other conditions,=20
such as when more than one client is interested in a file which=20
is changing--even if the file has only one writer, and many=20
readers[footnote:
NFSv4.1 in particular efficiently supports this scenario with=20
byte-range delegation, see[9].=20
]. In general, the AFS protocol arguably (still, considering=20
improvements made between AFS-2 and AFS-3) places too little=20
emphasis on efficient caching of mutable data. The current AFS=20
consistency model is insufficient to correctly support=20
single-file, multiple-writer scenarios, including those required=20
for POSIX semantics, and therefore is insufficient to support=20
many applications which may be run correctly on competing=20
distributed file systems (e.g., CIFS, Novell Netware, or NFSv4).
The efficiency of the current AFS cache management algorithm=20
could be substantially improved if specific triggering event=20
information and current status were included in the payload of=20
the callback notifications sent to clients. In particular,=20
inclusion of the current DataVersion number and affected byte=20
ranges in response to StoreData operations would significantly=20
reduce the need for cache revalidation and reconstruction traffic=20
in response to callbacks--in many cases, altogether. These=20
changes would allow efficient support for single-writer updates=20
on a file with multiple readers. More importantly, they would=20
permit AFS to correctly and efficiently support multiple writers=20
updating disjoint ranges on a single file, a prerequisite for=20
supporting granular file locking (and applications which require=20
it) in future.
2 The AFS Callback Mechanism
2.1 Description
When an AFS-3 client contacts a file server to perform any of=20
several operations on a file, or explicitly to fetch its status,=20
the file server includes in its RPC response an AFSCallBack=20
structure, representing the server's promise to call back the=20
client ``if any modifications are made to the data in the file''.[footnot=
e:
A key paper on AFS-2 has ``before allowing a modification by any=20
other workstation''[1]. The wording of this statement appears=20
calculated to imply that the file server's promise to execute=20
callbacks synchronously with the triggering operations (e.g.,=20
StoreData) specifically constitutes part of the AFS cache=20
consistency guarantee. In our analysis, it does not, though it=20
does contribute strongly to the simplification of the file server=20
design and to reduction of file server workload.
]The AFSCallBack structure contains the callback expiration time,=20
and two integer values treated as invariants.
When any client executes an operation which would change a file=20
(e.g., StoreData), and in a variety of other situations, the file=20
server invalidates the client's cached copy by executing a call=20
to the CallBack[footnote:
formerly BreakCallback
] procedure in the client's RPC interface. (The call includes in=20
its arguments an AFSCallBack structure for each file being=20
invalidated. However, the value of the passed AFSCallBack is=20
unused [e.g., afs/afs_callback.c:643 ff., openafs-1.5.54]).=20
Between the time of issue and either expiry or receipt of a=20
callback, the client may consider any information it has cached=20
on a file to be consistent with the file server's on-disk copy.=20
Conversely, on receipt of a callback, the client must consider=20
that it knows nothing about the file. Thus the client must=20
re-establish a relationship with the file at the file server=20
before executing any further operations on it.
The AFS callback mechanism obviates the need for clients to send=20
frequent cache validation requests before performing operations=20
on their locally cached copies of objects, reducing network=20
traffic as well as file server workload[4]. The callback=20
innovation has been since taken up, with variations, by other=20
distributed file system protocols[2, 3, 5].
2.2 Analysis
The AFS callback mechanism reliably notifies clients when=20
information they may have cached becomes invalidated, but omits=20
to send information it trivially knows, ie, the triggering event,=20
that could certainly be used by the client to more efficiently=20
manage cache state.
For example, consider the case where 2 clients A and B are=20
interested in a file F, each having read chunks 1-15 into cache.=20
Now another client C initiates a change in the file, writing a=20
new state to chunk 45. This event increments the dataversion of=20
the file, and triggers a callback to A and B. (C, because it=20
initiated the change, is not called back.) On receipt of the=20
callback, A and B must issue FetchStatus requests on F to acquire=20
its current status information, including its current data=20
version. Since the data version of F has increased, any chunks of=20
F which A or B has cached are invalidated, including 30 chunks=20
correctly cached. Should A or B remain interested, it must=20
refetch these chunks (up to 2 megabytes of data, in this case).=20
This scenario will occur reasonably often in environments where=20
mutable data is common, and a related scenario involving=20
directory entries (omitted for brevity) is much more common. In=20
these cases, an AFS callback mechanism capable of sending=20
triggering event information with the callback would have=20
facilitated a more efficient result, at small marginal cost. In=20
another set of scenarios where a client A has changed data in a=20
file invalidated by non-overlapping stores by B, a revised=20
mechanism would be capable of delivering a correct result,=20
whereas a correct result would be impossible with the mechanism=20
in AFS today. (In the AFS-3 callback model, either As or Bs=20
changes must be rejected. In the extended callback model, the=20
range-based invalidate mechanism means that As and Bs changes=20
will be merged, as they are disjoint.)
The justification for sending minimal information with the=20
callback is presumably to minimize the execution cost of the=20
callback procedure. The increased cost of sending a limited but=20
informative callback notification to clients, relative to sending=20
an uninformative one, is small. Analysis of the OpenAFS file=20
server code reveals that the file server always has the=20
information that would logically be sent as extended callback=20
information in response to file operations (e.g., file ranges=20
affected by StoreData operations, or changed entries for various=20
directory modification operations).=20
For these reasons, enhancement of the AFS callback interface to=20
supply triggering event information seems likely to improve both=20
correctness and performance of AFS implementations, and=20
experimental implementation and profiling appear justified.
3 Extended Callback Interface
3.1 Backward Compatibility
AFS clients will indicate their preference to receive extended=20
callback notifications through a new client capability flag:
const CLIENT_CAPABILITY_EXT_CALLBACK =3D 0x0002;
3.2 Interface Changes
3.2.1 Procedures
We propose a new procedure ExtendedCallback in the client's RPC=20
interface. The ExtendedCallBack procedure arguments consist of a=20
HostIdentifier containing the UUIDs of the sending fileserver and=20
of its cell, and a (variable-length) sequence of AFSXCBInvocation=20
structures. And AFSXCBInvocation represents a (variable-length)=20
sequence of AFSExtendedCallBack events on one AFSFid at Server.=20
One invocation of the ExtendedCallBack procedure can thus deliver=20
up to AFSXCBMAX event notifications on each of up to AFSXCBMAX=20
fids. An OUT-direction sequence of variant=20
AFSExtendedCallBackResult structures is added for future callback=20
notification styles (e.g., locks, delegations) which may return=20
structured data on receipt of notifications:
proc ExtendedCallBack(
IN HostIdentifier *Server,
IN AFSXCBInvocationSeq *Invocations_Array,
OUT AFSExtendedCallBackRSeq *CallBack_Result_Array
) multi =3D 65540;
3.2.2 Constants<sub:Constants>
All Sequences
The AFSXCBMAX constant is the maximum allowed length for=20
AFSXCBInvocation and AFSExtendedCallBack sequences:
const AFSXCBMAX =3D 512;
AFSXCBInvocation
The following constants are flag values are used as flag values=20
on AFSXCBInvocation instances:
const AFSCB_IFlag_SOrigin =3D 1; /* callbacks on this invocation=20
have a single origin host */
const AFSCB_IFlag_Release =3D 2; /* this invocation was triggered=20
by a volume release */
AFSExtendedCallBack
As detailed in section [sub:Data-Types], an AFSExtendedCallback=20
is an XDR union, discriminated on the callback event type. The=20
following callback event types are defined:
const AFSCB_Event_Cancel =3D 1; /* extended */
const AFSCB_Event_StoreData =3D 2; /* data in file changed */
const AFSCB_Event_StoreACL =3D 3; /* ACL changed on vnode */
const AFSCB_Event_StoreStatus =3D 4; /* status stored on vnode */
const AFSCB_Event_CreateFile =3D 5; /* file created in directory=20
vnode */
const AFSCB_Event_MakeDir =3D 6; /* dir created in directory vnode=20
*/
const AFSCB_Event_Symlink =3D 7; /* symlink created in directory=20
vnode */
const AFSCB_Event_Link =3D 8; /* hard link created in directory=20
vnode */
const AFSCB_Event_RemoveFile =3D 9; /* file removd from directory=20
vnode */
const AFSCB_Event_RemoveDir =3D 10; /* dir removed from directory=20
vnode */
const AFSCB_Event_Rename =3D 11; /* object renamed (moved) */
const AFSCB_Event_Deleted =3D 12; /* object no longer exists, ex=20
object */
const AFSCB_Event_ReleaseLock =3D 13; /* traditional AFS lock=20
released */
A flag constant is provided to indicate callback cancellation=20
along with an extended notification message of any of the above=20
types:
const AFSCB_Flag_Cancel =3D 1; /* Callback promise is cancelled */
The following constants indicate reasons for cancellation, when=20
(Flags & AFSCB_Flag_Cancel)
const AFSCB_Cancel_Shutdown =3D 1;
const AFSCB_Cancel_CallbackGC =3D 2;
const AFSCB_Cancel_VolumeOffline =3D 3;
const AFSCB_Cancel_VolumeMoved =3D 4;
const AFSCB_Cancel_LostMyMind =3D 5;
const AFSCB_Cancel_IHateYou =3D 6;=20
The following constants indicate direction (from or to called=20
back FID) in the atomic AFSCB_Event_Rename notification:
const AFSCB_Rename_From =3D 1;
const AFSCB_Rename_To =3D 2;
AFSExtendedCallBackResult
The following constants a provided as descriminator for the=20
AFSCB_ResultData union:
const AFSCB_Result_NoResult =3D 1; /* void result type */
const AFSCB_Result_Diag =3D 2; /* diagnostic result type (string)=20
*/
Sequence Types
The following sequences are defined, and are used to construct=20
the arguments for the ExtendedCallBack procedure:
typedef AFSExtendedCallBack AFSExtendedCallBackSeq<AFSXCBMAX>;
typedef AFSExtendedCallBackResult=20
AFSExtendedCallBackRSeq<AFSXCBMAX>;
3.2.3 Data Types<sub:Data-Types>
HostIdentifier
A HostIdentifier structure contains the unique server and cell=20
UUIDs of a specific host in some AFS cell.
struct HostIdentifier {
afsUUID ServerUuid;
afsUUID CellUuid;
};
AFSXCBInvocation
The AFSXCBInvocation data type represents a sequence of 1 or more=20
callback events on one fid. The enclosed AFSExtendedCallBack=20
objects MUST be in DataVersion order.
Fid
Fid is the fid object of the callback sequence.
Flags
Flags provide specializing information about the invocation.
lowDV
The lowest data version of Fid at all events in the sequence.
highDV
The highest data version of Fid at all events in the sequence.
ExpirationTime
ExpirationTime indicates a new expiration time for the receiving=20
client's callback on fid. And ExpirationTime of 0 indicates no=20
change in ExpirationTime.
CallBacks_Array
A sequence of 0 or more AFSExtendedCallBack notifications on FID.
struct AFSXCBInvocation {
AFSFid Fid;
afs_uint32 Flags;
afs_uint64 lowDV; /* lowest DV at invocation */
afs_uint64 highDV; /* highest */
afs_uint64 ExpirationTime; /* new expiration, or 0 if=20
unchanged */
AFSExtendedCallBackSeq CallBacks_Array;
};
AFSExtendedCallback
The AFSExtendedCallBack data type represents a single callback=20
event on some fid, that of its containing AFSXCBInvocation when=20
sent with an ExtendedCallback RPC.=20
Flags and ExtraFlags
Flags and ExtraFlags (added for future expansion) provide=20
possibly event-specific information.=20
Origin
Origin is the UUID of the host or server which originated the=20
event, ie, the client whose operation on fid triggered some=20
event, in the typical case. If the origin is unknown to the=20
server or would not be meaningful, it MAY send the null UUID.=20
NCoalesced
As specified later in this document, certain operations (ie,=20
StoreData, StoreStatus) MAY be regarded by the file server as=20
idempotent and sent as one callback. NCoalesced indicates the=20
number of equivalent or combined operations coalesced on the=20
event, or 0 if the event is singular.=20
DataVersion
DataVersion is the (possibly updated) data version of fid at the=20
completion of the operation which triggered the event.=20
Considering coalescing, DataVersion is the data version at the=20
completion of the first event in the coalesced sequence.=20
Data
Data is an object of the discriminated union type=20
AFSCB_NotificationData:
struct AFSExtendedCallBack {
afs_uint32 Flags;
afs_uint32 ExtraFlags;
afsUUID Origin; /* originator of changes */
afs_uint32 NCoalesced; /* calls combined on this */
afs_uint64 DataVersion;
AFSCB_NotificationData Data;
};
A non-zero value in Flags for the AFSCB_Flag_Cancel bit indicates=20
cancellation of the callback upon receipt of the message. In that=20
event, a non-zero value of ExtraFlags indicates the reason for=20
the cancellation.
AFSCBFileStatus
The AFSCBFileStatus structure is a reduced-footprint=20
AFSFetchStatus replacement intended to communicate changed vnode=20
information in response to StoreData operations:
struct AFSCBFileStatus {
afs_uint32 LinkCount;
afs_uint64 ClientModTime;
};
AFSCBDirStatus
The AFSCBDirStatus structure is a reduced-footprint=20
AFSFetchStatus replacement intended to communicate changed vnode=20
information in response to directory change operations:
struct AFSCBDirStatus {
afs_uint32 LinkCount;
afs_uint64 ClientModTime;
};
AFSCB_NotificationData
AFSCB_NotificationData is a union discriminated by callback event=20
type, ie, its value may be any of the constants defined in=20
section [sub:Constants].
union AFSCB_NotificationData switch (afs_uint32 Event_Type) {
case AFSCB_Event_StoreData:
AFSCB_Data_StoreData u_store_data;
case AFSCB_Event_StoreACL:
void;
case AFSCB_Event_StoreStatus:
AFSCB_Data_StoreStatus u_store_status;
case AFSCB_Event_CreateFile:
AFSCB_Data_CreateFile u_create_file;
case AFSCB_Event_MakeDir:
AFSCB_Data_MakeDir u_make_dir;
case AFSCB_Event_Symlink:
AFSCB_Data_Symlink u_symlink;
case AFSCB_Event_Link:
AFSCB_Data_Link u_link;
case AFSCB_Event_RemoveFile:
AFSCB_Data_RemoveFile u_remove_file;
case AFSCB_Event_RemoveDir:
AFSCB_Data_RemoveDir u_remove_dir;
case AFSCB_Event_Rename:
AFSCB_Data_Rename u_rename;
case AFSCB_Event_Deleted:
void;
case AFSCB_Event_ReleaseLock:
AFSCB_Data_Lock u_lock;
case AFSCB_Event_Cancel:
void;
};
The types for the variant member u_data are enumerated and=20
discussed in detail in section [sub:Callback-Invocations].=20
3.3 Semantic Changes
A file server MAY send traditional callback messages, with=20
traditional semantics, to any AFS client in response to any=20
event. A file server MAY send extended callback notifications to=20
any client which has announced the capability to use the extended=20
interface, with the following semantics:
=E2=80=A2 extended callback notification messages, in general, preserve=20
the file server's callback promise to send further=20
notifications for the called-back FID
=E2=80=A2 the file server MAY revoke the callback promise with any=20
extended callback notification message, by setting the=20
AFSCB_Flag_Cancel bit in the Flags member of the=20
AFSExtendedCallback structure
=E2=80=A2 the AFSCB_Event_Cancel message is similar to a traditional AFS=20
callback, breaking the callback promise, and requesting the=20
client not request further status on the FID
3.3.1 DataVersion Rule
The various extended callback notification messages include=20
information a client may use to selectively invalidate or=20
reconstruct its cache. In interpreting each message, the client=20
MUST observe the dataversion rule, which states:
If the client's cached DataVersion is DataVersion or=20
(DataVersion-1), the client MAY invalidate or update its cache=20
using the type-dependent information contained in the message. In=20
all other cases, the client MUST regard the message as equivalent=20
to a traditional AFS callback.
The semantics of specific callback events are enumerated in=20
section [sub:Callback-Invocations].
3.4 Callback Invocations<sub:Callback-Invocations>
The various extended callback notification types generally=20
respond to specific events at the file server, but present a view=20
of it relevant to a specific callback promise at one client. In=20
one case (ie, AFSCB_Event_Rename), the file server is sending=20
notification of an event which effects two FIDs, either or both=20
of which may be cached by the receiving client. A structure of=20
type AFSExtendedCallback is sent with each extended callback=20
notification message, as noted above. Unless otherwise noted, FID=20
is the FID of the object that is the subject of the callback.=20
3.4.1 AFSXCBInvocation
Origin
A file server MAY omit to send extended callback notifications=20
triggered by a file operation to the client host which originated=20
the change. (Omission to send such callbacks has been the general=20
behavior of AFS file servers.) A client MUST be prepared to=20
appropriately process (or ignore) callbacks for which its own=20
UUID is the Origin.
ExpirationTime
The new expiration time asserted for the server's callback=20
promise, not necessarily different from the existing expiration=20
cached by the client.
3.4.2 AFSExtendedCallback
The members of the AFSExtendedCallback structures are to be=20
interpreted as follows:
Flags
If the 1-bit (AFSCB_Flag_Cancel) is set, the notification effects=20
a callback break. The client may make use of the information sent=20
with the message.
ExtraFlags
If (Flags & AFSCB_Flag_Cancel), a non-zero value for ExtraFlags=20
indicates the reason for cancellation.
DataVersion
The value of DataVersion at completion of the event of which the=20
client is being notified. Considering coalescing, the new data=20
version after completion of all events summarized at this=20
callback is (DataVersion+NCoalesced).
Data
The message-specific data for this notification.
3.4.3 Reasons for Cancellation
The following reasons for cancellation are defined:
AFSCB_Cancel_Shutdown
The server or service is shutting down.
AFSCB_Cancel_CallbackGC
Callback has been disposed during periodic garbage collection.
AFSCB_Cancel_VolumeOffline
The volume associated with FID is now offline.
AFSCB_Cancel_VolumeMoved
The volume associated with FID has moved.
AFSCB_Cancel_LostMyMind
The server may be having problems related to provisioning an=20
insufficient number of callback structures.
AFSCB_Cancel_IHateYou
Callback has been administratively revoked.
3.4.4 ExtendedCallback Procedure
Extended callbacks are delivered through a new ExtendedCallback=20
procedure.
proc ExtendedCallBack(
IN HostIdentifier *Server,
IN AFSCBFids *Fids_Array,
IN AFSExtendedCallBackSeq *CallBacks_Array,
OUT AFSExtendedCallBackRSeq *CallBack_Result_Array
) multi =3D 65540;
ExtendedCallback is modelled on the traditional CallBack=20
procedure, but adds UUIDs uniquely identifying the file server=20
host.
3.4.5 Callback Coalescing
A server implementation electing to deliver extended callback=20
notifications asynchronously MAY, in addition, coalesce sequences=20
of effectively-simultaneous notifications to a single client.=20
This provision avoids performance regression in scenarios where a=20
single logical event or operation would otherwise trigger=20
potentially many notification messages per client (for example,=20
many near-simultaneous CreateFile operations in a single=20
directory as might occur on expansion of a tar archive, or many=20
near-simultaneous stores appending to a single file).[footnote:
The callback coalescing concept is re-introduced following=20
discussions at the 2008 AFS and Kerberos Best Practices Workshop.
]
``Effectively simultaneous'' is defined to mean, ``happening=20
within a small and administratively specified time window.'' An=20
implementation MUST permit windows as small as 100ms and as large=20
as a small number of seconds. An implementation MAY define make=20
window selection adaptive.=20
Call Consolidation (Sequences of Notifications)
A server implementation electing to deliver extended callback=20
notifications asynchronously MAY coalesce any sequence of=20
effectively simultaneous notifications into sequences of=20
AFSExtendedCallBack objects enclosed in one AFSXCBInvocation=20
object. Any number of such callbacks may be combined, up to the=20
limit of AFSXCBMAX.
Coalescing of Equivalent Notifications
A server implementation electing to deliver extended callback=20
notifications asynchronously MAY coalesce a sequence of=20
effectively simultaneous and equivalent notifications to the same=20
client into a single callback in a notification message. The=20
following combinations of operations are explicitly permitted:
=E2=80=A2 sequences of AFSCB_EventStoreAcl notifications on FID from a=20
single Origin MAY be delivered as a single notification
=E2=80=A2 sequences of AFSCB_EventStoreStatus notifications on FID from a=
=20
single Origin MAY be delivered as the single notification of=20
the most recently stored status
=E2=80=A2 sequences of AFSCB_Event_StoreData notifications on FID from a=20
single Origin at adjacent or overlapping byte ranges MAY=20
deliver a single notification at the consolidated range
3.4.6 Asynchronous Delivery
A server implementation MAY deliver extended callback=20
notifications asynchronously with respect to the operation which=20
triggered the notification.
Traditionally, AFS callbacks have been delivered synchronously,=20
in the sense that the callback invalidate invocation has been=20
required to be delivered to eligible clients before the=20
triggering operation/RPC completes). The primary effect of=20
synchronous callback delivery is simplication of the fileserver,=20
but there are potential implications for callback consistency.=20
This proposal asserts that the resulting affect on cache=20
consistency across clients is negligible, because, in particular=20
with sync-on-close semantics, AFS clients are permitted to=20
independently and disjointly cache changes for arbitrary periods=20
of time. Hence synchronous callback execution does not provide a=20
true guarantee of distributed data or metadata consistency (while=20
it does impose a potential performance penalty for mutating=20
operations). Synchronous callback delivery does bound the scope=20
of submitted changes not yet visible to interested, connected=20
clients of one fileserver to those being concurrently executed=20
and invalidated by the sum of its executing threads. For=20
asynchronous callback delivery, we propose that this guarantee is=20
equivalent to a bound on the total number of outstanding=20
asynchronous callback events. We propose that an implementation=20
MUST enforce an implementation-specific limit on the number of=20
outstanding callbacks, falling back to synchronous delivery if=20
the limit is exceeded.
3.4.7 AFSCB_Event_StoreData
The notification is sent in response to a successful StoreData=20
RPC on FID. A structure of type AFSCB_Data_StoreData is sent with=20
the message.
struct AFSCB_Data_StoreData {
afs_uint64 StoreOffset;
afs_uint64 StoreLength;
afs_uint64 Length;
AFSCBFileStatus FileStatus;
};
StoreLength bytes were stored starting at position StoreOffset in=20
FID. Length is the current file length and FileStatus contains=20
the modification time of FID following the operation. The client=20
must regard cached file data in the range [StoreOffset,=20
StoreOffset+StoreLength) as invalidated, and may regard data=20
outside that range as up-to-date. The client MUST discard=20
undirtied cached data in the invalidated range. The client MAY=20
send dirtied data in the invalidated range to the file server=20
prior to discarding (as allowed in current AFS semantics).
3.4.8 AFSCB_Event_StoreACL
ACL and/or access information cached by the client for FID, if=20
any, is invalidated.
3.4.9 AFSCB_Event_StoreStatus
A StoreStatus RPC was successfully executed on FID. A structure=20
of type AFSFetchStatus is sent with the message.
struct AFSCB_Data_StoreStatus {
struct AFSFetchStatus Status;
};
Status is the new AFSFetchStatus of FID, ie, the message=20
communicates the current status information of FID.[footnote:
This is changed from earlier drafts.
]
3.4.10 AFSCB_Event_CreateFile
A file has been created in the vnode corresponding to FID. A=20
structure of type AFSCB_Data_CreateFile is sent with the message.
struct AFSCB_Data_CreateFile {
string Name<AFSNAMEMAX>;
AFSFid Fid;
AFSFetchStatus FidStatus;
AFSCBDirStatus DirStatus;
};
Name and Fid are, respectively, the name and FID of the created=20
file. FidStatus is the AFSFetchStatus of the created file, and=20
DirStatus the current modification time and link count of FID, at=20
the completion of the call.
3.4.11 AFSCB_Event_MakeDir
A directory has been created in the vnode corresponding to FID. A=20
structure of type AFSCB_Data_MakeDir is sent with the message.
struct AFSCB_Data_MakeDir {
string Name<AFSNAMEMAX>;
AFSFid Fid;
AFSFetchStatus FidStatus;
AFSCBDirStatus DirStatus;
};;
Name and Fid are, respectively, the name and FID of the created=20
directory. FidStatus is the AFSFetchStatus of the created=20
directory, and DirStatus the current modification time and link=20
count of FID, at the completion of the call.
3.4.12 AFSCB_Data_Symlink
A symbolic link has been created in the vnode corresponding to=20
FID. A structure of type AFSCB_Data_Symlink is sent with the=20
message.
struct AFSCB_Data_Symlink {
string Name<AFSNAMEMAX>;
AFSFid Fid;
string LinkContents<AFSPATHMAX>;
AFSFetchStatus FidStatus;
AFSCBDirStatus DirStatus;
};
Name is the name of the symbolic link. Fid is its AFSFid. The=20
link points to LinkContents. FidStatus is the AFSFetchStatus of=20
the created symbolic link, and DirStatus the current modification=20
time and link count of FID, at the completion of the call.
3.4.13 AFSCB_Event_Link
A hard link has been created in the vnode corresponding to FID. A=20
structure of type AFSCB_Data_Link is sent with the message.
struct AFSCB_Data_Link {
string Name<AFSNAMEMAX>;
AFSFid LinkTarget;
AFSFetchStatus FidStatus;
AFSCBDirStatus DirStatus;
};
Name is the name of the link. The link is a synonym for=20
LinkTarget. FidStatus is the AFSFetchStatus of the created link,=20
and DirStatus the current modification time and link count of=20
FID, at the completion of the call.
3.4.14 AFSCB_Event_RemoveFile
A file has been removed from the vnode corresponding to FID. A=20
structure of type AFSCB_Data_RemoveFile is sent with the message.
struct AFSCB_Data_RemoveFile {
string Name<AFSNAMEMAX>;
AFSCBDirStatus DirStatus;
};
Name indicates the removed entry. DirStatus the current=20
modification time and link count of FID, at the completion of the=20
call.
3.4.15 AFSCB_Event_RemoveDir
A directory has been removed from the vnode corresponding to FID.=20
A structure of type AFSCB_Data_RemoveDir is sent with the=20
message.
struct AFSCB_Data_RemoveDir {
string Name<AFSNAMEMAX>;
AFSCBDirStatus DirStatus;
};
Name indicates the removed entry. DirStatus the current=20
modification time and link count of FID, at the completion of the=20
call.
3.4.16 AFSCB_Event_Rename
A file or directory has been renamed, ie moved, from or to the=20
vnode corresponding to FID. A structure of type=20
AFSCB_Data_RemoveDir is sent with the message.
const AFSCB_Rename_From =3D 1;
const AFSCB_Rename_To =3D 2;
struct AFSCB_Data_Rename {
afs_uint32 Direction;
string OldName<AFSNAMEMAX>;
string NewName<AFSNAMEMAX>;
AFSCBDirStatus FromStatus;
AFSCBDirStatus ToStatus;
};
Direction indicates whether FID is the source or the destination=20
directory of the move. OldName is the name of the object in its=20
old location, NewName the name of the object in its new location.=20
FromStatus is the current modification time and link count of the=20
source directory vnode, and ToStatus is the current modification=20
time and link count of the destination directory vnode, and=20
FidStatus the at the completion of the call.=20
To preserve atomicity, the AFSCB_Data_Rename message is=20
constructed so that changes to cached copies of both the source=20
and directory vnodes may be recovered from a single notification.=20
If a client owns callbacks for both the source and destination=20
FIDs, a file server MAY elect to send only one notification, for=20
either the source or the destination FID.
3.4.17 AFSCB_Event_Deleted
The object corresponding to FID not longer exists, and so may no=20
longer be cached. It is an ex-object. (I.e., the client MUST=20
discard any information it has cached about FID.)
3.4.18 AFSCB_Event_ReleaseLock
A traditional AFS whole-file lock has been released on FID. A=20
structure of type AFSCB_Data_Lock is sent with the message.
struct AFSCB_Data_Lock {
afs_uint32 LockType;
};
LockType is the type of the lock released.
Receipt of an AFSCB_Event_ReleaseLock notification in no way=20
implies an intention on the part of a file server to grant a lock=20
on FID to client. Non-receipt of a notification of this type in=20
no way implies non-release of locks that may be held on FID. The=20
file server SHOULD send notifications of this type only to=20
clients which have indicated probable interest in the event,=20
e.g., by having recently requested a lock on FID.
3.5 Callbacks And Read-Only Volume Replicas
Callbacks associated with read-only volume replicas have=20
traditionally been handled specially in AFS. When any file in an=20
RO volume is accessed the AFS file server establishes a single=20
callback promise considered to be on the entire volume. Any event=20
which updates the replica (e.g., vos release) triggers a=20
whole-volume callback break. The whole-volume callback=20
optimization significantly reduced file server memory=20
utilization, which was at a premium in 1988. However, the=20
whole-volume callback is less of an optimization in OpenAFS in=20
2008:
=E2=80=A2 modern AFS file servers have sufficient memory to track=20
millions of callbacks (and do track up to 1 million callbacks=20
at one site we know of, with up to 3 million callback=20
structures available)[8]
=E2=80=A2 whole-volume callback semantics require clients (and the file=20
server) to potentially expend considerable effort=20
re-establishing cache consistency, and so whole-volume=20
callbacks are necessarily a considerable protocol inefficiency=20
for sites relying heavily on AFS replication (in particular,=20
incremental replication now possible in OpenAFS)
For these reasons, we propose that the scope of extended callback=20
information include notifications concerning changes that=20
originate in the release of a volume. We provide the option for=20
the file server to provide whole-volume or per file=20
notifications, at its discretion. We provide the option for the=20
file server to track client interest in specific files (ie, issue=20
per-file callbacks on files in RO volumes), and speculate that=20
this implementation would be preferred, but do not mandate it.
3.5.1 Constants
The following flag constants are added:
const AFSCB_Flag_Release =3D 2;
const AFSCB_Release_WholeVolumeCancel =3D 1;
AFSCB_Flag_Release
In an AFSExtendedCallback instance, (Flags & AFSCB_Flag_Release)=20
indicates a notification in response to the (possibly=20
incremental) release of a read-only replica.=20
AFSCB_Release_WholeVolumeCancel
If additionally (ExtraFlags & AFSCB_Release_WholeVolumeCancel),=20
then the callback invalidates the entire volume, otherwise it is=20
a selective invalidation of just the FIDs in Fids_Array.
3.5.2 Semantic Changes
An AFS file server MAY send selective or whole-volume extended=20
callback notifications. The file server MAY choose to regard=20
files in RO volumes equivalently to files in RW volumes, ie,=20
effectively maintain callback state on them. Alternatively it MAY=20
send selective notifications on any FIDs changed, removed, or=20
added in the volume without regard to client cache state. The AFS=20
client must handle such notifications gracefully.
4 Appendix A: XDR Grammar
/* Cache Manager Capability Flags */
const CLIENT_CAPABILITY_EXT_CALLBACK =3D 0x0002;
/* Host Tracking/Extended Information */
struct HostIdentifier {
afsUUID ServerUuid;
afsUUID CellUuid;
};
/* Extended Callback Information */
/* callback event types, predominantly events on the vnode for=20
* which the callback is being made, but also (e.g., Deleted) side=20
* effects of operations on related vnodes */
const AFSCB_Event_Cancel =3D 1;
const AFSCB_Event_StoreData =3D 2;
const AFSCB_Event_StoreACL =3D 3;
const AFSCB_Event_StoreStatus =3D 4;
const AFSCB_Event_CreateFile =3D 5;
const AFSCB_Event_MakeDir =3D 6;
const AFSCB_Event_Symlink =3D 7;
const AFSCB_Event_Link =3D 8;
const AFSCB_Event_RemoveFile =3D 9;
const AFSCB_Event_RemoveDir =3D 10;
const AFSCB_Event_Rename =3D 11;
const AFSCB_Event_Deleted =3D 12;
const AFSCB_Event_ReleaseLock =3D 13;
/* for use in AFSExtendedCallBack Flags */
const AFSCB_Flag_Cancel =3D 1;
const AFSCB_Flag_Release =3D 2;
/* intended for use in AFSExtendedCallback ExtraFlags,=20
* when (flags & AFSCB_Flag_Cancel), to indicate reason for
* cancellation */
const AFSCB_Cancel_Shutdown =3D 1;
const AFSCB_Cancel_CallbackGC =3D 2;
const AFSCB_Cancel_VolumeOffline =3D 3;
const AFSCB_Cancel_VolumeMoved =3D 4;
const AFSCB_Cancel_LostMyMind =3D 5; /* ran out of callbacks? */
const AFSCB_Cancel_IHateYou =3D 6; /* callback administratively=20
revoked */
/* for use in AFSXCBInvocation Flags */
const AFSCB_IFlag_SOrigin =3D 1;
const AFSCB_IFlag_Release =3D 2;
/* flags intended for use in AFSExtendedCallback ExtraFlags=20
* to indicate RO volume callback events */
const AFSCB_Release_WholeVolumeCancel =3D 1;
/* callback result types */
const AFSCB_Result_NoResult =3D 1;
const AFSCB_Result_Diag =3D 2;
/* differential status to be sent with StoreData msgs */
struct AFSCBFileStatus {
afs_uint32 LinkCount;
afs_uint64 ClientModTime;
};
/* differential status to be sent with directory change msgs */
struct AFSCBDirStatus {
afs_uint32 LinkCount;
afs_uint64 ClientModTime;
};
/* variant data types for AFSCB_Notification_Data */
struct AFSCB_Data_StoreData {
afs_uint64 StoreOffset;
afs_uint64 StoreLength;
afs_uint64 Length;
AFSCBFileStatus FileStatus;
};
struct AFSCB_Data_StoreStatus {
struct AFSFetchStatus Status;
};
struct AFSCB_Data_CreateFile {
string Name<AFSNAMEMAX>;
AFSFid Fid;
AFSFetchStatus FidStatus;
AFSCBDirStatus DirStatus;
};
struct AFSCB_Data_MakeDir {
string Name<AFSNAMEMAX>;
AFSFid Fid;
AFSFetchStatus FidStatus;
AFSCBDirStatus DirStatus;
};
struct AFSCB_Data_Symlink {
string Name<AFSNAMEMAX>;
AFSFid Fid;
string LinkContents<AFSPATHMAX>;
AFSFetchStatus FidStatus;
AFSCBDirStatus DirStatus;
};
struct AFSCB_Data_Link {
string Name<AFSNAMEMAX>;
AFSFid LinkTarget;
AFSFetchStatus FidStatus;
AFSCBDirStatus DirStatus;
};
struct AFSCB_Data_RemoveFile {
string Name<AFSNAMEMAX>;
AFSCBDirStatus DirStatus;
};
struct AFSCB_Data_RemoveDir {
string Name<AFSNAMEMAX>;
AFSCBDirStatus DirStatus;
};
const AFSCB_Rename_From =3D 1;
const AFSCB_Rename_To =3D 2;
struct AFSCB_Data_Rename {
afs_uint32 Direction;
string OldName<AFSNAMEMAX>;
string NewName<AFSNAMEMAX>;
AFSCBDirStatus FromStatus;
AFSCBDirStatus ToStatus;
};
struct AFSCB_Data_Lock {
afs_uint32 LockType;
};
union AFSCB_NotificationData switch (afs_uint32 Event_Type) {
case AFSCB_Event_StoreData:
AFSCB_Data_StoreData u_store_data;
case AFSCB_Event_StoreACL:
void;
case AFSCB_Event_StoreStatus:
AFSCB_Data_StoreStatus u_store_status;
case AFSCB_Event_CreateFile:
AFSCB_Data_CreateFile u_create_file;
case AFSCB_Event_MakeDir:
AFSCB_Data_MakeDir u_make_dir;
case AFSCB_Event_Symlink:
AFSCB_Data_Symlink u_symlink;
case AFSCB_Event_Link:
AFSCB_Data_Link u_link;
case AFSCB_Event_RemoveFile:
AFSCB_Data_RemoveFile u_remove_file;
case AFSCB_Event_RemoveDir:
AFSCB_Data_RemoveDir u_remove_dir;
case AFSCB_Event_Rename:
AFSCB_Data_Rename u_rename;
case AFSCB_Event_Deleted:
void;
case AFSCB_Event_ReleaseLock:
AFSCB_Data_Lock u_lock;
case AFSCB_Event_Cancel:
void;
};
struct AFSExtendedCallBack {
afs_uint32 Flags;
afs_uint32 ExtraFlags;
afsUUID Origin; /* originator of changes */
afs_uint32 NCoalesced; /* calls [StoreData] combined on this=20
*/
afs_uint64 DataVersion;
AFSCB_NotificationData Data;
};
const AFSXCBMAX =3D 512;
typedef AFSExtendedCallBack AFSExtendedCallBackSeq<AFSXCBMAX>;
struct AFSXCBInvocation {
AFSFid Fid;
afs_uint32 Flags;
afs_uint64 lowDV; /* lowest DV at invocation */
afs_uint64 highDV; /* highest */
afs_uint64 ExpirationTime; /* new expiration, or 0 if=20
unchanged */
AFSExtendedCallBackSeq CallBacks_Array;
};
typedef AFSXCBInvocation AFSXCBInvocationSeq<AFSXCBMAX>;
/* Forward-looking union for callback results */
union AFSCB_ResultData switch (afs_uint32 Result_Type) {
case AFSCB_Result_NoResult:
void;
case AFSCB_Result_Diag:
string msg<30>;
};
/* extended callback result structure */
struct AFSExtendedCallBackResult {
afs_uint32 Flags;
afs_uint32 ExtraFlags;
AFSCB_ResultData Data;
};
typedef AFSExtendedCallBackResult=20
AFSExtendedCallBackRSeq<AFSXCBMAX>;
proc ExtendedCallBack(
IN HostIdentifier *Server,
IN AFSXCBInvocationSeq *Invocations_Array,
OUT AFSExtendedCallBackRSeq *CallBack_Result_Array
) multi =3D 65540;
References
[1] Bradner, S., "Key words for use in RFCs to Indicate=20
Requirement Levels", BCP 14, RFC 2119, March 1997.
[2] Howard, J.H., Kazar, M.L., Menees, S.G., Nichols, D.A.,=20
Satyanarayanan, M., Sidebotham, R.N. and West, M. "Scale and=20
Performance in a Distributed File System" ACM Transactions on=20
Computer Systems, February 1988=20
[3] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame,=20
C., Eisler, M., and D. Noveck, "Network File System (NFS) version=20
4 Protocol", RFC 3530, April 2003.
[4] Edward R Zayas, "AFS-3 Programmer's Reference: File=20
Server/Cache Manager Interface", Transarc Corporation,=20
FS-00-D162, 20th August 1991
[5] Paul J. Leach, Dilip C. Naik. A Common Internet File System=20
(CIFS/1.0) Protocol=20
[http://www.tools.ietf.org/html/draft-leach-cifs-v1-spec-01],=20
1997.
[6] Kazar, Michael Leon, "Synchronization and Caching Issues in=20
the Andrew File System," USENIX Conference Proceedings, USENIX=20
Association, Berkeley, CA, Dallas Winter 1988, pages 27-36.
[7] Lily B. Mummert, Mahadev Satyanarayanan: Large Granularity=20
Cache Coherence for Intermittent Connectivity. USENIX Summer=20
1994: 279-289
[8] Alistair Ferguson. OpenAFS and the Dawn of a New Era. AFS and=20
Kerberos Best Practices Workshop, 2008.
[9] Trond Myklebust. Byte Range Delegations.=20
[https://www3.ietf.org/proceedings/05nov/slides/nfsv4-3.pdf ],=20
November 2006.
--------------040502080901020400030007--