[OpenAFS-win32-devel] New client-side locking implementation

[Eric Williams]

On Tue, 26 Jul 2005, Jeffrey Altman wrote:

> Asanka Herath wrote:
>
> > /* Byte range locks:
> >
> >    The OpenAFS Windows client has to fake byte range locks given no
> >    server side support for such locks.  This is implemented as keyed
> >    byte range locks on the cache manager.  The logistics of generating
> >    useful keys are dealt with at the SMB server and IFS layer.
> >
> >    Keyed byte range locks:
> >
> >    Each cm_scache_t structure keeps track of a list of keyed locks.
> >    The key for a lock is essentially a token which identifies an owner
> >    of a set of locks (referred to as a client).  In the keyed lock
> >    implementation in the cache manager code, a key is represented as
> >    an unsigned 32 bit quantity.  The set of keys used within a
> >    specific cm_scache_t structure form a namespace that has a scope of
> >    just that cm_scache_t structure.  The same key value can be used
> >    with another cm_scache_t structure and correspond to a completely
> >    different client.  However it is advantageous for the SMB or IFS
> >    layer to make sure that there is a 1-1 mapping between client and
> >    keys irrespective of the cm_scache_t.
> >
> >    Assume a client C has key Key(C) (although, since the scope of the
> >    key is a cm_scache_t, the key can be Key(C,S), where S is the
> >    cm_scache_t.  But assume a 1-1 relation between keys and clients).
> >    A byte range (O,L) denotes byte addresses (O) through (O+L-1)
> >    inclusive (a.k.a. [O,O+L-1]).  The function Key(x) is left upto the
> >    SMB and IFS layer to figure out.
>
> In order for there to be a shared cache manager accessed simultaneously
> by both SMB and IFS, there needs to be a single Key() that takes
> sessionID, processID, and FID as input.   The IFS should use a session
> ID  that is unique and would never be considered valid for the SMB.

there need to be two key values associated with each lock.

1. KeyA(processID, FileObject),
   KeyA(processID, FileObject, FID_HASH(FID)), or
   KeyA(processID, FileObject, FID),
   although i see no advantage of using any but the first (as there is a
   one-to-many mapping between FIDs and FileObjects).

2. KeyB(FileLockKey), which, of course, could be equivalent to
   Identity(FileLockKey).  FileLockKey is a parameter of IFS-received lock
   requests as they come from the i/o manager.

lock requests would contain at least these parameters:
 lock_Lock(FID, KeyA, KeyB, LOffset, LLength, LockType)
  - places a lock on the file specified by FID, with the other
    properties as specified, if no conflict occurs.

unlock requests have three variants:
 lock_Unlock(FID, KeyA, KeyB, LOffset, LLength)
  - removes the single lock (if found) on the file specified by FID that
    has properties which match the specified parameters KeyA, KeyB,
    LOffset, LLength.
 lock_UnlockByKeyA(FID, KeyA)
  - removes all locks (if any) on the file specified by FID that have a
    stored KeyA equal to the passed parameter KeyA (all locks that were
    set by the process specified by KeyA on certain FileObject
    [roughly a file HANDLE] to file FID).
 lock_UnlockByKeyB(FID, KeyA, KeyB)
  - removes all locks (if any) on the file specified by FID that have
    stored KeyA equal to passed KeyA and stored KeyB equal to passed KeyB
    (all locks that were set by the process specified by KeyA with LockKey
    KeyB on a certain FileObject of the file specified by FID).

all four lock request forms correspond to lock requests received by the
IFS interface.  the IFS interface would call the four requests with the
parameters as specified.  the SMB interface would call only lock_Lock and
lock_Unlock, and would use a constant KeyB.  i did not see any bulk
(key-based) lock request handling code in smb_ReceiveV3LockingX.

> >    The cache manager will set a lock on the AFS file server in order
> >    to assert the locks in S->fileLocks.  If only shared locks are in
> >    place for S, then the cache manager will obtain a LockRead lock,
> >    while if there are any exclusive locks, it will obtain LockWrite
> >    lock.  If the exclusive locks are all released while the shared
> >    locks remain, then the cache manager will downgrade the lock from
> >    LockWrite to LockRead.
> >

(all of this modified to read 'L->KeyA == KeyA, L->KeyB == KeyB', where
appropriate)

> >    A client C can read range (O,L) of cm_scache_t S iff:
> >
> >    1. for all _a_ in (O,L), one of the following is true:
> >
> >        1.1 There does NOT exist a VALID or LOST lock L in S->fileLocks
> >          such that _a_ in (L->LOffset,L->LLength) (IOW: byte _a_ of S
> >          is unowned)
> >
> >        1.2 There is an ACTIVE lock L in S->fileLocks such that:
> >          L->key == Key(C) && _a_ in (L->LOffset,L->LLength) (IOW: byte
> >          _a_ of S is owned by C under lock L)
> >
> >        1.3 There is an ACTIVE lock L in S->fileLocks such that
> >          _a_ in (L->LOffset,L->LLength) && L->LockType is shared
> >          (IOW: byte _a_ of S is shared under lock L)
> >
> >    A client C can write range (O,L) of cm_scache_t S iff:
> >
> >    2. for all _a_ in (O,L), one of the following is true:
> >
> >        2.1 Byte _a_ of S is unowned (as above)
> >
> >        2.2 Byte _a_ of S is owned by C under lock L (as above) AND
> >          L->LockType is exclusive.
> >
> >    A client C can lock range (O,L) of cm_scache_t S iff:
> >
> >    3 for all _a_ in (O,L), the following is true:
       3. for all _a_ in (0,L), one of the following is true:
> >
> >        3.1 Byte _a_ is unowned (as above)

           3.2 There is an ACTIVE lock L in S->fileLocks such that
             _a_ in (L->LOffset, L->LOffset+L->LLength) &&
             L->LockType is shared && LockType is shared.

(also read as 'there is no exclusive lock on the byte, and the requested
type is shared, OR there is no lock on the byte, and the requested type is
exclusive')

> >    A client C can only unlock locks L in S->fileLocks which have
> >    L->key == Key(C).

	where L->KeyA == KeyA, and other parameters (KeyB, Offset, etc.)
	match as appropriate for the type of unlock requested.

> >
> >    Note:
> >
> >    1. A lock L is VALID if it exists and is not INVALID.  A lock that
> >       is WAITING is still valid.  A lock will be in a waiting state
> >       (WAITING) from the time the cache manager accepts the lock until
> >       the AFS file server acknowledges the lock.
>
> ... is VALID if it exists and is not INVALID or LOST.
>
> >    2. A lock L is ACTIVE if it is VALID is not in a waiting
> >       state. I.e. the cache manager has asserted the lock with the AFS
> >       file server.
>
>
> >    3. A lock L is LOST if it was formerly VALID or ACTIVE but the
> >       cache manager failed to extend the lock.  The cache manager
> >       rechecks locks once every minute and extends them.  If this is
> >       not done for 5 minutes, the AFS file server will release the
> >       lock.  Once release, the lock cannot be re-obtained without
> >       verifying that the contents of the file hasn't been modified
> >       since the time the lock was released.  Doing so may cause data
> >       corruption.
> >
> >    The representation and invariants are as follows:
> >
> >    - Each cm_scache_t structure keeps:
> >
> >        - A queue of byte-range locks (cm_scache_t::fileLocks) which
> >          are of type cm_file_lock_t.
> >
> >        - A record of the highest server-side lock that has been
> >          obtained for this object (cm_scache_t::serverLock), which is
> >          one of (-1), LockRead, LockWrite.
> >
> >        - A count of VALID exclusive and shared locks that are in the
> >          queue (cm_scache_t::sharedLocks and
> >          cm_scache_t::exclusiveLocks)
> >
> >    - Each cm_file_lock_t structure keeps:
> >
> >        - The type of lock (cm_file_lock_t::LockType)
> >
> >        - The key associated with the lock (cm_file_lock_t::key)
> >
           - The process/session hash associated with the lock
           - The unlocking/access key associated with the lock

> >        - The offset and length of the lock (cm_file_lock_t::LOffset
> >          and cm_file_lock_t::LLength)
> >
> >        - The state of the lock.  Whether it is VALID or INVALID, and
> >          if VALID, whether it is in a WAITING state or not.  If the
> >          lock is VALID and not WAITING, then it's ACTIVE.
>
>          - Time of issuance or last renewal
>
>
> >    Semantic invariants:
> >
> >        I1. The number of VALID locks in S->fileLocks are
> >            (S->sharedLocks + S->exclusiveLocks)
> >
> >        I2. If L1 and L2 are both VALID locks in S->fileLocks, then L1
> >            and L2 do NOT overlap. (enforced by 3.1 above)
> >
> >    External invariants:
> >
> >        I3. S->serverLock is the lock that we have asserted with the
> >            AFS file server for this cm_scache_t.
> >
> >        I4. S->serverLock == LockRead iff there is at least one ACTIVE
> >            shared lock, but no ACTIVE exclusive locks.
> >
> >        I5. S->serverLock == LockWrite iff there is at least one ACTIVE
> >            exclusive lock.
> >
> >    --asanka
> >  */
>
> Jeffrey Altman
>

eric