[OpenAFS-win32-devel] New client-side locking implementation
Jeffrey Altman
jaltman@secure-endpoints.com
Tue, 26 Jul 2005 01:56:35 -0400
This is a cryptographically signed message in MIME format.
--------------ms060906050502020604000306
Content-Type: multipart/mixed;
boundary="------------040303000802050601060403"
This is a multi-part message in MIME format.
--------------040303000802050601060403
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Asanka Herath wrote:
> Hello,
>
> The following is a draft of the new file locking implementation that will
> replace the current not-very-functional locks. Discuss.
One of the things we discovered when examining the locking code
carefully is that on both Windows and Unix the cache manager is actually
broken when it comes to deal with locks. At the current time the client
will obtain a lock and renew it on a regular basis. If the server does
not see a renewal in five minutes it drops the lock. Now let's consider
the case where there is a network failure preventing communication for
five minutes. In this case the server drops the lock and the client
receives either "network failure" or EINVAL (a server returns this error
to indicate the renewal is for a dropped lock).
At this point the client must attempt to re-establish the lock. But
wait, what if the contents of the file changed while the lock was
dropped? In this case the client is in deep trouble. Perhaps this is
why the current Windows and Unix cache manager code always ignores the
success or failure of a lock renewal. Unfortunately this means that
any application that relies on locks is susceptible to data corruption
when using AFS.
We have attempted to deal with this in the following manner:
(1) if the lock was dropped but the callback is still valid that means
the file has not changed and we can attempt to re-establish the lock.
If the client is successful, no harm was done.
(2) if the lock was dropped but the callback is not valid that means the
file has potentially changed and we can no longer assume the validity of
the data. at this point we mark the lock as being lost and prevent any
further reads from or writes to the file until the file is closed and
re-opened.
Given the current semantics of the afs cache manager with regards to
locks, the cache manager upon receiving an error from the Renew Lock RPC
might as well overwrite the file with random data. At least that way
the user would know the data was trashed.
> /* Byte range locks:
>
> The OpenAFS Windows client has to fake byte range locks given no
> server side support for such locks. This is implemented as keyed
> byte range locks on the cache manager. The logistics of generating
> useful keys are dealt with at the SMB server and IFS layer.
>
> Keyed byte range locks:
>
> Each cm_scache_t structure keeps track of a list of keyed locks.
> The key for a lock is essentially a token which identifies an owner
> of a set of locks (referred to as a client). In the keyed lock
> implementation in the cache manager code, a key is represented as
> an unsigned 32 bit quantity. The set of keys used within a
> specific cm_scache_t structure form a namespace that has a scope of
> just that cm_scache_t structure. The same key value can be used
> with another cm_scache_t structure and correspond to a completely
> different client. However it is advantageous for the SMB or IFS
> layer to make sure that there is a 1-1 mapping between client and
> keys irrespective of the cm_scache_t.
>
> Assume a client C has key Key(C) (although, since the scope of the
> key is a cm_scache_t, the key can be Key(C,S), where S is the
> cm_scache_t. But assume a 1-1 relation between keys and clients).
> A byte range (O,L) denotes byte addresses (O) through (O+L-1)
> inclusive (a.k.a. [O,O+L-1]). The function Key(x) is left upto the
> SMB and IFS layer to figure out.
In order for there to be a shared cache manager accessed simultaneously
by both SMB and IFS, there needs to be a single Key() that takes
sessionID, processID, and FID as input. The IFS should use a session
ID that is unique and would never be considered valid for the SMB.
> The cache manager will set a lock on the AFS file server in order
> to assert the locks in S->fileLocks. If only shared locks are in
> place for S, then the cache manager will obtain a LockRead lock,
> while if there are any exclusive locks, it will obtain LockWrite
> lock. If the exclusive locks are all released while the shared
> locks remain, then the cache manager will downgrade the lock from
> LockWrite to LockRead.
>
> A client C can read range (O,L) of cm_scache_t S iff:
>
> 1. for all _a_ in (O,L), one of the following is true:
>
> 1.1 There does NOT exist a VALID or LOST lock L in S->fileLocks
> such that _a_ in (L->LOffset,L->LLength) (IOW: byte _a_ of S
> is unowned)
>
> 1.2 There is an ACTIVE lock L in S->fileLocks such that:
> L->key == Key(C) && _a_ in (L->LOffset,L->LLength) (IOW: byte
> _a_ of S is owned by C under lock L)
>
> 1.3 There is an ACTIVE lock L in S->fileLocks such that
> _a_ in (L->LOffset,L->LLength) && L->LockType is shared
> (IOW: byte _a_ of S is shared under lock L)
>
> A client C can write range (O,L) of cm_scache_t S iff:
>
> 2. for all _a_ in (O,L), one of the following is true:
>
> 2.1 Byte _a_ of S is unowned (as above)
>
> 2.2 Byte _a_ of S is owned by C under lock L (as above) AND
> L->LockType is exclusive.
>
> A client C can lock range (O,L) of cm_scache_t S iff:
>
> 3 for all _a_ in (O,L), the following is true:
>
> 3.1 Byte _a_ is unowned (as above)
>
> A client C can only unlock locks L in S->fileLocks which have
> L->key == Key(C).
>
> Note:
>
> 1. A lock L is VALID if it exists and is not INVALID. A lock that
> is WAITING is still valid. A lock will be in a waiting state
> (WAITING) from the time the cache manager accepts the lock until
> the AFS file server acknowledges the lock.
... is VALID if it exists and is not INVALID or LOST.
> 2. A lock L is ACTIVE if it is VALID is not in a waiting
> state. I.e. the cache manager has asserted the lock with the AFS
> file server.
> 3. A lock L is LOST if it was formerly VALID or ACTIVE but the
> cache manager failed to extend the lock. The cache manager
> rechecks locks once every minute and extends them. If this is
> not done for 5 minutes, the AFS file server will release the
> lock. Once release, the lock cannot be re-obtained without
> verifying that the contents of the file hasn't been modified
> since the time the lock was released. Doing so may cause data
> corruption.
>
> The representation and invariants are as follows:
>
> - Each cm_scache_t structure keeps:
>
> - A queue of byte-range locks (cm_scache_t::fileLocks) which
> are of type cm_file_lock_t.
>
> - A record of the highest server-side lock that has been
> obtained for this object (cm_scache_t::serverLock), which is
> one of (-1), LockRead, LockWrite.
>
> - A count of VALID exclusive and shared locks that are in the
> queue (cm_scache_t::sharedLocks and
> cm_scache_t::exclusiveLocks)
>
> - Each cm_file_lock_t structure keeps:
>
> - The type of lock (cm_file_lock_t::LockType)
>
> - The key associated with the lock (cm_file_lock_t::key)
>
> - The offset and length of the lock (cm_file_lock_t::LOffset
> and cm_file_lock_t::LLength)
>
> - The state of the lock. Whether it is VALID or INVALID, and
> if VALID, whether it is in a WAITING state or not. If the
> lock is VALID and not WAITING, then it's ACTIVE.
- Time of issuance or last renewal
> Semantic invariants:
>
> I1. The number of VALID locks in S->fileLocks are
> (S->sharedLocks + S->exclusiveLocks)
>
> I2. If L1 and L2 are both VALID locks in S->fileLocks, then L1
> and L2 do NOT overlap. (enforced by 3.1 above)
>
> External invariants:
>
> I3. S->serverLock is the lock that we have asserted with the
> AFS file server for this cm_scache_t.
>
> I4. S->serverLock == LockRead iff there is at least one ACTIVE
> shared lock, but no ACTIVE exclusive locks.
>
> I5. S->serverLock == LockWrite iff there is at least one ACTIVE
> exclusive lock.
>
> --asanka
> */
Jeffrey Altman
--------------040303000802050601060403
Content-Type: text/x-vcard; charset=utf-8;
name="jaltman.vcf"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="jaltman.vcf"
begin:vcard
fn:Jeffrey Altman
n:Altman;Jeffrey
org:Secure Endpoints Inc.
adr:;;255 W 94TH ST PHB;NEW YORK;NY;10025;United States
email;internet:jaltman@secure-endpoints.com
title:President
tel;work:+1 212 769-9018
x-mozilla-html:TRUE
url:http://www.secure-endpoints.com
version:2.1
end:vcard
--------------040303000802050601060403--
--------------ms060906050502020604000306
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJXzCC
AwowggJzoAMCAQICAw7NrTANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJaQTElMCMGA1UE
ChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNv
bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDUwNTI3MTc0NzU3WhcNMDYwNTI3MTc0NzU3
WjBzMQ8wDQYDVQQEEwZBbHRtYW4xFTATBgNVBCoTDEplZmZyZXkgRXJpYzEcMBoGA1UEAxMT
SmVmZnJleSBFcmljIEFsdG1hbjErMCkGCSqGSIb3DQEJARYcamFsdG1hbkBzZWN1cmUtZW5k
cG9pbnRzLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKjPyrF+rdjOUSK/
bWwZHdx5p1+y6iiCd4vvYEVDxouYFp5C/fZEWm5n45ubBUbMSUI1MAZN6ooEoH09UTj6BXhM
S8B987ls81dKOIUphTF2jOzq8gsFmeA15yHMRAD20LqUWeLyvYk8FCNQw+dsKMMhX+WdsxOm
RY/1jPkJL6oN8kEwoUFkOX9/OfWWh6oFnV6faiEHUKDMFubsb9X0KVD8iIeR7Cxz7i4kXqRX
wMlp2fyoxcDIJrBaTY8nA++g3p34IkWt1a5po6g683nIgSnGpwYIwuJheBqSEZfLYWa+1KdD
6Sn27Ud94GqUvPVG5jC6zVC5EJ2aWuoAu+nNuV8CAwEAAaM5MDcwJwYDVR0RBCAwHoEcamFs
dG1hbkBzZWN1cmUtZW5kcG9pbnRzLmNvbTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBAUA
A4GBADtvO//tjiAV6VJGtoNtrl34mB5jGyGTiotzw8riB6zz0GvY11bcWDmp6JKif+pVG+8L
IySDosbuva13qu2HwYUxBmWc7CoNd2k9kRlcrfbDUTTrGOZK8qyqNqT3gQZTAa9ZnUI0su9G
y/n2o5bQcaYdqR3htNrpvdLSPOWhILOXMIIDCjCCAnOgAwIBAgIDDs2tMA0GCSqGSIb3DQEB
BAUAMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBM
dGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQTAeFw0w
NTA1MjcxNzQ3NTdaFw0wNjA1MjcxNzQ3NTdaMHMxDzANBgNVBAQTBkFsdG1hbjEVMBMGA1UE
KhMMSmVmZnJleSBFcmljMRwwGgYDVQQDExNKZWZmcmV5IEVyaWMgQWx0bWFuMSswKQYJKoZI
hvcNAQkBFhxqYWx0bWFuQHNlY3VyZS1lbmRwb2ludHMuY29tMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAqM/KsX6t2M5RIr9tbBkd3HmnX7LqKIJ3i+9gRUPGi5gWnkL99kRa
bmfjm5sFRsxJQjUwBk3qigSgfT1ROPoFeExLwH3zuWzzV0o4hSmFMXaM7OryCwWZ4DXnIcxE
APbQupRZ4vK9iTwUI1DD52wowyFf5Z2zE6ZFj/WM+Qkvqg3yQTChQWQ5f3859ZaHqgWdXp9q
IQdQoMwW5uxv1fQpUPyIh5HsLHPuLiRepFfAyWnZ/KjFwMgmsFpNjycD76DenfgiRa3Vrmmj
qDrzeciBKcanBgjC4mF4GpIRl8thZr7Up0PpKfbtR33gapS89UbmMLrNULkQnZpa6gC76c25
XwIDAQABozkwNzAnBgNVHREEIDAegRxqYWx0bWFuQHNlY3VyZS1lbmRwb2ludHMuY29tMAwG
A1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEEBQADgYEAO287/+2OIBXpUka2g22uXfiYHmMbIZOK
i3PDyuIHrPPQa9jXVtxYOanokqJ/6lUb7wsjJIOixu69rXeq7YfBhTEGZZzsKg13aT2RGVyt
9sNRNOsY5kryrKo2pPeBBlMBr1mdQjSy70bL+fajltBxph2pHeG02um90tI85aEgs5cwggM/
MIICqKADAgECAgENMA0GCSqGSIb3DQEBBQUAMIHRMQswCQYDVQQGEwJaQTEVMBMGA1UECBMM
V2VzdGVybiBDYXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xGjAYBgNVBAoTEVRoYXd0ZSBDb25z
dWx0aW5nMSgwJgYDVQQLEx9DZXJ0aWZpY2F0aW9uIFNlcnZpY2VzIERpdmlzaW9uMSQwIgYD
VQQDExtUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgQ0ExKzApBgkqhkiG9w0BCQEWHHBlcnNv
bmFsLWZyZWVtYWlsQHRoYXd0ZS5jb20wHhcNMDMwNzE3MDAwMDAwWhcNMTMwNzE2MjM1OTU5
WjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRk
LjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwgZ8wDQYJ
KoZIhvcNAQEBBQADgY0AMIGJAoGBAMSmPFVzVftOucqZWh5owHUEcJ3f6f+jHuy9zfVb8hp2
vX8MOmHyv1HOAdTlUAow1wJjWiyJFXCO3cnwK4Vaqj9xVsuvPAsH5/EfkTYkKhPPK9Xzgnc9
A74r/rsYPge/QIACZNenprufZdHFKlSFD0gEf6e20TxhBEAeZBlyYLf7AgMBAAGjgZQwgZEw
EgYDVR0TAQH/BAgwBgEB/wIBADBDBgNVHR8EPDA6MDigNqA0hjJodHRwOi8vY3JsLnRoYXd0
ZS5jb20vVGhhd3RlUGVyc29uYWxGcmVlbWFpbENBLmNybDALBgNVHQ8EBAMCAQYwKQYDVR0R
BCIwIKQeMBwxGjAYBgNVBAMTEVByaXZhdGVMYWJlbDItMTM4MA0GCSqGSIb3DQEBBQUAA4GB
AEiM0VCD6gsuzA2jZqxnD3+vrL7CF6FDlpSdf0whuPg2H6otnzYvwPQcUCCTcDz9reFhYsPZ
Ohl+hLGZGwDFGguCdJ4lUJRix9sncVcljd2pnDmOjCBPZV+V2vf3h9bGCE6u9uo05RAaWzVN
d+NWIXiC3CEZNd4ksdMdRv9dX2VPMYIDOzCCAzcCAQEwaTBiMQswCQYDVQQGEwJaQTElMCMG
A1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBl
cnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECAw7NrTAJBgUrDgMCGgUAoIIBpzAYBgkqhkiG
9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0wNTA3MjYwNTU2MzVaMCMGCSqG
SIb3DQEJBDEWBBQuaPh0h2c1Td9b6eTjVpkO5E1e6jBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqG
SIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG
9w0DAgIBKDB4BgkrBgEEAYI3EAQxazBpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3
dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJl
ZW1haWwgSXNzdWluZyBDQQIDDs2tMHoGCyqGSIb3DQEJEAILMWugaTBiMQswCQYDVQQGEwJa
QTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhh
d3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECAw7NrTANBgkqhkiG9w0BAQEFAASC
AQAVeqp4DxxcP+3Hf9NWgIZVJVeuHxNdz3JsnFiUlUox9+BhUqSRg8nCPrwsez8/Zdny4DxE
SbcX1nkGsMFyy9yOc17VZtw/s1YbKXC9MKGLdomSEBLlY5oM7OPYjbKvOLldm9gJ5K7c/bw9
YhUUi9PKcljN8b61cgSHCU5KbFI0iU7iNw37NnDyGs6UR0DN0VXrh7gj7NCdYkKzz8xD1stN
WG3IesgV0uj1rNoXNTlC+820+Ld4aX4i08Ls8OF1qEHeLL007Sh8jWFzG2dodK1nHdr1rdX5
3trfDMH+7efERopBOc5qFSES9YMMqlRuuWotTZWB+xsyfRDelaX988wZAAAAAAAA
--------------ms060906050502020604000306--