[OpenAFS-win32-devel] Re: file locking

Wed, 20 Jul 2005 16:06:05 -0400

This is a multi-part message in MIME format.
--------------050602090506090200010409
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Eric,

>jeff suggests:

>> How about we take this discussion to the openafs-win32-devel list?
>  
>
Happy to do this on-list, if you both agree.  Forgive me for putting 
some snippets in from different mails.

>> Eric,
>>
>> Excellent.  This is the fun part of it, I assume for IFS only?  If I
>> understand correctly, that effectively puts IFS client and traditional
>> client on a par wrt locking?

> yes, ifs only.  the traditional client may not have functional byte-range
> locks, but, being of the advisory type, they are not extremely useful.

I think I need some help parsing that...

I had written some trivial code to see what Win32 LockFile did in various scenarios.  Didn't attempt ones with conflicting sessions.  Running out of AFS (recent client) on a single machine, it certainly appears you can lock ranges as a reader of the Win32 API docs would expect, but of course these don't correspond to anything in AFS.  I have assumed that these are IFS locks managed by windows file sharing, and that enforcement would also be what you would expect...  (Please correct here, if I have this all wrong.)

In the Linux client, I used the Trond-enhanced posix_lock_file_wait to 
do more or less what you do for IFS in your patch.  It wasn't hard to do 
there either.  A lock request fails if, after getting the posix lock, 
the client doesn't already have, and cannot get, a corresponding 
whole-file lock in AFS.  Of course, this does not require an upcall in 
that context, etc.

If I follow the "alternatives" discussion correctly, the issues raised 
_here_

if the cache manager were to start enforcing mandatory locks (just as a
>scenario), there would still be problems.  mapped file (or, paging i/o)
>requests should never be rejected because of lock conflicts.  it is
>possible to bypass the lock on a region by using mem-mapped i/o.  this is
>an expected and necessary behaviour; not completing such a request
>successfully is bad.  so, if all instances have locks on them, and a
>paging request is received, the cache manager would reject the upcall
>read/write request.  of course, in this case, the read and write calls
>could also be modified...

could be addressed if the code responsible for doing the upcall knew 
what the RsRtl routines know about IO operations?  I'm missing how doing 
our own range reader-writer locks would address that, but, you don't 
advocate that.

In particular, it seems to me if the FsRtl routines exist, provide a 
locking implementation for free that is likely the default implmentation 
for everyone else, too ("**Lock* 'Em Up - Byte Range Locking")*, then it 
seems the more difficult argument is for _not_ using them. 

Now, when you say the protocols are incompatible, it appears that the only issues unaccounted for are related to the time of validity in the cm locking interface?  

>i found the locking protocols to be incompatible.
>
>the cache manager allows locks to be placed and removed using the four
>parameters (instance, lock_type, offset, length).  additionally, locks
>require a time-period of validity.  these are advisory locks.
>

The time-period of validity throws me--seems to be a Windows CM-ism?  I didn't run into this in afs_lockctl, nor do I see it in afs.h, afsint.xg, and so forth.

>the ifs layer only receives read-write locks on regions:  the caller is
>guaranteed exclusive access to the locked region.  the lock request
>requires the four parameters (instance, offset, length, key).  however,
>unlock requests have one of three forms:  (instance, offset, length, key),
>(instance, key), (instance).  it is possible to remove all file locks on a
>specified handle with a single call.  if granted, these are mandatory
>locks.

This is not too dissimilar to what you see on the various Unixes, right?

>a simpler solution is to use the kernel's FsRtl functions to manage locks,
>and to send the whole-file-lock request up to a daemon which spins on
>making lock requests as often as necessary (because of expiry).  this is
>similar to smb_WaitingLocksDaemon.  even if the lock is dropped by the cm,
>for one of any number of reasons, the mandatory lock is still enforced
>correctly in the kernel.  this is what i would like to do.

Aside from the the lock expiry "spinning" behavior, the solution you propose does sound to me like the analog of the behavior I implemented, Nathan Neulinger described, etc.

Would we need a lock-re-upping daemon if the Windows CM worked like the Unix cms in this regard? 

Matt

my work-up of locking:

>
>>
>>i could modify the cache manager's locking functions to conform to the ifs
>>model for lock requests.  the ifs redirector would pass the lock requests
>>to the cache manager.  for a read request, the cm would have to be
>>consulted to make sure there is no lock conflict (granted locks are
>>mandatory), and then the i/o request would be processed.  if the lock is
>>destroyed by the cm, then improper i/o could take place.  this is a
>>serious problem (ms office, for instance, may crash or fail), as i'm sure
>>you know.
>>
>>if the cache manager were to start enforcing mandatory locks (just as a
>>scenario), there would still be problems.  mapped file (or, paging i/o)
>>requests should never be rejected because of lock conflicts.  it is
>>possible to bypass the lock on a region by using mem-mapped i/o.  this is
>>an expected and necessary behaviour; not completing such a request
>>successfully is bad.  so, if all instances have locks on them, and a
>>paging request is received, the cache manager would reject the upcall
>>read/write request.  of course, in this case, the read and write calls
>>could also be modified...
>>
>>another possibility is to not change the cache manager's code, and to keep
>>a list of locks in the kernel.  then, an appropriate number of unlock
>>requests could be sent via upcalls.  this is redundant at best.
>>
>>a simpler solution is to use the kernel's FsRtl functions to manage locks,
>>and to send the whole-file-lock request up to a daemon which spins on
>>making lock requests as often as necessary (because of expiry).  this is
>>similar to smb_WaitingLocksDaemon.  even if the lock is dropped by the cm,
>>for one of any number of reasons, the mandatory lock is still enforced
>>correctly in the kernel.  this is what i would like to do.
>

Eric Williams wrote:

>On Wed, 20 Jul 2005, Matt Benjamin wrote:
>
>  
>
>>Eric,
>>
>>Excellent.  This is the fun part of it, I assume for IFS only?  If I
>>understand correctly, that effectively puts IFS client and traditional
>>client on a par wrt locking?
>>    
>>
>
>yes, ifs only.  the traditional client may not have functional byte-range
>locks, but, being of the advisory type, they are not extremely useful.
>
>  
>

-- 

Matt Benjamin

The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

--------------050602090506090200010409
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
  <title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Eric,<br>
<br>
<pre wrap="">&gt;jeff suggests:

</pre>
<blockquote type="cite">
 <pre wrap="">&gt; How about we take this discussion to the openafs-win32-devel list?
 </pre>
</blockquote>
Happy to do this on-list, if you both agree.&nbsp; Forgive me for putting
some snippets in from different mails. 
 
<pre wrap="">&gt;&gt; Eric,
&gt;&gt;
&gt;&gt; Excellent. This is the fun part of it, I assume for IFS only? If I
&gt;&gt; understand correctly, that effectively puts IFS client and traditional
&gt;&gt; client on a par wrt locking?

&gt; yes, ifs only.  the traditional client may not have functional byte-range
&gt; locks, but, being of the advisory type, they are not extremely useful.

I think I need some help parsing that...

I had written some trivial code to see what Win32 LockFile did in various scenarios. Didn't attempt ones with conflicting sessions. Running out of AFS (recent client) on a single machine, it certainly appears you can lock ranges as a reader of the Win32 API docs would expect, but of course these don't correspond to anything in AFS.&nbsp; I have assumed that these are IFS locks managed by windows file sharing, and that enforcement would also be what you would expect... (Please correct here, if I have this all wrong.)
</pre>
In the Linux client, I used the Trond-enhanced posix_lock_file_wait to
do more or less what you do for IFS in your patch.&nbsp; It wasn't hard to
do there either.&nbsp; A lock
request fails if, after getting the posix lock, the client doesn't
already have, and cannot get, a corresponding whole-file lock in AFS.&nbsp;
Of course, this does not require an upcall in that context, etc. 
 
If I follow the "alternatives" discussion correctly, the issues raised
_here_ 
<pre wrap="">if the cache manager were to start enforcing mandatory locks (just as a
&gt;scenario), there would still be problems. mapped file (or, paging i/o)
&gt;requests should never be rejected because of lock conflicts. it is
&gt;possible to bypass the lock on a region by using mem-mapped i/o. this is
&gt;an expected and necessary behaviour; not completing such a request
&gt;successfully is bad. so, if all instances have locks on them, and a
&gt;paging request is received, the cache manager would reject the upcall
&gt;read/write request. of course, in this case, the read and write calls
&gt;could also be modified...</pre>
could be addressed if the code responsible for doing the upcall knew
what the RsRtl routines know about IO operations?&nbsp; I'm missing how
doing our own range reader-writer locks would address that, but, you
don't advocate that. 
 
In particular, it seems to me if the FsRtl routines exist, provide a
locking implementation for free that is likely the default
implmentation for everyone else, too ("Lock
'Em Up - Byte Range Locking"), then it seems the more
difficult argument is for _not_ using them.&nbsp; 
<pre wrap="">Now, when you say the protocols are incompatible, it appears that the only issues unaccounted for are related to the time of validity in the cm locking interface?

&gt;i found the locking protocols to be incompatible.
&gt;
&gt;the cache manager allows locks to be placed and removed using the four
&gt;parameters (instance, lock_type, offset, length). additionally, locks
&gt;require a time-period of validity. these are advisory locks.
&gt;

The time-period of validity throws me--seems to be a Windows CM-ism? I didn't run into this in afs_lockctl, nor do I see it in afs.h, afsint.xg, and so forth.

&gt;the ifs layer only receives read-write locks on regions: the caller is
&gt;guaranteed exclusive access to the locked region. the lock request
&gt;requires the four parameters (instance, offset, length, key). however,
&gt;unlock requests have one of three forms: (instance, offset, length, key),
&gt;(instance, key), (instance). it is possible to remove all file locks on a
&gt;specified handle with a single call. if granted, these are mandatory
&gt;locks.

This is not too dissimilar to what you see on the various Unixes, right?

&gt;a simpler solution is to use the kernel's FsRtl functions to manage locks,
&gt;and to send the whole-file-lock request up to a daemon which spins on
&gt;making lock requests as often as necessary (because of expiry). this is
&gt;similar to smb_WaitingLocksDaemon. even if the lock is dropped by the cm,
&gt;for one of any number of reasons, the mandatory lock is still enforced
&gt;correctly in the kernel. this is what i would like to do.</pre>
<pre wrap="">Aside from the the lock expiry "spinning" behavior, the solution you propose does sound to me like the analog of the behavior I implemented, Nathan Neulinger described, etc.

Would we need a lock-re-upping daemon if the Windows CM worked like the Unix cms in this regard? 

Matt

my work-up of locking:

</pre>
<blockquote type="cite">
 <pre wrap="">
&gt;
&gt;i could modify the cache manager's locking functions to conform to the ifs
&gt;model for lock requests. the ifs redirector would pass the lock requests
&gt;to the cache manager. for a read request, the cm would have to be
&gt;consulted to make sure there is no lock conflict (granted locks are
&gt;mandatory), and then the i/o request would be processed. if the lock is
&gt;destroyed by the cm, then improper i/o could take place. this is a
&gt;serious problem (ms office, for instance, may crash or fail), as i'm sure
&gt;you know.
&gt;
&gt;if the cache manager were to start enforcing mandatory locks (just as a
&gt;scenario), there would still be problems. mapped file (or, paging i/o)
&gt;requests should never be rejected because of lock conflicts. it is
&gt;possible to bypass the lock on a region by using mem-mapped i/o. this is
&gt;an expected and necessary behaviour; not completing such a request
&gt;successfully is bad. so, if all instances have locks on them, and a
&gt;paging request is received, the cache manager would reject the upcall
&gt;read/write request. of course, in this case, the read and write calls
&gt;could also be modified...
&gt;
&gt;another possibility is to not change the cache manager's code, and to keep
&gt;a list of locks in the kernel. then, an appropriate number of unlock
&gt;requests could be sent via upcalls. this is redundant at best.
&gt;
&gt;a simpler solution is to use the kernel's FsRtl functions to manage locks,
&gt;and to send the whole-file-lock request up to a daemon which spins on
&gt;making lock requests as often as necessary (because of expiry). this is
&gt;similar to smb_WaitingLocksDaemon. even if the lock is dropped by the cm,
&gt;for one of any number of reasons, the mandatory lock is still enforced
&gt;correctly in the kernel. this is what i would like to do.</pre>
</blockquote>
 
 
 
 
Eric Williams wrote:
<blockquote cite="midPine.BSO.4.58.0507201318250.11288@citi.umich.edu"
 type="cite">
 <pre wrap="">On Wed, 20 Jul 2005, Matt Benjamin wrote:

  </pre>
  <blockquote type="cite">
    <pre wrap="">Eric,

Excellent.  This is the fun part of it, I assume for IFS only?  If I
understand correctly, that effectively puts IFS client and traditional
client on a par wrt locking?
    </pre>
  </blockquote>
  <pre wrap=""><!---->
yes, ifs only.  the traditional client may not have functional byte-range
locks, but, being of the advisory type, they are not extremely useful.

  </pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">-- 

Matt Benjamin

The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

<a class="moz-txt-link-freetext" href="http://linuxbox.com">http://linuxbox.com</a>

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

</pre>
</body>
</html>

--------------050602090506090200010409--