[OpenAFS-devel] locking fairness on Linux?

[Chaskiel M Grundman]

On Thu, 5 May 2005, John S. Bucy wrote:

> On Wed, May 04, 2005 at 04:19:40PM -0400, John S. Bucy wrote:
>
>> Not only is it serializing but as far as I can tell, the first one
>> runs exclusively until it finishes, then the second and so on.
[...]
> Also, it pipelines nicely if I run it in separate directories; the
> throughput goes up linearly at least for the first few processes.
I don't think this will happen with a single directory even if 
the scheduling becomes "fair". the vcache object *will* be locked across 
rpcs that are intended to update it. If your goal is to increase 
aggregate throughput, I don't think this will help. if your goal is to 
reduce individual process "latency", then I don't much care.

> Reading a little further, the struct afs_lock has this characteristic.
> Obtaining it consists of essentially while(still locked) sleep().  In
> my example, the only way one of the starved threads will get to run is
> if the running thread is preempted just after unlocking and one of
> the starving ones wakes up from sleep and is scheduled in the next
> timeslice.  I'd imagine that if I ran this for a long time that I
> might see that happen once but haven't so far.
That won't happen, since if what you describe is actually happening, the 
GLOCK is held at all times that the vnode lock is not (except when the 
process is running in usermode, of course).
sleep() "expands" to
AFS_GUNLOCK()
really_sleep()
AFS_GLOCK()

> IMHO, the Right thing here is for struct afs_lock to be a wrapper for
> a native primitive that has the desired properties.  It seems easy
> enough to do but I have no idea if it would Just Work or if it would
> break in strange and subtle ways because of assumptions elsewhere.
I actually did this, with an eye towards reducing dependence on the GLOCK. 
It did break. it might not be broken now, but since I couldn't actually 
reduce GLOCK use this way (there is code that does interlocked sleeps that 
relies on the glock), I didn't see much point in adding per-platform code 
here.

> Looking around, there only seem to be ~10 global instances and its
> used in struct afs_vcache (only in AIX32_ENV apparently) and in
> struct memCacheEntry.
afs_rwlock_t's have the same properties, and there is one in the vcache on 
all platforms This is probably the lock that is being held across the 
rpc's, not one of the global ones.