[OpenAFS-devel] Java VM native libs call flock() with l_len=0x7fffffff00000000

[Adam Megacz]

Strange, huh?  I was eventually able to track down the source of the
"afs: warning byte-range lock/unlock ignored" with the help of Matt's
patch.

It appears that Sun's JVM (and perhaps others) have only one low-level
C routine for requesting locks from the OS.  That routine is a
byte-range locking routine.  To implement the higher-level "lock this
entire file" API (FileChannel.tryLock()), the libraries (written in
Java) invoke the low-level C routine with the file length set to -1.
This constant is provided as a *Java long*, which is always 64 signed
bits regardless of OS or CPU.

Anyways, the ultimate result is that AFS winds up getting a lock
request like this:

   l_start  = 0
   l_whence = 0
   l_len    = 0x7fffffff00000000

Clearly this is a Java bug, but Sun is unlikely to ever do anything
about it.  Since it's also unlikely that any file is ever going to be
anywhere close to being (unsigned)0x7fffffff00000000 bytes long (more
than 2^63), the patch below has OpenAFS interpret such requests as
whole-file locks.

Also available from:

  /afs/megacz.com/debian/patches/make-openafs-and-java-cooperate.patch

  - a

--- openafs-1.4.2/src/afs/VNOPS/afs_vnop_flock.c        2007-02-11 12:40:51.000000000 -0800
+++ openafs-1.4.2/src/afs/VNOPS/afs_vnop_flock.c        2007-02-11 13:42:27.000000000 -0800
@@ -554,6 +554,10 @@
        af->l_len = 0;          /* since some systems indicate it as EOF */
 #endif
 #endif
+    /* Java VMs ask for l_len=(long)-1 regardless of OS/CPU; bottom 32 bits
+     * sometimes get masked off by OS */
+    if ((af->l_len >> 32) == 0x7fffffff)
+       af->l_len = 0;
     /* next line makes byte range locks always succeed,
      * even when they should block */
     if (af->l_whence != 0 || af->l_start != 0 || af->l_len != 0) {