[OpenAFS] State of OpenAFS

[Todd M. Lewis]

Wout Mertens wrote:
> 
> [...] Also, afs does write-on-close, does it
> write-on-flush as well?

You're probably thinking about flush as in fflush(3) which only flushes
buffered user space data.  As far as AFS is concerned, writes don't
happen until the user space buffers flush, so no, AFS doesn't do
write-on-flush.

> What about logparsers running on one machine and
> the logwriter on another (i.e. procmail)?

Well this is a more interesting question, and the usual "AFS does
write-on-close" is really to simple to explain it.  I was seeing really
strange behaviour from one suite of cooperating processes -- at least it
seemed strange trying to figure out what happened looking at the logs --
because it appeared things were happening out of sequence.  I eventually
got it working as expected, and here's a Perl code snippet that shows
what I ended up using:

>  # The '+>>' (read/write append) mode is to trick the AFS cache
>  # manager to flush these writes out when they happen.
>  # "$| = 1" by itself isn't enough!
>  open LOG, "+>>/afs/isis.unc.edu/<more.stuff...>";
>  select LOG; $| = 1;

By using O_RDWR|O_APPEND ("+>>" in Perl), the log entries started
showing up in the expected order.  Note that just O_WRONLY|O_APPEND
(">>" in Perl) was insufficient.  However:
  * all the writers were on the same box, thus using the same cache
  * the man page for open(2) gives explicit warnings about NFS not
handling O_APPEND well; the same issues may be relevant wrt AFS if your
writers are on different machines, but I don't know the answer to that. 
I haven't tried it with multiple writers on multiple machines.

But what appears to be the case from my experience is that AFS handles
flushed writes in O_APPEND mode much the same as write-on-close, only
without the close.  I can't say that's actually how it's implemented,
however.

> Are these really problems?

They really are issues, but once you understand how they work, they're
just more basic assumptions that you work with/from.  Whether that
constitutes a problem is your call.

> Anything else that is different versus 'regular' unix filesystems?

* ACLs apply to whole directories
* Only the first 3 bits of a file's permissions are relevant
  (which gives you 6 free bits to do with whatever crazy
   things you want -- not that I'd ever do such a thing :-)
* A cell-wide switch for support of AFS suid/sgid executables
* volume mount points are just specially formatted symlinks, so
  volumes can be mounted almost anywhere any number of times
* @sys macro expansion makes serious architecture magic possible
* PTS cool/weird (groups in groups, IP ACLs, user-defined groups)
-- 
   +----------------------------------------------------------------+
  / Todd_Lewis@unc.edu                  http://www.unc.edu/~utoddl /
 /(919) 962-5273  Linux - It's now safe to turn on your computer. /
+----------------------------------------------------------------+