[OpenAFS-devel] Solaris afs.rc file damage

[Tom Keiser]

On 4/13/07, Dean Anderson <dean@av8.com> wrote:
> On Fri, 13 Apr 2007, Tom Keiser wrote:
>
> > On 4/12/07, Dean Anderson <dean@av8.com> wrote:
> > > There may not be any degradation. The performance differences come if
> > > you have to replace every syscall with an open/ioctl/close cycle.  This
> >
> > I'm not concerned with open/close overhead -- the uses I'm referring
> > to fall back to ioctl fd caching if syscall(AFS_SYSCALL,...) probes
> > fail during application startup.
>
> I'm not thinking of a fall-back if probes fail. The
> syscall(AFS_SYSCALL...) will go away completely.  Afsd won't make these
> calls, but will make ioctls in the same place.
>

Re-Read what Derrick said.  I'm talking about new code under
development.  These new uses of the syscall mux aren't specific to
afsd -- they're used across all userspace afs code that runs on
unix-like platforms.


> > What I am concerned about are call path latency differences, and far
> > more importantly, locking model differences, between syscall() and
> > ioctl() on our supported unix platforms.  For the project I'm working
> > on, the path into the kernel module has to be lockless, and call
> > latency needs to be very low, with a small jitter.
>
> You'll have to explain this more.  The both afs_syscall and ioctl are
> system calls.  The latency is the same, unless you have to make
> additional system calls, as I explained previously.
>

No, it's not the same.  The ioctl syscall entry point involves file
descriptor lookups followed by vfs ops in order to demux onto a
function pointer.  On almost every operating system this is a
heavyweight operation which will involve several synchronization
primitives.  Afs_Syscall() does not necessarily involve any locking --
on the instrumentation branch, AFS_GLOCK is acquired conditionally
based upon the top level mux operation code.


> However, nearly every unix system I know of allows sched() to be called
> on return through the system call trampoline, so it is unclear what it

This argument misses the point.  The goal is to minimize latency
wherever _possible_.  For the purposes of this discussion, preemption
is assumed to be outside the realm of control.


> means to be lockless or why that would be particularly important. There
> are locks in the afs module. Certainly system calls have jitter as a

You're assuming that new additions to OpenAFS are written in the same
style as the cache manager.  The code I'm talking about is lockless
and mp-scaleable.


> result of process scheduling.  This behavior isn't any different between
> an ioctl and an afs_syscall---its true of every system call.
>
> If you are trying to improve the performance of the afs kernel module,
> it is usually the lock trains that kill performance---taking 6 or 10
> locks to do one thing instead of one lock.  On the other hand, too few
> locks are unnecessarilly blocky.  But this is all in the kernel module;
> it doesn't matter which system call you use to transfer from user mode
> to kernel mode.
>

Indeed, but I'm not discussing the afs kernel module as you know it.
The new addition is lockless -- the only penalties we incur are
preemption enable/disable calls (and occasionally we incur overhead to
move a process context into a runnable queue, but I'm working on ways
to eliminate that overhead).


> Whether system call latency is low or high depends on the machine
> architecture. Some are quite fast. Some execute 10000+ instructions in
> the trap handler on each system call, not including the purpose of the
> system call.
>

Once again, not relevant.  I'm taking the overhead from trap through
trampoline to be an unchangeable per-platform constant.  If we
formalize this concept and treat it as an optimization problem, the
goal is to tweak variables we control (openafs code, how the afs
syscall mux is hooked into the kernel, etc.) in order to minimize call
path latency.


> I'm not aware of any locks taken that specific to the ioctl call that
> would be taken on every ioctl call. Whether locks are necessary or taken
> usually depends on the driver. We'll be providing the driver code...
>

System call jump tables are assumed to be quasi-static on most
platforms.  A fairly obvious optimization is to not worry about
synchronizing lookups.

Ioctls, on the other hand, involve demuxing through lookup tables
which have to deal with races against module unloads, struct file and
descriptor table changes, etc.  Hence, synchronization is necessary
for both table updates, and demux ops during the ioctl syscall entry
point.  Add VFS layer overhead on top of this, and you start to have a
lot of overhead.  Compound this with the fact that most core AFS
services are multithreaded daemons, and you can see why file
descriptor table and/or struct file locking makes ioctls a
non-starter.

Regards,

-- 
Tom Keiser
tkeiser@gmail.com