[OpenAFS] Callback issues

[Dan Hyde]

We're seeing problems with our (patched) OpenAFS-1.4.2 threaded
fileservers.  Here's what I know (those who want less detail should
skip to "Core files"):

Structures:
Inside callback.c there are CallBack's and FileEntry's.  There used to
be a limit of 65535 of shared space for both, but the two are now split
out and limited by available memory.  

There are lists which link these two structs, including the timeout
lists, host lists, CallBack lists, FileEntry lists, and free lists.
In-use FileEntry's contain single-linked lists for FileEntry's and
CallBack's; these contain index values into the FileEntry and CallBack
arrays, terminated by a 0.  CallBack's contain single-linked lists for
CallBack's and FileEntry's, and double-linked lists (including heads of
list) for timeouts and host entries; these are index values, with the
single-linked lists terminated by a 0, and the double-linked lists
terminated by links to themselves.  The first thing in a struct
CallBack is the link to the next CallBack; the first thing in a
struct FileEntry used to be the link to the next FileEntry, but that is
no longer true today.

The two free lists, CBfree and FEfree, are single-linked lists of
CallBack's and FileEntry's using the first word of their struct cast as
pointers; they are null-terminated.  Removing a CallBack from an in-use
list and putting it on the free chain overwrites the link to the next
CallBack; this guarentee is no longer true for FileEntry's.

Locks:
There used to be a CBFElock used when adding and removing CallBack's
and FileEntry's from their free chains; this lock is no longer there.
There is now a host_glock_mutex that is set with H_LOCK.  There is a
host-specific lock that used to be set with h_Lock but is now set with
h_Lock_r, which does an H_UNLOCK, h_Lock, and H_LOCK -- h_Lock_r will
allow any thread blocking on an H_LOCK to get into the critical region
protected by the host_glock_mutex.  There is also a host-specific hold
that "prevents this structure and inferior structures from
disappearing".  

It is not clear if a host lock or hold is needed when adding or
removing the host entries of a CallBack.  Two routines do not:
CleanupTimedOutCallBacks and DeleteFileCallBacks (aka DeleteFile),
and the comments for DeleteFileCallBacks are unclear on the issue:

	"This call doesn't set all of the various host locks, but it
	shouldn't really matter since we're not adding callbacks, but
	deleting them.  I'm not sure why it doesn't set the lock,
	however; perhaps it should."

Core files:
In two of the three core files, in iGetFE, FEfree is bad; it's not a
pointer.  In the other core file, in CleanupTimedOutCallBacks_r, a
timeout chain of CallBack's contains a FileEntry firstcb that leads to
another CallBack with a cnext index which is a pointer.  In all three
cases, we are ignoring the resulting segfault, repeatedly, while
holding the H_LOCK global mutex, blocking most other threads.

Log files:
On two of our fileservers, there are FileLog entries that contain
'Request for unallocated vnode' followed by a pointer.  In one FileLog,
there are 41 references to 0x40000000 as a vnode, and 1 for 0xffffffff;
on another FileLog, there are 16 references to 0x40000000.

History:
We have access to Transarc AFS as far back as 1995.  There have been
concerns regarding the manipulation of these lists all along.

Next steps:
- check the three core files we have to see if the CBfree chain is bad,
too.
- find a general solution to removing the contents of a KeyFile from
the data of a core file so we can share it

We hope the above information proves useful.