[OpenAFS-devel] volserver hangs possible fix

[Horst Birthelmer]

On Apr 6, 2005, at 1:40 PM, Tom Keiser wrote:

> Your stack trace looks like an LWP volserver, so I'm assuming that's
> either a production version, or an old devel build.
>
> In newer 1.3.x versions, V_BreakVolumeCallbacks is a function pointer
> to BreakVolumeCallbacksLater, which doesn't do any callbacks itself.
> Rather, it walks the callback hash chains, and sets the callback
> delayed flag for all the callbacks for the requested volumeId.
> BreakVolumeCallbacksLater and BreakLaterCallBacks both need H_LOCK,
> but MultiBreakCallBack_r drops H_LOCK before the multirx loop.  So, if
> I'm reading the code correctly, recent 1.3's aren't vulnerable to this
> deadlock.
>
> As a side note, if you find annoying problems like this, and you're
> running an OS where you can get a core without destroying the process
> (e.g. with gcore on solaris), drop the core snapshots to disk for
> future analysis ;)
>
> While auditing the callback code, I noticed some funky condition
> variable usage between FsyncCheckLWP and BreakVolumeCallbacksLater.
> One issue is BreakVolumeCallbacksLater calls pthread_cond_broadcast on
> fsync_cond without holding the associated lock.  Here's a quick patch
> for that race:
>

I don't actually think that would be a race condition. It just leads to 
some undefined scheduling behavior.
If you don't have locked the mutex with which the other thread is doing 
it's wait you may just continue with that thread depending on your 
schedulers mood ;-)

Locking that mutex means you _will_ definitely continue when you call 
the unlock and not at some undefined point in time.
Here's what the POSIX standard says about this:

<quote>
If more than one thread is blocked on a condition variable, the 
scheduling policy shall determine the order in which threads are 
unblocked. When each thread unblocked as a result of a 
pthread_cond_broadcast() or pthread_cond_signal() returns from its call 
to pthread_cond_wait() or pthread_cond_timedwait(), the thread shall 
own the mutex with which it called pthread_cond_wait() or 
pthread_cond_timedwait(). The thread(s) that are unblocked shall 
contend for the mutex according to the scheduling policy (if 
applicable), and as if each had called pthread_mutex_lock().

The pthread_cond_broadcast() or pthread_cond_signal() functions may be 
called by a thread whether or not it currently owns the mutex that 
threads calling pthread_cond_wait() or pthread_cond_timedwait() have 
associated with the condition variable during their waits; however, if 
predictable scheduling behavior is required, then that mutex shall be 
locked by the thread calling pthread_cond_broadcast() or 
pthread_cond_signal().
<\quote>


Horst