[OpenAFS-devel] volserver hangs possible fix
Horst Birthelmer
horst@riback.net
Mon, 11 Apr 2005 20:35:49 +0200
On Apr 6, 2005, at 1:40 PM, Tom Keiser wrote:
> Your stack trace looks like an LWP volserver, so I'm assuming that's
> either a production version, or an old devel build.
>
> In newer 1.3.x versions, V_BreakVolumeCallbacks is a function pointer
> to BreakVolumeCallbacksLater, which doesn't do any callbacks itself.
> Rather, it walks the callback hash chains, and sets the callback
> delayed flag for all the callbacks for the requested volumeId.
> BreakVolumeCallbacksLater and BreakLaterCallBacks both need H_LOCK,
> but MultiBreakCallBack_r drops H_LOCK before the multirx loop. So, if
> I'm reading the code correctly, recent 1.3's aren't vulnerable to this
> deadlock.
>
> As a side note, if you find annoying problems like this, and you're
> running an OS where you can get a core without destroying the process
> (e.g. with gcore on solaris), drop the core snapshots to disk for
> future analysis ;)
>
> While auditing the callback code, I noticed some funky condition
> variable usage between FsyncCheckLWP and BreakVolumeCallbacksLater.
> One issue is BreakVolumeCallbacksLater calls pthread_cond_broadcast on
> fsync_cond without holding the associated lock. Here's a quick patch
> for that race:
>
I don't actually think that would be a race condition. It just leads to
some undefined scheduling behavior.
If you don't have locked the mutex with which the other thread is doing
it's wait you may just continue with that thread depending on your
schedulers mood ;-)
Locking that mutex means you _will_ definitely continue when you call
the unlock and not at some undefined point in time.
Here's what the POSIX standard says about this:
<quote>
If more than one thread is blocked on a condition variable, the
scheduling policy shall determine the order in which threads are
unblocked. When each thread unblocked as a result of a
pthread_cond_broadcast() or pthread_cond_signal() returns from its call
to pthread_cond_wait() or pthread_cond_timedwait(), the thread shall
own the mutex with which it called pthread_cond_wait() or
pthread_cond_timedwait(). The thread(s) that are unblocked shall
contend for the mutex according to the scheduling policy (if
applicable), and as if each had called pthread_mutex_lock().
The pthread_cond_broadcast() or pthread_cond_signal() functions may be
called by a thread whether or not it currently owns the mutex that
threads calling pthread_cond_wait() or pthread_cond_timedwait() have
associated with the condition variable during their waits; however, if
predictable scheduling behavior is required, then that mutex shall be
locked by the thread calling pthread_cond_broadcast() or
pthread_cond_signal().
<\quote>
Horst