[OpenAFS-devel] volserver hangs possible fix
Jeffrey Hutzelman
jhutz@cmu.edu
Mon, 18 Apr 2005 12:33:38 -0400
On Monday, April 18, 2005 05:01:06 PM +0200 Horst Birthelmer
<horst@riback.net> wrote:
> Most implementations don't have an atomic cond_wait since it's not
> mandatory by POSIX ;-)
> It's just you have to treat it that way since there's no guaranty that
> you can rely on an atomic implementation.
> It's no "problem" at all, it's just one aspect you have to keep in mind.
>
> I introduced that with "where it's not atomic" because that's the very
> assumption in this discussion. None of the arguments would be true if it
> was.
>
> By definition, a con_wait is used for waiting on some event inside a
> critical section. This means you entered the cond_wait with the mutex
> held. Now the cond_wait call enqueues the thread into the queue of
> threads waiting on this condition variable and unlocks the mutex. If you
> use the broadcast call protected by the same mutex you will always have a
> "settled" environment. This means you're sure you don't have any threads
> inside the critical section since you won't get to do broadcast (you
> would be waiting on the mutex_lock) and during the broadcast you won't
> have any threads entering the critical section. I can just repeat myself.
> It's more safe but from my point of view not a fix to the problem we had.
There seems to be a certain amount of confusion here about how condition
variables work. The wait operation on a CV is atomic with respect to the
broadcast operation on the same CV and to that CV's mutex. That is, it is
impossible for the broadcast to happen "between" when cond_wait releases
the mutex and when the thread calling cond_wait becomes blocked on the CV.
This is because conceptually, those operations occur at the same time.
CV's are hardly a uniquely POSIX concept, but since we are talking here
about code written against the pthreads API, we can rely on POSIX's
description of how they work, which has this to say:
These functions [pthread_cont_timedwait, pthread_cond_wait]
atomically release _mutex_ and cause the calling thread to block on
the condition variable _cond_; atomically here means "atomically
with respect to access by another thread to the mutex and then the
condition variable". That is, if another thread is able to acquire
the mutex after the about-to-block thread has released it, then a
subsequence call to pthread_cond_broadcast() or
pthread_cond_signal() in that thread shall behave as if it were
issued after the about-to-block thread has blocked.
-- IEEE Std. 1003.1-2001, p. 1032
Also:
The pthread_cond_broadcast() or pthread_cond_signal() functions may
be called by a thread whether or not it currently owns the mutex
that threads calling pthread_cond_wait() or
pthread_cond_timedwait() have associated with the condition
variable during their waits; however, if predictable scheduling
behaviour is required, then that mutex shall be locked by the
thread calling pthread_cond_broadcast() or pthread_cond_signal().
-- IEEE Std. 1003.1-2001, p. 1025
In other words, if you want any control over when the waiting threads wake
up, you need to hold the mutex, because they can't wake up while you are
holding the mutex.
We don't need that kind of control. What we do need is the guarantee that
once the waiting thread acquires the mutex, it won't "miss" an event due to
something being put on the queue and cond_broadcast() being called between
when the waiting thread checks the queue and when it becomes blocked on the
CV. Since it holds the mutex for that entire duration, it is sufficient to
prevent _either_ modification of the queue _or_ the broadcast while the
waiting thread is holding the mutex; it is not necessary to prevent both
when they only happen in sequence.
-- Jeff