[OpenAFS-devel] volserver hangs possible fix

[Jeffrey Hutzelman]

On Monday, April 18, 2005 05:01:06 PM +0200 Horst Birthelmer 
<horst@riback.net> wrote:

> Most implementations don't have an atomic cond_wait since it's not
> mandatory by POSIX ;-)
> It's just you have to treat it that way since there's no guaranty that
> you can rely on an atomic implementation.
> It's no "problem" at all, it's just one aspect you have to keep in mind.
>
> I introduced that with "where it's not atomic" because that's the very
> assumption in this discussion. None of the arguments would be true if it
> was.
>
> By definition, a con_wait is used for waiting on some event inside a
> critical section. This means you entered the cond_wait with the mutex
> held. Now the cond_wait call enqueues the thread into the queue of
> threads waiting on this condition variable and unlocks the mutex. If you
> use the broadcast call protected by the same mutex you will always have a
> "settled" environment. This means you're sure you don't have any threads
> inside the critical section since you won't get to do broadcast (you
> would be waiting on the mutex_lock) and during the broadcast you won't
> have any threads entering the critical section. I can just repeat myself.
> It's more safe but from my point of view not a fix to the problem we had.



There seems to be a certain amount of confusion here about how condition 
variables work.  The wait operation on a CV is atomic with respect to the 
broadcast operation on the same CV and to that CV's mutex.  That is, it is 
impossible for the broadcast to happen "between" when cond_wait releases 
the mutex and when the thread calling cond_wait becomes blocked on the CV. 
This is because conceptually, those operations occur at the same time.

CV's are hardly a uniquely POSIX concept, but since we are talking here 
about code written against the pthreads API, we can rely on POSIX's 
description of how they work, which has this to say:


        These functions [pthread_cont_timedwait, pthread_cond_wait]
        atomically release _mutex_ and cause the calling thread to block on
        the condition variable _cond_; atomically here means "atomically
        with respect to access by another thread to the mutex and then the
        condition variable".  That is, if another thread is able to acquire
        the mutex after the about-to-block thread has released it, then a
        subsequence call to pthread_cond_broadcast() or
        pthread_cond_signal() in that thread shall behave as if it were
        issued after the about-to-block thread has blocked.

        -- IEEE Std. 1003.1-2001, p. 1032

Also:

        The pthread_cond_broadcast() or pthread_cond_signal() functions may
        be called by a thread whether or not it currently owns the mutex
        that threads calling pthread_cond_wait() or
        pthread_cond_timedwait() have associated with the condition
        variable during their waits; however, if predictable scheduling
        behaviour is required, then that mutex shall be locked by the
        thread calling pthread_cond_broadcast() or pthread_cond_signal().

        -- IEEE Std. 1003.1-2001, p. 1025

In other words, if you want any control over when the waiting threads wake 
up, you need to hold the mutex, because they can't wake up while you are 
holding the mutex.

We don't need that kind of control.  What we do need is the guarantee that 
once the waiting thread acquires the mutex, it won't "miss" an event due to 
something being put on the queue and cond_broadcast() being called between 
when the waiting thread checks the queue and when it becomes blocked on the 
CV.  Since it holds the mutex for that entire duration, it is sufficient to 
prevent _either_ modification of the queue _or_ the broadcast while the 
waiting thread is holding the mutex; it is not necessary to prevent both 
when they only happen in sequence.


-- Jeff