[OpenAFS-devel] volserver hangs possible fix

[Horst Birthelmer]

Hi Jeff,

On Apr 18, 2005, at 6:33 PM, Jeffrey Hutzelman wrote:
>
> On Monday, April 18, 2005 05:01:06 PM +0200 Horst Birthelmer 
> <horst@riback.net> wrote:
>
>> Most implementations don't have an atomic cond_wait since it's not
>> mandatory by POSIX ;-)
>> It's just you have to treat it that way since there's no guaranty that
>> you can rely on an atomic implementation.
>> It's no "problem" at all, it's just one aspect you have to keep in 
>> mind.
>>
>> I introduced that with "where it's not atomic" because that's the very
>> assumption in this discussion. None of the arguments would be true if 
>> it
>> was.
>>
>> By definition, a con_wait is used for waiting on some event inside a
>> critical section. This means you entered the cond_wait with the mutex
>> held. Now the cond_wait call enqueues the thread into the queue of
>> threads waiting on this condition variable and unlocks the mutex. If 
>> you
>> use the broadcast call protected by the same mutex you will always 
>> have a
>> "settled" environment. This means you're sure you don't have any 
>> threads
>> inside the critical section since you won't get to do broadcast (you
>> would be waiting on the mutex_lock) and during the broadcast you won't
>> have any threads entering the critical section. I can just repeat 
>> myself.
>> It's more safe but from my point of view not a fix to the problem we 
>> had.
>
>
>
> There seems to be a certain amount of confusion here about how 
> condition variables work.  The wait operation on a CV is atomic with 
> respect to the broadcast operation on the same CV and to that CV's 
> mutex.  That is, it is impossible for the broadcast to happen 
> "between" when cond_wait releases the mutex and when the thread 
> calling cond_wait becomes blocked on the CV. This is because 
> conceptually, those operations occur at the same time.
>
> CV's are hardly a uniquely POSIX concept, but since we are talking 
> here about code written against the pthreads API, we can rely on 
> POSIX's description of how they work, which has this to say:
>
>
>        These functions [pthread_cont_timedwait, pthread_cond_wait]
>        atomically release _mutex_ and cause the calling thread to 
> block on
>        the condition variable _cond_; atomically here means "atomically
>        with respect to access by another thread to the mutex and then 
> the
>        condition variable".  That is, if another thread is able to 
> acquire
>        the mutex after the about-to-block thread has released it, then 
> a
>        subsequence call to pthread_cond_broadcast() or
>        pthread_cond_signal() in that thread shall behave as if it were
>        issued after the about-to-block thread has blocked.
>
>        -- IEEE Std. 1003.1-2001, p. 1032
>

That's one passage I didn't post in my last postings, which actually 
started the fire... ;-)
I still don't see the confusion. It's sort of what I said in the first 
place.
You still can hold the mutex but miss the broadcast and wait forever 
there ...

> Also:
>
>        The pthread_cond_broadcast() or pthread_cond_signal() functions 
> may
>        be called by a thread whether or not it currently owns the mutex
>        that threads calling pthread_cond_wait() or
>        pthread_cond_timedwait() have associated with the condition
>        variable during their waits; however, if predictable scheduling
>        behaviour is required, then that mutex shall be locked by the
>        thread calling pthread_cond_broadcast() or 
> pthread_cond_signal().
>
>        -- IEEE Std. 1003.1-2001, p. 1025
>

If you take a look a few days back, that's what I posted. So, I still 
don't see the confusion.

> In other words, if you want any control over when the waiting threads 
> wake up, you need to hold the mutex, because they can't wake up while 
> you are holding the mutex.

That's one point the other is, you can be in the critical section with 
one thread and broadcasting the others,
which as I pointed out for I have no idea how many times now, is _not_ 
a race condition.

>
> We don't need that kind of control.  What we do need is the guarantee 
> that once the waiting thread acquires the mutex, it won't "miss" an 
> event due to something being put on the queue and cond_broadcast() 
> being called between when the waiting thread checks the queue and when 
> it becomes blocked on the CV.  Since it holds the mutex for that 
> entire duration, it is sufficient to prevent _either_ modification of 
> the queue _or_ the broadcast while the waiting thread is holding the 
> mutex; it is not necessary to prevent both when they only happen in 
> sequence.
>
>

I'm not sure what people did get wrong from my previous statements but 
that is more or less what you said here.
There still is this theoretical possibility where the thread will be 
waiting forever on the cv, but let's put that aside.

NOW I'm confused what your point is here since it looks pretty much 
like we're talking about the _same_ thing in many different ways.

Horst