[OpenAFS-devel] .35 sec rx delay bug?

[Rainer Toebbicke]

Tom Keiser wrote:

> Ok.  Looking at your patch I see we're talking about two different rx
> subsystems.  I was referring to the Rx packet _allocator_ being
> lockless.  I see you're referring to a call->lock fairness problem
> caused by a transmitter holding the lock for too long.  Well,
> actually, that's not really true, either.  If you dig far enough down
> into the stack, you'll see that we drop call->lock around the
> rxi_SendPacketList()/rxi_SendPacket() call at the bottom of
> rxi_SendList().  On SMPs this should be enough, but on uniprocessor

No, actually it isn't: if you're grabbing, say, 64 KB (is that 
anexotic buffer size) of packets you're holding the lock for 40 or so 
packets, even across the malloc() which may eventually happen. It's 
only released once the packets are all queued for transmission. That's 
eons later, particularly if ever malloc goes through garbage 
collection and the like which I have seen, baffled. If I calculate 
this correctly, at 100 MB/s we're talking some 70000 packets per 
second! Shouldn't waste milliseconds too often.

In order to keep the network interface busy the window must stay open, 
which means every ACK has to be processed as soon as possible. Parts 
of this patch are to ensure this, i.e. give more priority to ACK 
processing on the assumption that there is always spare time later to 
queue more of the user's packets. This would also have the effect of 
freeing up packets for reuse earlier, and keep the queue smaller. A 
"useless" mutex drop/reacquire costs some 2-3us BTW on a modest Xeon.

As to Chas William's remark: gettimeofday() resp. clock_GetTime is 
called like hell. I only worried about switching to itimer() when I 
saw that the call itself already is that slow.

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Rainer Toebbicke
European Laboratory for Particle Physics(CERN) - Geneva, Switzerland
Phone: +41 22 767 8985       Fax: +41 22 767 7155