[OpenAFS-devel] RX thread quota & scaling

[Jeffrey Hutzelman]

--On Wednesday, January 23, 2008 11:47:49 AM +0100 Rainer Toebbicke 
<rtb@pclella.cern.ch> wrote:

> My question is: does anybody care for the "sophisticated" solution? As
> far as I can see, "servers" normally use one rx service for stats and the
> other for normal serving, similar for voting and serving in the ubik
> case. The only slightly more elaborate one is the kaserver. All those
> probably run fine with a single thread pool and first-come-first-serve.

Careful here.  First-come-first-served is probably fine for balancing 
fileserver requests with the occasional xstat request -- those are separate 
services mostly to maintain some modularity.

However, the ubik vote and disk services must be separate services both 
from each other and from any application service, with their own threads. 
Otherwise you may have a livelock where an application call to a 
non-coordinator is blocked waiting for a lock while the coordinator updates 
the database, but the database updates can't get through because there are 
no threads available to handle the disk RPC.  Similarly, you may lose 
quorum or see significant instability if UVOTE_Beacon RPC's are not handled 
in a timely fashion.

It would seem that one possible approach to this problem is to use separate 
queues as you describe.  Another would be to abandon the concept of 
overlapping quota, instead dedicating a specific number of workers to each 
service, and keeping counters to record the number of active and pending 
calls for each configured service.  This should allow proper scheduling 
without wasting large amounts of time scanning through the queue to find 
that there are no xstat requests pending.

-- Jeff