[OpenAFS-devel] RX thread quota & scaling

[Rainer Toebbicke]

We've been playing around with an 8-core file server serving a massive 
amount of clients (thousands) in a stress test. The result was 
pathetic despite cranking up to 512 threads and not vbusying any 
requests:

the simple scanning of the incoming call queue in order to implement 
per-service thread quota monopolized about 12% of the CPU (we 
typically ran with 3000 calls waiting for a thread), which on an 
8-core this means that basically a complete CPU was in that loop which 
is run through under a lock.

Removing the per-service thread-quota handling short-circuited the 
queue handling  and we got 30% more traffic out of the box than 
previously in a quick hack. At the only expense that now xstat is 
about as slow as the rest.

Now, one fix would be to postulate that all calls are served 
first-come-first-serve regardless of rx service. But this of course 
means a change in semantics.

A more sophisticated solution would be to implement different 
incoming_call_queues per service, start the threads right away only on 
the service they are meant for. These preserves the original semantics.

My question is: does anybody care for the "sophisticated" solution? As 
far as I can see, "servers" normally use one rx service for stats and 
the other for normal serving, similar for voting and serving in the 
ubik case. The only slightly more elaborate one is the kaserver. All 
those probably run fine with a single thread pool and 
first-come-first-serve.

Any opinions?


-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Rainer Toebbicke
European Laboratory for Particle Physics(CERN) - Geneva, Switzerland
Phone: +41 22 767 8985       Fax: +41 22 767 7155