[OpenAFS-devel] performance study

[Ken Hornstein]

>Fixing this brought the speed from 20-30 MB/s to >110 MB/s (memory to 
>memory, LAN). What was funny is that Hartmut ran my program without any 
>tweaks and got 114 MB/s immediatly. The only explanation I have is that 
>he hit some sort of "sweet spot" or "standing wave" on the two identical 
>uni-processor machines he tried it on.

This is comforting.  Your results roughly match what I have seen.  Maybe
Hartmut has some extra special networking gear we don't know about :-)

>Another one: RX has an implicit limit of a send/receive window of 255 
>packets. For trans-atlantic traffic that's not too much with a standard 
>MTU of 1416 (or so) bytes. In 1.3.x the variables are 32 bit numbers, 
>but the offset for the ACKs is still a u_char and that sits in the 
>packet. Perhaps fixable by sending several small ACK packets.

I thought that I found that the maximum window size was capped at 32 packets
(rx_maxSendWindow seems to be set to 32).

>RX over TCP might be interesting - if it scales! I'm a bit worried about 
>file servers doing poll() (forget select()) on 10000+ TCP connections...

I think that has been a solved problem for a little while now.  Most modern
operating systems have some sort of system to handle lots of descriptors.
Solaris has /dev/poll, the BSD-derived systems have kqueue, and I know that
Linux has something similar.

>Besides that probably much more thought went into TCP than into RX also 
>comes the fact that the routers very likely are better tuned to TCP.

That's my thought as well.

--Ken