[OpenAFS] Re: Fwd: Re: vos move slowness remedies?

Jeffrey Altman jaltman@secure-endpoints.com
Tue, 26 Oct 2010 19:25:07 -0400

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 10/26/2010 4:26 PM, Dan Pritts wrote:
> On Oct 26, 2010, at 1:56 PM, Eric Chris Garrison wrote:
>> [root@rfs4 ~]# dumpcap -w /tmp/ecg1.dumpcap -a duration:60
>> File: /tmp/ecg1.dumpcap
>> Packets: 201081 Packets dropped: 28064
>> So, nothing dropped for a straight iperf test at almost full bandwidth=
>> but over 10% dropped at 30Mb/sec for AFS vos moves.
> If you are getting the "10% dropped" number from your wireshark output,=

> it is a red herring.  That is telling you how many packets the kernel
> was unable to copy to wireshark, not anything about what goes on  on
> your network.

Not necessarily.  The rx implementation has a single thread that
reads all incoming packets and has to manage new rx connections,
process all rx packets and either queue them on the appropriate
call or respond to them directly, and handle ack/nak processing.
There is a lot of overhead that gets in the way of pulling off
packets from the wire.  The problems get worse as the window size

Your File System developers have spent the last couple of months
refactoring the source code in order to reduce the overhead.
Some of the changes are minor such as avoiding calling gettimeofday()
for every packet processed and repeatedly allocating and deallocating
the xmit list array.  Other changes are more significant relating to
how locking is performed, the use of atomic operations, how round trip
times are calculated, how timeout periods are handled, under what
conditions fast recovery is trigger, and which thread is responsible
for transmiting acks/naks.

Many of these improvements are now on master and there are more to
come in the near future.

When the rx listener thread is unable to read packets from the network
and those packets are dropped the overall throughput of the rx
connection collapses quite rapidly.

On master in the src/rx/tests directory is a tool called rxperf that
can be used to simulated a variety of rx communication scenarios.
It would be interesting to see how rxperf performs across Eric's
network path using both 1.4 and master.  Note that rxperf may require
back porting to 1.4 to get it to build.

Jeffrey Altman

Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

Version: GnuPG v1.4.9 (MingW32)