[OpenAFS] Re: Fwd: Re: vos move slowness remedies?

[Jeffrey Altman]

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enigD21A60224EE38957DDED8653
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 10/26/2010 4:26 PM, Dan Pritts wrote:
>=20
> On Oct 26, 2010, at 1:56 PM, Eric Chris Garrison wrote:
>>
>> [root@rfs4 ~]# dumpcap -w /tmp/ecg1.dumpcap -a duration:60
>> File: /tmp/ecg1.dumpcap
>> Packets: 201081 Packets dropped: 28064
>=20
>> So, nothing dropped for a straight iperf test at almost full bandwidth=
,
>> but over 10% dropped at 30Mb/sec for AFS vos moves.
>=20
> If you are getting the "10% dropped" number from your wireshark output,=

> it is a red herring.  That is telling you how many packets the kernel
> was unable to copy to wireshark, not anything about what goes on  on
> your network.

Not necessarily.  The rx implementation has a single thread that
reads all incoming packets and has to manage new rx connections,
process all rx packets and either queue them on the appropriate
call or respond to them directly, and handle ack/nak processing.
There is a lot of overhead that gets in the way of pulling off
packets from the wire.  The problems get worse as the window size
increases.

Your File System developers have spent the last couple of months
refactoring the source code in order to reduce the overhead.
Some of the changes are minor such as avoiding calling gettimeofday()
for every packet processed and repeatedly allocating and deallocating
the xmit list array.  Other changes are more significant relating to
how locking is performed, the use of atomic operations, how round trip
times are calculated, how timeout periods are handled, under what
conditions fast recovery is trigger, and which thread is responsible
for transmiting acks/naks.

Many of these improvements are now on master and there are more to
come in the near future.

When the rx listener thread is unable to read packets from the network
and those packets are dropped the overall throughput of the rx
connection collapses quite rapidly.

On master in the src/rx/tests directory is a tool called rxperf that
can be used to simulated a variety of rx communication scenarios.
It would be interesting to see how rxperf performs across Eric's
network path using both 1.4 and master.  Note that rxperf may require
back porting to 1.4 to get it to build.

Jeffrey Altman


--------------enigD21A60224EE38957DDED8653
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)

iQEcBAEBAgAGBQJMx2NVAAoJENxm1CNJffh4NcIH/3m7XrdSC+DbioodyNGl0Rty
xdAbqNsa7uau1AuUXDf+kPIiIBzM78Wk2BjLYF9qIasd7MULAVoXHKLXtW+/4PyH
rMqJKh0XcR4xjtNObonrZxl0zQBPiJ1GSsVCCDm0lHjBTQI0DQTGkjXoMtNWVy5O
3ONp8uWw3isYstpotjo51WC/uEJZliIoTJo+MaC42+0oyLHyAvLyY28wYT1D8QFo
CbPMdqpSIOXUs41wbZ0CVwTz75gHc3SquU7iPNJ4atr73HoGyw//zJZADGWg0MLa
vHYhatuUXrQHc1o1Jqaxpi8WKJrz2hOWBsJtSemT104oJvaRasVbVxvu35x3C80=
=qn+x
-----END PGP SIGNATURE-----

--------------enigD21A60224EE38957DDED8653--