[OpenAFS-devel] Vos dump "performance"

[Jeffrey Altman]

This is a cryptographically signed message in MIME format.

--------------ms000307080402010803070305
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 5/19/2015 8:54 AM, Harald Barth wrote:
>=20
>> 1444 is the number of octets after subtracting the ip/ip6 and udp/udp6=

>> headers for a network with MTU of 1500.
>=20
> Yes but here I was on localhost and that has a loopback does have an
> MTU of 16436. If the MTU detection code does the right thing(TM).

Unlike IPv6 when using IPv4 there is no reliable path mtu detection for
UDP.  Nor is there a reliable method of delivering fragmented packets.
Rx doesn't even know which interface the packet is going to be sent
over.  What Rx sees is the set of MTUs for all interfaces and must
select the smallest of the set.   Sending a packet larger than that
might result in fragmentation and an undeliverable packet.

With IPv6 the rules change.

Rx provides mtu advertisements to the peers but this mechanism is not
helpful when there is more than one router/switch between the two peers
since neither peer has a complete view of the path mtu and the paths
which packets travel can be distinct in each direction.

>> Remember that Rx takes
>> advantage of larger network MTUs via the use of Jumbograms which combi=
ne
>> multiple Rx packets into one UDP message.
>=20
>> Also remember that jumbo
>> grams are disabled by default because of the negative impact that occu=
rs
>> when using them over the public Internet.
>=20
> I think there are two mechanisms which may interact:
>=20
> 1. packing multiple RX packets into one UDP packet (mostly a good
>    thing unless high packet drop)
>=20
> 2. making UDP packets bigger than MTU (only a good thing if one knows
>    that there is almost no packet drop and reassembly is cheap.

An Rx packet is never larger than 1500 - Header Sizes.  A jumbogram
consists of many of these Rx packet data payloads combined into one UDP
packet so that if the large packet cannot be delivered it is possible to
transmit the Rx packets individually.  This permits the Rx packet
acknowledgment scoreboard to properly indicate which packets have been
received and which haven't when using jumbograms.

>> The biggest cause of the performance problems you are seeing is Rx.
>=20
> My newest measurements make me believe that that's actually not the
> case because my numbers are equally bad when using voldump or tar
> after forcibly throwing away the buffer cache. No rx involved at all.
> But when I use tar multiple times buffer cache seems to help. When
> using voldump not so. I'll have to confirm my measurements by more
> runs (which take each approx 2.5h). Some of my numbers I get from
> strace feel fishy, so I have to confirm them again.

One interpretation of your results is that buffering can help with tar
because the data source can fill the buffer but with Rx the data source
cannot.

>> The
>> window size is small and every time that the sender doesn't have data =
to
>> send the connection stalls.
>=20
> True.
>=20
>> Every time the reader performs disk i/o it
>> stops reading from the stream and the Rx connection stalls.
>=20
> Does not seem to be a problem in my case because even when I write to
> /dev/null the numbers are the same.

I used "disk i/o" to describe the steps of parsing the input stream,
deciding what to writing, and writing it.   The actual device is less
important.  When the application is performing these steps the Rx
transfer is stalling.

>> Volume moves, releases and dumps are some of the operations in which t=
he
>> AuriStor Rx stack performance changes are quite noticeable.
>=20
> ;-)
>=20
> Harald.





--------------ms000307080402010803070305
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIINXTCC
BkIwggUqoAMCAQICEDirAC//rpa3Vv85Wvtd5xswDQYJKoZIhvcNAQEFBQAwgcoxCzAJBgNV
BAYTAlVTMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1
c3QgTmV0d29yazE6MDgGA1UECxMxKGMpIDE5OTkgVmVyaVNpZ24sIEluYy4gLSBGb3IgYXV0
aG9yaXplZCB1c2Ugb25seTFFMEMGA1UEAxM8VmVyaVNpZ24gQ2xhc3MgMSBQdWJsaWMgUHJp
bWFyeSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEczMB4XDTExMDkwMTAwMDAwMFoXDTIx
MDgzMTIzNTk1OVowgaYxCzAJBgNVBAYTAlVTMR0wGwYDVQQKExRTeW1hbnRlYyBDb3Jwb3Jh
dGlvbjEfMB0GA1UECxMWU3ltYW50ZWMgVHJ1c3QgTmV0d29yazEeMBwGA1UECxMVUGVyc29u
YSBOb3QgVmFsaWRhdGVkMTcwNQYDVQQDEy5TeW1hbnRlYyBDbGFzcyAxIEluZGl2aWR1YWwg
U3Vic2NyaWJlciBDQSAtIEc0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxuwn
/R1j9DsdisHTHMjIgoa2uEqGkqqBXHLKMA0vnkEiVzAhJZCao/SsKsaIF4ZhchN2LuwDyyeb
jyCAN+DkitpVplAP/LlcI2mJQqG6H6/vDvmkyQrx+DeyxtmSSq5937hEH5u6P4wG/tgjT0hR
I2pghKjuJy9g35byGiqMPI8AzE/L+iCOvDX24fCatgXz/B0/xhR7DtryBeTTgwKmxWlwtKnk
VunbHVz0pjbia7UeKi3cvrvuOgSwMAitX2hsxr0GloiE5+apZC28ODC7iCbDZ2ZmtLR3+cCh
xw5y72bi5bnK4POFdzWY3tQcsP5mceI4y258T0BV65fZqBge7QIDAQABo4ICRDCCAkAwOAYI
KwYBBQUHAQEELDAqMCgGCCsGAQUFBzABhhxodHRwOi8vcGtpLW9jc3AudmVyaXNpZ24uY29t
MBIGA1UdEwEB/wQIMAYBAf8CAQAwbAYDVR0gBGUwYzBhBgtghkgBhvhFAQcXATBSMCYGCCsG
AQUFBwIBFhpodHRwOi8vd3d3LnN5bWF1dGguY29tL2NwczAoBggrBgEFBQcCAjAcGhpodHRw
Oi8vd3d3LnN5bWF1dGguY29tL3JwYTA0BgNVHR8ELTArMCmgJ6AlhiNodHRwOi8vY3JsLnZl
cmlzaWduLmNvbS9wY2ExLWczLmNybDAOBgNVHQ8BAf8EBAMCAQYwKQYDVR0RBCIwIKQeMBwx
GjAYBgNVBAMTEVZlcmlTaWduTVBLSS0yLTk3MB0GA1UdDgQWBBSt+cOTci21uShh5KTXYNXE
Cl4aATCB8QYDVR0jBIHpMIHmoYHQpIHNMIHKMQswCQYDVQQGEwJVUzEXMBUGA1UEChMOVmVy
aVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlTaWduIFRydXN0IE5ldHdvcmsxOjA4BgNVBAsT
MShjKSAxOTk5IFZlcmlTaWduLCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxRTBD
BgNVBAMTPFZlcmlTaWduIENsYXNzIDEgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBB
dXRob3JpdHkgLSBHM4IRAItbdVaEVIULAM+vOEjOsaQwDQYJKoZIhvcNAQEFBQADggEBANaP
wdqbiPKzbE0fWC+6AVFddMFG6MO4e5/WQPHv/zK6iWvADjRDn6SZ5qTwXUgzYoWFYf4jiCKM
YJsrnGVJlMSiOCRIpVylUEto6WIip5PomSJuPVu7EEIOH0x1RzRWCY/4vYw881y70pZwVHBi
Te/REL6dSCxe7IZrB4LwPeElJygs4BZ2HrP95WKW0oo9Xyuu+1zCE7dlY8s0dkOf1oeZq26t
lcEAP0Yngf813iMOQ9wUXzL5yinvwlIw9ZnduYH4OiUgjYJo8rkhhXRmBOGGORYy8i3WKqjJ
3tkAAk/jGCDFpYFWtpXe04Kt+HslvmR8LqC6cCz4+XXidE0HbYQwggcTMIIF+6ADAgECAhAW
xDBJuKAcJVa7b+TJhwvEMA0GCSqGSIb3DQEBBQUAMIGmMQswCQYDVQQGEwJVUzEdMBsGA1UE
ChMUU3ltYW50ZWMgQ29ycG9yYXRpb24xHzAdBgNVBAsTFlN5bWFudGVjIFRydXN0IE5ldHdv
cmsxHjAcBgNVBAsTFVBlcnNvbmEgTm90IFZhbGlkYXRlZDE3MDUGA1UEAxMuU3ltYW50ZWMg
Q2xhc3MgMSBJbmRpdmlkdWFsIFN1YnNjcmliZXIgQ0EgLSBHNDAeFw0xNDEyMTgwMDAwMDBa
Fw0xNTEyMTkyMzU5NTlaMIHOMS4wLAYDVQQDDCVQZXJzb25hIE5vdCBWYWxpZGF0ZWQgLSAx
NDE4ODgyMzg2MTAyMSswKQYJKoZIhvcNAQkBFhxqYWx0bWFuQHlvdXItZmlsZS1zeXN0ZW0u
Y29tMQ8wDQYDVQQLDAZTL01JTUUxHjAcBgNVBAsMFVBlcnNvbmEgTm90IFZhbGlkYXRlZDEf
MB0GA1UECwwWU3ltYW50ZWMgVHJ1c3QgTmV0d29yazEdMBsGA1UECgwUU3ltYW50ZWMgQ29y
cG9yYXRpb24wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCteTFLbuPm2vx85lH2
Y6LHdMIqcKQPN9m4XYVTe0L8ZvMvnJ1YQ720ET52CF18RYdTc4to92+ffdmjWlBedK4YtLam
htsUkJ6WL3krwNVTYfeej0wgF9kVQ2FI8XTNngxnJ2CRkQX4Z9/1TI4wTkSNcAw5T0Y2HKM9
4p7wAJOefl+3oPwxXn338w8T7LsfwS9FOADZ8uRItv/J7S8BJEP0XtjZZlBoSyqQ4qxl5PtI
uybHVdwoo2PlK8LxU6r8Vcje1+OXmc5VoCBlXiYDWHl/wSDtZEWR6431/az1Z45n1AHHber2
G+Ijb0nF4RLiiFMkyJl3qx+46wqcqWrjrRxDAgMBAAGjggMRMIIDDTAMBgNVHRMBAf8EAjAA
MA4GA1UdDwEB/wQEAwIFoDAgBgNVHSUBAf8EFjAUBggrBgEFBQcDBAYIKwYBBQUHAwIwHQYD
VR0OBBYEFBUlv/9jizZz4uwaFtPzwdX6FsqwMCcGA1UdEQQgMB6BHGphbHRtYW5AeW91ci1m
aWxlLXN5c3RlbS5jb20wHwYDVR0jBBgwFoAUrfnDk3IttbkoYeSk12DVxApeGgEwggErBggr
BgEFBQcBAQSCAR0wggEZMIIBFQYIKwYBBQUHMAKGggEHbGRhcDovL2RpcmVjdG9yeS52ZXJp
c2lnbi5jb20vQ04lMjAlM0QlMjBTeW1hbnRlYyUyMENsYXNzJTIwMSUyMEluZGl2aWR1YWwl
MjBTdWJzY3JpYmVyJTIwQ0ElMjAtJTIwRzQlMkMlMjBPVSUyMCUzRCUyMFBlcnNvbmElMjBO
b3QlMjBWYWxpZGF0ZWQlMkMlMjBPVSUyMCUzRCUyMFN5bWFudGVjJTIwVHJ1c3QlMjBOZXR3
b3JrJTJDJTIwTyUyMCUzRCUyMFN5bWFudGVjJTIwQ29ycG9yYXRpb24lMkMlMjBDJTIwJTNE
JTIwVVM/Y0FDZXJ0aWZpY2F0ZTtiaW5hcnkwXQYDVR0fBFYwVDBSoFCgToZMaHR0cDovL3Br
aS1jcmwuc3ltYXV0aC5jb20vY2FfNTYxYzEwMzY5MGM5N2E2OTI0N2EwZWYwNzFhYzgxYWYv
TGF0ZXN0Q1JMLmNybDBsBgNVHSAEZTBjMGEGC2CGSAGG+EUBBxcBMFIwJgYIKwYBBQUHAgEW
Gmh0dHA6Ly93d3cuc3ltYXV0aC5jb20vY3BzMCgGCCsGAQUFBwICMBwaGmh0dHA6Ly93d3cu
c3ltYXV0aC5jb20vcnBhMCsGCmCGSAGG+EUBEAMEHTAbBhJghkgBhvhFARABAgIEAYbHzm8W
BTEwOTIyMDkGCmCGSAGG+EUBEAUEKzApAgEAFiRhSFIwY0hNNkx5OXdhMmt0Y21FdWMzbHRZ
WFYwYUM1amIyMD0wDQYJKoZIhvcNAQEFBQADggEBAJIvoMatM56/QjaVAlEUbeWZNOPkwuiC
gaaekWJi1h33fAVfdF+WZqh7dF4hBNalyPuxRcyZX7HxuPyBc3ajDCqew9MlmCJ3Cm4Co3fZ
Yh50OX4jnem2RmpHeKbJW6zUjZcAGqV6DPMl04kgrI2whJX7729HoRyUPwZS7CZSRFZO147X
2+/JogDYKffa/+q5AwgrHYvdECHxc3Iz9KnHSmit+DhWS9t+XxE0gHr3sW7zwcQ/GYyrJ3s3
VdWHTjM+3iGFeTOI06h1aBgFR/+8fTmuZXZz9+OdWVar0Crt9bn0cFN3u00Q6YAyjYhRnbXy
zYVIOS4oJmRoK79p/xodeDUxggRSMIIETgIBATCBuzCBpjELMAkGA1UEBhMCVVMxHTAbBgNV
BAoTFFN5bWFudGVjIENvcnBvcmF0aW9uMR8wHQYDVQQLExZTeW1hbnRlYyBUcnVzdCBOZXR3
b3JrMR4wHAYDVQQLExVQZXJzb25hIE5vdCBWYWxpZGF0ZWQxNzA1BgNVBAMTLlN5bWFudGVj
IENsYXNzIDEgSW5kaXZpZHVhbCBTdWJzY3JpYmVyIENBIC0gRzQCEBbEMEm4oBwlVrtv5MmH
C8QwCQYFKw4DAhoFAKCCAmswGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0B
CQUxDxcNMTUwNTE5MTMyNzQwWjAjBgkqhkiG9w0BCQQxFgQUMvDE6QU8JLhZnKG4mrn1HLZG
/OowbAYJKoZIhvcNAQkPMV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3
DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0D
AgIBKDCBzAYJKwYBBAGCNxAEMYG+MIG7MIGmMQswCQYDVQQGEwJVUzEdMBsGA1UEChMUU3lt
YW50ZWMgQ29ycG9yYXRpb24xHzAdBgNVBAsTFlN5bWFudGVjIFRydXN0IE5ldHdvcmsxHjAc
BgNVBAsTFVBlcnNvbmEgTm90IFZhbGlkYXRlZDE3MDUGA1UEAxMuU3ltYW50ZWMgQ2xhc3Mg
MSBJbmRpdmlkdWFsIFN1YnNjcmliZXIgQ0EgLSBHNAIQFsQwSbigHCVWu2/kyYcLxDCBzgYL
KoZIhvcNAQkQAgsxgb6ggbswgaYxCzAJBgNVBAYTAlVTMR0wGwYDVQQKExRTeW1hbnRlYyBD
b3Jwb3JhdGlvbjEfMB0GA1UECxMWU3ltYW50ZWMgVHJ1c3QgTmV0d29yazEeMBwGA1UECxMV
UGVyc29uYSBOb3QgVmFsaWRhdGVkMTcwNQYDVQQDEy5TeW1hbnRlYyBDbGFzcyAxIEluZGl2
aWR1YWwgU3Vic2NyaWJlciBDQSAtIEc0AhAWxDBJuKAcJVa7b+TJhwvEMA0GCSqGSIb3DQEB
AQUABIIBAHFV1lwUYgOahIuI4Mp4lLCgaRx8Ei1qp4wCrPFtb/Z+PsSyeapoTaJzgzhOYANf
jhyktvYoOp68li/xZ0uSNOgG+T572ngO5aenu1c0fSBWvxer1Dt6qjgOBwvAsRaeAzdc02gE
/Q+7kylz5Q5754xKhb/x5L6iTW+J2kyEQNyliLAUUt2PsHb1B/jnHr1mAnxn6+i15xS25EXL
LUg6OeR6VYy/A/ZxrY+EtM8ZMIZDUHvGwPFihcjDuVUycWs9h/K1xT+Y6m1wts8dmxUDMej7
AE2RjKnGRQwQzhtiyltVAthlIAV7xER2X56OI8IS2mZ/MoU/f8HgmM8gItQAhksAAAAAAAA=
--------------ms000307080402010803070305--