[Port-solaris] Re: [OpenAFS] Re: mkdir() performance on AFS client

[Jeffrey Altman]

This is a cryptographically signed message in MIME format.

--------------ms010904080006090905020204
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 1/23/2014 11:53 AM, Andrew Deason wrote:
> On Thu, 23 Jan 2014 11:43:50 +0000
> Simon Wilkinson <simonxwilkinson@gmail.com> wrote:
>=20
>> The real question here is how widely we should be applying the abort
>> threshold - should it apply to all aborts sent to a particular client,=

>> or should we be more selective? There are a lot of competing views
>> here, as it depends on what you believe the abort threshold is
>> actually protecting us against.

Agreed.  Personally I'm not in favor of abort thresholds but they have
prevented a large number of file servers from drowning under the load
applied by very broken clients in the wild.

Back in the 1.2.x days there were clients that were broken in some
extreme ways.  For example a client that produced a new connection for
every RPC triggering a new TMAY and tying up a thread for a bit.
Another was a problem with users that obtained tokens in the morning and
left file browser open to their home directory.  When the tokens expired
the file browser would read status for every file in the directory tree
it knew about and fail.  Then repeat in rapid succession.

Take an org with a few hundred such desktops triggering the same
behavior ten hours after the work day starts and the file servers would
fall over each night.

Lest you say that EEXIST and ENOTEMPTY are different.  Well, we have
seen cache corruption that makes the cache manager think the file does
exist when it does.  This has triggered the application to retry
creation of the file over and over in a loop.

The primary purpose of the abort threshold is to protect the file server
from an abusive client whether or not the client intends to be abusive.

>=20
> Well, the issue also goes away if we do either of two other things that=

> are advantageous for other reasons:
>=20
>  - Don't issue RPCs we know will fail (mkdir EEXIST, rmdir ENOTEMPTY,
>    maybe others). Even without the abort threshold, this causes an
>    unnecessary delay waiting for the fileserver to respond. This is
>    really noticeable with git in AFS.

I am firmly in the camp that says the cache manager should avoid sending
any RPC it can assume with good justification will fail based upon known
state and file server rules.  The EEXIST and ENOTEMPTY certainly fall
into that case.  So do things like create file, create directory,
unlink, etc when it is known the permissions are wrong.

The same for writing to a file when it is known that quota has been
exceeded.

The Windows cache manager even takes things a step further by
maintaining a negative cache for EACCESS errors on {FID, user}.  This
has avoided hitting the abort threshold limits triggered by Windows that
assumes that if it can list a directory it must be able to read the
status of all the objects within it.

>  - Don't raise aborts for every single kind of error (like
>    InlineBulkStatus). Aborts aren't secure, for one thing.

InlineBulkStatus can throw aborts.  It just doesn't do so when an
individual FetchStatus on an included FID fails.  The problem with this
approach is that OpenAFS cannot turn off the existing RPC support and
will always need abort thresholds to protect the file servers.  At least
until it becomes possible for a file server to scale with the load so
that a pool of misbehaving clients cannot deny service to other clients.

Jeffrey Altman







--------------ms010904080006090905020204
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIINITCC
BkIwggUqoAMCAQICEDirAC//rpa3Vv85Wvtd5xswDQYJKoZIhvcNAQEFBQAwgcoxCzAJBgNV
BAYTAlVTMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1
c3QgTmV0d29yazE6MDgGA1UECxMxKGMpIDE5OTkgVmVyaVNpZ24sIEluYy4gLSBGb3IgYXV0
aG9yaXplZCB1c2Ugb25seTFFMEMGA1UEAxM8VmVyaVNpZ24gQ2xhc3MgMSBQdWJsaWMgUHJp
bWFyeSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAtIEczMB4XDTExMDkwMTAwMDAwMFoXDTIx
MDgzMTIzNTk1OVowgaYxCzAJBgNVBAYTAlVTMR0wGwYDVQQKExRTeW1hbnRlYyBDb3Jwb3Jh
dGlvbjEfMB0GA1UECxMWU3ltYW50ZWMgVHJ1c3QgTmV0d29yazEeMBwGA1UECxMVUGVyc29u
YSBOb3QgVmFsaWRhdGVkMTcwNQYDVQQDEy5TeW1hbnRlYyBDbGFzcyAxIEluZGl2aWR1YWwg
U3Vic2NyaWJlciBDQSAtIEc0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxuwn
/R1j9DsdisHTHMjIgoa2uEqGkqqBXHLKMA0vnkEiVzAhJZCao/SsKsaIF4ZhchN2LuwDyyeb
jyCAN+DkitpVplAP/LlcI2mJQqG6H6/vDvmkyQrx+DeyxtmSSq5937hEH5u6P4wG/tgjT0hR
I2pghKjuJy9g35byGiqMPI8AzE/L+iCOvDX24fCatgXz/B0/xhR7DtryBeTTgwKmxWlwtKnk
VunbHVz0pjbia7UeKi3cvrvuOgSwMAitX2hsxr0GloiE5+apZC28ODC7iCbDZ2ZmtLR3+cCh
xw5y72bi5bnK4POFdzWY3tQcsP5mceI4y258T0BV65fZqBge7QIDAQABo4ICRDCCAkAwOAYI
KwYBBQUHAQEELDAqMCgGCCsGAQUFBzABhhxodHRwOi8vcGtpLW9jc3AudmVyaXNpZ24uY29t
MBIGA1UdEwEB/wQIMAYBAf8CAQAwbAYDVR0gBGUwYzBhBgtghkgBhvhFAQcXATBSMCYGCCsG
AQUFBwIBFhpodHRwOi8vd3d3LnN5bWF1dGguY29tL2NwczAoBggrBgEFBQcCAjAcGhpodHRw
Oi8vd3d3LnN5bWF1dGguY29tL3JwYTA0BgNVHR8ELTArMCmgJ6AlhiNodHRwOi8vY3JsLnZl
cmlzaWduLmNvbS9wY2ExLWczLmNybDAOBgNVHQ8BAf8EBAMCAQYwKQYDVR0RBCIwIKQeMBwx
GjAYBgNVBAMTEVZlcmlTaWduTVBLSS0yLTk3MB0GA1UdDgQWBBSt+cOTci21uShh5KTXYNXE
Cl4aATCB8QYDVR0jBIHpMIHmoYHQpIHNMIHKMQswCQYDVQQGEwJVUzEXMBUGA1UEChMOVmVy
aVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlTaWduIFRydXN0IE5ldHdvcmsxOjA4BgNVBAsT
MShjKSAxOTk5IFZlcmlTaWduLCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxRTBD
BgNVBAMTPFZlcmlTaWduIENsYXNzIDEgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBB
dXRob3JpdHkgLSBHM4IRAItbdVaEVIULAM+vOEjOsaQwDQYJKoZIhvcNAQEFBQADggEBANaP
wdqbiPKzbE0fWC+6AVFddMFG6MO4e5/WQPHv/zK6iWvADjRDn6SZ5qTwXUgzYoWFYf4jiCKM
YJsrnGVJlMSiOCRIpVylUEto6WIip5PomSJuPVu7EEIOH0x1RzRWCY/4vYw881y70pZwVHBi
Te/REL6dSCxe7IZrB4LwPeElJygs4BZ2HrP95WKW0oo9Xyuu+1zCE7dlY8s0dkOf1oeZq26t
lcEAP0Yngf813iMOQ9wUXzL5yinvwlIw9ZnduYH4OiUgjYJo8rkhhXRmBOGGORYy8i3WKqjJ
3tkAAk/jGCDFpYFWtpXe04Kt+HslvmR8LqC6cCz4+XXidE0HbYQwggbXMIIFv6ADAgECAhA5
oFEXaG88XscBgkTPSsu4MA0GCSqGSIb3DQEBBQUAMIGmMQswCQYDVQQGEwJVUzEdMBsGA1UE
ChMUU3ltYW50ZWMgQ29ycG9yYXRpb24xHzAdBgNVBAsTFlN5bWFudGVjIFRydXN0IE5ldHdv
cmsxHjAcBgNVBAsTFVBlcnNvbmEgTm90IFZhbGlkYXRlZDE3MDUGA1UEAxMuU3ltYW50ZWMg
Q2xhc3MgMSBJbmRpdmlkdWFsIFN1YnNjcmliZXIgQ0EgLSBHNDAeFw0xMzEyMjMwMDAwMDBa
Fw0xNTAxMTYyMzU5NTlaMIHOMS4wLAYDVQQDDCVQZXJzb25hIE5vdCBWYWxpZGF0ZWQgLSAx
MzU4Mjc1NTk5Njg2MSswKQYJKoZIhvcNAQkBFhxqYWx0bWFuQHNlY3VyZS1lbmRwb2ludHMu
Y29tMQ8wDQYDVQQLDAZTL01JTUUxHjAcBgNVBAsMFVBlcnNvbmEgTm90IFZhbGlkYXRlZDEf
MB0GA1UECwwWU3ltYW50ZWMgVHJ1c3QgTmV0d29yazEdMBsGA1UECgwUU3ltYW50ZWMgQ29y
cG9yYXRpb24wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCtcVgrUA+Zl527P0yx
xfkiLZzymUZLAwif9amX6c79OHd17CN5Hg3TIRXi0exjq/z/K2eftbSfj3XatgllPtlmCktq
i4daJTVLifx5G7qbcYjQgKpex/8FA3gfvJJNgMef5OwOTE1HqMJsp5CAquFjw/ReiXQqduou
1nHhUhX1AXBpaQmYDOQZwrTD41yT7qm22N67vV0viWG9x/1RdFqqtIOIyKR+ojD3IN0wufES
gy7hsWgmghh4jkrclmMIXuo+AAtmJHzwzF4hCrSqdwiRXU4aghTjsmehtMT1nFfzDPylaclO
p7IY4xeQm/q1cbU3uEqxhy06sYeVbh6zfTDBAgMBAAGjggLVMIIC0TAMBgNVHRMBAf8EAjAA
MA4GA1UdDwEB/wQEAwIFoDAgBgNVHSUBAf8EFjAUBggrBgEFBQcDBAYIKwYBBQUHAwIwHQYD
VR0OBBYEFLNBzIZb/xB6K+oeDwfBVLaB3qbUMCcGA1UdEQQgMB6BHGphbHRtYW5Ac2VjdXJl
LWVuZHBvaW50cy5jb20wHwYDVR0jBBgwFoAUrfnDk3IttbkoYeSk12DVxApeGgEwggErBggr
BgEFBQcBAQSCAR0wggEZMIIBFQYIKwYBBQUHMAKGggEHbGRhcDovL2RpcmVjdG9yeS52ZXJp
c2lnbi5jb20vQ04lMjAlM0QlMjBTeW1hbnRlYyUyMENsYXNzJTIwMSUyMEluZGl2aWR1YWwl
MjBTdWJzY3JpYmVyJTIwQ0ElMjAtJTIwRzQlMkMlMjBPVSUyMCUzRCUyMFBlcnNvbmElMjBO
b3QlMjBWYWxpZGF0ZWQlMkMlMjBPVSUyMCUzRCUyMFN5bWFudGVjJTIwVHJ1c3QlMjBOZXR3
b3JrJTJDJTIwTyUyMCUzRCUyMFN5bWFudGVjJTIwQ29ycG9yYXRpb24lMkMlMjBDJTIwJTNE
JTIwVVM/Y0FDZXJ0aWZpY2F0ZTtiaW5hcnkwXQYDVR0fBFYwVDBSoFCgToZMaHR0cDovL3Br
aS1jcmwuc3ltYXV0aC5jb20vY2FfNTYxYzEwMzY5MGM5N2E2OTI0N2EwZWYwNzFhYzgxYWYv
TGF0ZXN0Q1JMLmNybDBsBgNVHSAEZTBjMGEGC2CGSAGG+EUBBxcBMFIwJgYIKwYBBQUHAgEW
Gmh0dHA6Ly93d3cuc3ltYXV0aC5jb20vY3BzMCgGCCsGAQUFBwICMBwaGmh0dHA6Ly93d3cu
c3ltYXV0aC5jb20vcnBhMCoGCmCGSAGG+EUBEAMEHDAaBhFghkgBhvhFARABAgIEAYazFxYF
MTA5MjIwDQYJKoZIhvcNAQEFBQADggEBAACPkJV5NIxzjKc+WveaoM8Uc86wX0yLBm1A33z4
rLVXTWPi5kMIJ6kfE+dFWcMdyyOgZ2VcxIwneZ50LcITFv1VOfRkrX32vVChQqs8XGqerIo/
K3epyFEg01qHq/4byolXW6UOvmZb3oHhtHDGS94Vv6Fu6wV7irAdoM18cqzQsxU0nZDMnY5k
0pKJHLTrsC/uKuoWGz8xLLyeayi37ZsXsbGdazqzVMIoLvFTMjaFuoCetEbiFQZvnuHKwdbV
YqyCY28Cl8DVRHrInZrz84xqFiGZNSfFRWOougT47VRDA8SVy6pOtDaOmkxcYXlh5Ezo29FB
OiA0+tF8qgMmq3QxggRSMIIETgIBATCBuzCBpjELMAkGA1UEBhMCVVMxHTAbBgNVBAoTFFN5
bWFudGVjIENvcnBvcmF0aW9uMR8wHQYDVQQLExZTeW1hbnRlYyBUcnVzdCBOZXR3b3JrMR4w
HAYDVQQLExVQZXJzb25hIE5vdCBWYWxpZGF0ZWQxNzA1BgNVBAMTLlN5bWFudGVjIENsYXNz
IDEgSW5kaXZpZHVhbCBTdWJzY3JpYmVyIENBIC0gRzQCEDmgURdobzxexwGCRM9Ky7gwCQYF
Kw4DAhoFAKCCAmswGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcN
MTQwMTI0MDExMjMzWjAjBgkqhkiG9w0BCQQxFgQUviBYk3TnkXVVuoovf1bKNp86z5UwbAYJ
KoZIhvcNAQkPMV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4G
CCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCB
zAYJKwYBBAGCNxAEMYG+MIG7MIGmMQswCQYDVQQGEwJVUzEdMBsGA1UEChMUU3ltYW50ZWMg
Q29ycG9yYXRpb24xHzAdBgNVBAsTFlN5bWFudGVjIFRydXN0IE5ldHdvcmsxHjAcBgNVBAsT
FVBlcnNvbmEgTm90IFZhbGlkYXRlZDE3MDUGA1UEAxMuU3ltYW50ZWMgQ2xhc3MgMSBJbmRp
dmlkdWFsIFN1YnNjcmliZXIgQ0EgLSBHNAIQOaBRF2hvPF7HAYJEz0rLuDCBzgYLKoZIhvcN
AQkQAgsxgb6ggbswgaYxCzAJBgNVBAYTAlVTMR0wGwYDVQQKExRTeW1hbnRlYyBDb3Jwb3Jh
dGlvbjEfMB0GA1UECxMWU3ltYW50ZWMgVHJ1c3QgTmV0d29yazEeMBwGA1UECxMVUGVyc29u
YSBOb3QgVmFsaWRhdGVkMTcwNQYDVQQDEy5TeW1hbnRlYyBDbGFzcyAxIEluZGl2aWR1YWwg
U3Vic2NyaWJlciBDQSAtIEc0AhA5oFEXaG88XscBgkTPSsu4MA0GCSqGSIb3DQEBAQUABIIB
AFBM6KECCDN1zVLuuPb1SdcODYjxHlonbi2AUmPcDXNKVzqC93dObgtD6DcPpGBb9nKjI2TI
rtA+6l3IykViRdeKiw/BcyREKOurHqzOBCPoXNZuNpy5OEUX4iBN/GHLZUIzsS8i9SqbyP59
R3I74OSl80AJi+Kl3Xy8ircrQXLYiCoTNY7P8ypwX0N4+M6ZSe/dDP66QEEgYZ7bbKRiLV4D
TAnhuFRIAgouMXCL5gUbSuCGmr5bv0Pd9EXBSsSA266Hya4/NEk/cfAw684tNR/I/qRP2EFD
LrZ8P3sAZ3H97g0FpiDmLaxMJGqCZBE63mw+x//N2Sxs1VYxglmLOMgAAAAAAAA=
--------------ms010904080006090905020204--