[OpenAFS] fs: server not responding promptly

Thu, 10 Feb 2011 09:55:02 +0100

--Apple-Mail-22--954161035
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=iso-8859-1

Just having had my nose pinned down to that area for the past weeks I'd =
risk a quick diagnosis: if your clients experience long delays and even =
timeouts on writes (and you could check whether this also applies to =
mkdir, rmdir, rm, chmod, etc..), then callback breaking is the main =
suspect as those are synchronous to operations altering the state of =
files.

Your FileLog shows ports other than 7001 or 4711, a strong hint that =
those are Nat gateways, perhaps simply virtual machines with non-static =
assignments and short UDP port mapping intervals.

On the client side you can help by cranking up UDP port mapping timeouts =
from a few minutes to 1-2 hour, easy in vmware for example, in personal =
NAT routers you'd try to assign static mappings.

Unfortunately, AFS/RX agnostic firewalls and NAT gateways are hard to =
deal with on the server side. The server can protect itself to a certain =
degree, yesterday I submitted the second in a series of patches =
addressing parts of the problem, but it remains an effort to keep =
adverse affects down to a bearable level. RX over TCP would solve many =
problems in this area - if only it got a little more attention.

Le 10 f=E9vr. 2011 =E0 05:44, John Tang Boyland a =E9crit :

> Since the start of the semester, OpenAFS seems to occasionally hang
> for a few seconds (5? 10?) when trying to do things like write files.
> I finally had it happen while running a script that was doing fs =
calls,
> and got the message:
> fs:'path-to-directory-in-afs': server not responding promptly
>=20
> The FileLog for the server (jeremiah.cs.uwm.edu) from the appropriate =
time has:
> ...
> Wed Feb  9 22:14:06 2011 CB: ProbeUuid for 999.102.202.55:2841 failed =
-01
> Wed Feb  9 22:15:07 2011 CheckHost_r: Probing all interfaces of host =
999.35.48.249:56648 failed, code -01
> Wed Feb  9 22:15:09 2011 CB: ProbeUuid for 999.131.13.134:7001 failed =
-01
> Wed Feb  9 22:16:04 2011 CB: ProbeUuid for 999.59.5.145:63713 failed =
-01
> Wed Feb  9 22:16:05 2011 CB: WhoAreYou failed for host gge5870 =
(999.30.179.54:7001), error -01
> Wed Feb  9 22:16:12 2011 CB: ProbeUuid for 999.100.203.66:53467 failed =
-01
> Wed Feb  9 22:16:36 2011 CB: ProbeUuid for 999.229.195.248:49341 =
failed -01
> Wed Feb  9 22:18:12 2011 CB: ProbeUuid for 999.102.202.55:2846 failed =
-01
> Wed Feb  9 22:19:14 2011 CB: ProbeUuid for 999.131.13.134:12627 failed =
-01
>=20
> (I have obscured the first byte in each network id.)
>=20
> % rxdebug jeremiah.cs.uwm.edu -version
> Trying 129.89.143.70 (port 7000):
> AFS version:  OpenAFS 1.4.12 built  2010-03-09=20
> % fs --version
> openafs 1.4.3
> % rxdebug localhost -version -port 7001
> Trying 127.0.0.1 (port 7001):
> AFS version:  OpenAFS 1.4.11 built  2009-07-13=20
>=20
> People notice the delays on Windows machines, MacOSX and on Solaris.
> (The machine I caught it on above was solaris 10.)
>=20
> On MacOSX and Windows, the delays are particularly disturbing
> because they are long enough for the OS to time out
> and give the application an IO error.  This causes the application
> to say the files aren't there anymore, which is highly disturbing
> to my students.
>=20
> I'm running the file server with all default values.
> Perhaps I need to tune the number of daemons?
>=20
> My original guess is that the server is hanging while waiting to break =
callbacks
> from clients that are behind firewalls and not responding.  But even
> running 'fs checks' from all possible clients that are accessing the
> volume doesn't seem to work; at least it still takes a few seconds =
more.
> But this sort of behavior presumably would drive everyone mad and =
would
> have been fixed before 1.4.12, so now I'm at a loss.
>=20
> Suggestions always appreciated,
> John
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info

--Apple-Mail-22--954161035
Content-Disposition: attachment; filename="smime.p7s"
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIOhDCCBpow
ggWCoAMCAQICAwHkyDANBgkqhkiG9w0BAQUFADCBjDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0
YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcx
ODA2BgNVBAMTL1N0YXJ0Q29tIENsYXNzIDEgUHJpbWFyeSBJbnRlcm1lZGlhdGUgQ2xpZW50IENB
MB4XDTEwMTIwMTEwMjYzMloXDTExMTIwMTIxNDUyMlowgZExIDAeBgNVBA0TFzMwNTcwMi0yN0Fm
NnRkZTNKM2pmRER4MR4wHAYDVQQKExVQZXJzb25hIE5vdCBWYWxpZGF0ZWQxKTAnBgNVBAMTIFN0
YXJ0Q29tIEZyZWUgQ2VydGlmaWNhdGUgTWVtYmVyMSIwIAYJKoZIhvcNAQkBFhNydGJAcGNsZWxs
YS5jZXJuLmNoMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApgLpZzpWFRB3nxXEL8hu
JajBSVLV7Uqi352zKPa3bd2qyu+fLefEWDp1jmTHQiR+qZ4VyXkfMuMzNzqQm1JndnHGAA2HVQ34
WDnvzeV1+DIoTY9cz7PWqEqv9xySJDeQ03iv8fs+XuVFncfWGxa3J9tFkWOLmH1VJpeiJ5OR+kG2
JamJUGksdOW5lGPNtBSc+LSkoCk+ZLetHkcrVR6fr6OB7FeE13RmJqocX7RyzWWtS1Z8NFFQ9wwU
4qyzTxseStVWz7LZ8SdjCu706gOMdjcJR2Wg6n+7byN6bblvQ34WvfwjhkbMRRfT72VX0q8aq37d
bIxYMySKUqCkJX0wjQIDAQABo4IC/DCCAvgwCQYDVR0TBAIwADALBgNVHQ8EBAMCBLAwHQYDVR0l
BBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBRScYhfw3malpCAoaTd74V8lnQUZzAf
BgNVHSMEGDAWgBRTcu2SnODaywFcfH6WNU7y1LhRgjAeBgNVHREEFzAVgRNydGJAcGNsZWxsYS5j
ZXJuLmNoMIIBQgYDVR0gBIIBOTCCATUwggExBgsrBgEEAYG1NwECAjCCASAwLgYIKwYBBQUHAgEW
Imh0dHA6Ly93d3cuc3RhcnRzc2wuY29tL3BvbGljeS5wZGYwNAYIKwYBBQUHAgEWKGh0dHA6Ly93
d3cuc3RhcnRzc2wuY29tL2ludGVybWVkaWF0ZS5wZGYwgbcGCCsGAQUFBwICMIGqMBQWDVN0YXJ0
Q29tIEx0ZC4wAwIBARqBkUxpbWl0ZWQgTGlhYmlsaXR5LCBzZWUgc2VjdGlvbiAqTGVnYWwgTGlt
aXRhdGlvbnMqIG9mIHRoZSBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSBQb2xpY3kg
YXZhaWxhYmxlIGF0IGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tL3BvbGljeS5wZGYwYwYDVR0fBFww
WjAroCmgJ4YlaHR0cDovL3d3dy5zdGFydHNzbC5jb20vY3J0dTEtY3JsLmNybDAroCmgJ4YlaHR0
cDovL2NybC5zdGFydHNzbC5jb20vY3J0dTEtY3JsLmNybDCBjgYIKwYBBQUHAQEEgYEwfzA5Bggr
BgEFBQcwAYYtaHR0cDovL29jc3Auc3RhcnRzc2wuY29tL3N1Yi9jbGFzczEvY2xpZW50L2NhMEIG
CCsGAQUFBzAChjZodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS9jZXJ0cy9zdWIuY2xhc3MxLmNsaWVu
dC5jYS5jcnQwIwYDVR0SBBwwGoYYaHR0cDovL3d3dy5zdGFydHNzbC5jb20vMA0GCSqGSIb3DQEB
BQUAA4IBAQAS50w6igdEXU8/vRQzoEQPSgKso+0KKIS8rnJIScWUU47/JQuawpdP4odAfMYe4583
u/iMZkAvkloPK98zX16XdtuADShjEnzyvrSx+DocXHai8I7LFPxQVVhkPskOZJRhipF9vmP02TQe
+AWz9O9341DrikFe8Cwv+edYBtz0jGbYJkWPUd+V3qGDA5+SuqjoW40lPFvJ6fhi+yzkTBhO1HVI
fl4u/SPNS0spaqWfwMlDQjT06PcJ1Ox0how6kjGyGN3Gdkdo/quXyGj7/7aTC8IF8dPAx6xn0jqF
jewzMYoiSX/1bBy49mTKbYGyfmelviiRnOc02kjpslsLWbcXMIIH4jCCBcqgAwIBAgIBDTANBgkq
hkiG9w0BAQUFADB9MQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UE
CxMiU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzEpMCcGA1UEAxMgU3RhcnRDb20g
Q2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMDcxMDI0MjEwMTU0WhcNMTIxMDIyMjEwMTU0WjCB
jDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBE
aWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxODA2BgNVBAMTL1N0YXJ0Q29tIENsYXNzIDEgUHJp
bWFyeSBJbnRlcm1lZGlhdGUgQ2xpZW50IENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAxwmDzM4t2BqxKaQuE6uWvooyg4ymiEGWVUet1G8SD+rqvyNH4QrvnEIaFHxOhESip7vMz39S
cLpNLbL1QpOlPW/tFIzNHS3qd2XRNYG5Sv9RcGE+T4qbLtsjjJbi6sL7Ls/f/X9ftTyhxvxWkf8K
W37iKrueKsxw2HqolH7GM6FX5UfNAwAu4ZifkpmZzU1slBhyWwaQPEPPZRsWoTb7q8hmgv6Nv3Hg
9rmA1/VPBIOQ6SKRkHXG0Hhmq1dOFoAFI411+a/9nWm5rcVjGcIWZ2v/43Yksq60jExipA4l5uv9
/+Hm33mbgmCszdj/Dthf13tgAv2O83hLJ0exTqfrlwIDAQABo4IDWzCCA1cwDAYDVR0TBAUwAwEB
/zALBgNVHQ8EBAMCAaYwHQYDVR0OBBYEFFNy7ZKc4NrLAVx8fpY1TvLUuFGCMIGoBgNVHSMEgaAw
gZ2AFE4L7xqkQFulF2mHMMo0aEPQQa7yoYGBpH8wfTELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0
YXJ0Q29tIEx0ZC4xKzApBgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcx
KTAnBgNVBAMTIFN0YXJ0Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5ggEBMAkGA1UdEgQCMAAw
PQYIKwYBBQUHAQEEMTAvMC0GCCsGAQUFBzAChiFodHRwOi8vd3d3LnN0YXJ0c3NsLmNvbS9zZnNj
YS5jcnQwYAYDVR0fBFkwVzAsoCqgKIYmaHR0cDovL2NlcnQuc3RhcnRjb20ub3JnL3Nmc2NhLWNy
bC5jcmwwJ6AloCOGIWh0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL3Nmc2NhLmNybDCCAV0GA1UdIASC
AVQwggFQMIIBTAYLKwYBBAGBtTcBAQQwggE7MC8GCCsGAQUFBwIBFiNodHRwOi8vY2VydC5zdGFy
dGNvbS5vcmcvcG9saWN5LnBkZjA1BggrBgEFBQcCARYpaHR0cDovL2NlcnQuc3RhcnRjb20ub3Jn
L2ludGVybWVkaWF0ZS5wZGYwgdAGCCsGAQUFBwICMIHDMCcWIFN0YXJ0IENvbW1lcmNpYWwgKFN0
YXJ0Q29tKSBMdGQuMAMCAQEagZdMaW1pdGVkIExpYWJpbGl0eSwgcmVhZCB0aGUgc2VjdGlvbiAq
TGVnYWwgTGltaXRhdGlvbnMqIG9mIHRoZSBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0
eSBQb2xpY3kgYXZhaWxhYmxlIGF0IGh0dHA6Ly9jZXJ0LnN0YXJ0Y29tLm9yZy9wb2xpY3kucGRm
MBEGCWCGSAGG+EIBAQQEAwIABzBQBglghkgBhvhCAQ0EQxZBU3RhcnRDb20gQ2xhc3MgMSBQcmlt
YXJ5IEludGVybWVkaWF0ZSBGcmVlIFNTTCBFbWFpbCBDZXJ0aWZpY2F0ZXMwDQYJKoZIhvcNAQEF
BQADggIBAKqa4eBbjM4dG/wdxiwwIKC3kyb98QK2zREovyn/xzDP/4H/Bc8FFDTgoJR+nX2Li0EP
3U7TsjG+CaIi90+8YlShADpkPrfm/8SzjGtJtfM6EaluJOhpcqMr3OyzK3aYGJP5RIeZ6vLT3fQa
DZsIooXl6YSFR/0HpU4FJDc0wuyFaZmFbCrjTp8RNYyRWTTX6mWSv+TraOwuj3zrrddSpgUEi2Wq
wM9G/5o4IXQbGHx7oXTvL6zrw9IOYO3QOKZDgFNhHeKUgqMAUiLcg/+WhcGe+Y4umKuxghtwaYsg
D/bLfIfop3NC/u5JqwDCWizAJruhmbOV4LG859MFCb2w/YeY55zDPVGmQ3MZdriwdOKrhlFjOjYi
hmm28UHOvND2G3kK0LvnuieLqjQMc6GuUcZAQOWv96pW4BfbiQXpAqibMMeb0PZISa7PFEzGiBc2
xAuVRkM4kB9/+iieA1D/OTiRJwsf6rkoVgOsN9fCw522tzOmuVfiqDS4bFYv00sX/dFGwasHUUf3
DsLhpDSYdejb74SKjtuqLDIOuAm2bA1axA6+7kjFeNIngSU6OPSMre+xAjoc/6coaMGthFD+mimr
/i/8F8wDwdyzas7oxkdCtaW8hVir8mJnbp4CbckllDMPkeQ6qQNmxSDhOeqX1jyx2cTi/vPq+/Ty
xV/stlehMYIDbzCCA2sCAQEwgZQwgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBM
dGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQD
Ey9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQQIDAeTIMAkG
BSsOAwIaBQCgggGvMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEx
MDIxMDA4NTUwMlowIwYJKoZIhvcNAQkEMRYEFHi7hoB7uMiyA0YQ72zqblJ0zTwTMIGlBgkrBgEE
AYI3EAQxgZcwgZQwgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYD
VQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENv
bSBDbGFzcyAxIFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQQIDAeTIMIGnBgsqhkiG9w0B
CRACCzGBl6CBlDCBjDELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzApBgNV
BAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxODA2BgNVBAMTL1N0YXJ0Q29t
IENsYXNzIDEgUHJpbWFyeSBJbnRlcm1lZGlhdGUgQ2xpZW50IENBAgMB5MgwDQYJKoZIhvcNAQEB
BQAEggEAEa2Wl1HW6L20n3LZcuBI8I3glfDsdbKAqU8RsEcwayytq1InD1k0yNP/9E8cc7xE7rAP
1DI+291r+urr3rORkztoCMhhg8nXIiwz1fFYNd4fVCUeFieh89NM6ft2ziSakoRfqzKVYNTD41Cv
MCEXqfbnne7B2IAvAXFKbWxWXTaOKn1i2C7DE1bQJHUDdDuulwmfk13rgK0Bi2yIHCOahW+EZl1X
Qd5JQASBPASOdgk+77QvsPx/ziJ9GGIxQ1LsoBEuUTlFwa/V24r4YR08GpFR0imCBBahYcz189pR
g/pmpz+j1gbk+5lozgNAMihzxHXp9msI/28z9q8vEEgTcQAAAAAAAA==

--Apple-Mail-22--954161035--