[OpenAFS-devel] 1.3.X: Problem with many connections exhausting resources?

[Jeffrey Altman]

This is a cryptographically signed message in MIME format.

--------------ms080801030604030402040500
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Harald Barth wrote:
>>Harald:
>>
>>I wonder if this condition is a side effect of the bug that has been
>>fixed by
>>
>>   DELTA STABLE12-rx-makecall-race-fix-20050518
> 
> 
> Hi, thanks, but I have a feling that it is older because I have the
> same behaviour against a server with 1.3.77
> 
> 1.3.77 has rx.c 1.58.2.4 and rx-makecall-race-fix-20050518 was introduced
> with 1.58.2.18.

DELTA STABLE12-rx-makecall-race-fix-20050518 fixes a bug that has
existed since the beginning of time.

>>With this bug the rx_connection objects on one side can be destroyed
>>while they are in use.   New rx_connection objects would then need to
>>be established.
> 
> 
> I'm still fumbling in the rx-dark. Anyone got the bestseller "the life
> and death of an rx connection"? Is there a way to get rid of these
> zillions of connections? Am I the only one seeing this?
> 
> And I still don't know if this is related to the sudden "server
> down" problems I have to start with.
> 
> Harald.

Last October/November we fixed a large number of bugs related to
rx connection objects being mismanaged due to an inability of
applications which use rx to reference count the rx_connection
objects.   This meant that although internally the rx library was
thread safe, a single rx_connection object could not safely be used
by multiple threads in the application.   In 1.3.72 we added the
ability to reference count the rx_connection objects and in turn
removed the premature destructon of the objects while in use that
we had been seeing.

We have also seen huge number of connections being produced by the
Windows clients.  The Windows clients were creating connections,
using them once, and then destroying them.   Most of these Windows
client bugs were fixed in 1.3.72 and one more was fixed in 1.3.80.

Having multiple connections from a client is not necessarily a problem.
If the connection properties are different, there will need to be
separate connections.   It is also possible that you will have
connections in the hash table that are marked as DESTROYED that have
not been cleaned up yet.   I'm not sure that your script produces the
results you are expecting.

Do you have any other information you can provide on your server down
problem?

Jeffrey Altman


--------------ms080801030604030402040500
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJPzCC
AvowggJjoAMCAQICAw7NrDANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJaQTElMCMGA1UE
ChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNv
bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDUwNTI3MTc0MjQzWhcNMDYwNTI3MTc0MjQz
WjBrMQ8wDQYDVQQEEwZBbHRtYW4xFTATBgNVBCoTDEplZmZyZXkgRXJpYzEcMBoGA1UEAxMT
SmVmZnJleSBFcmljIEFsdG1hbjEjMCEGCSqGSIb3DQEJARYUamFsdG1hbkBjb2x1bWJpYS5l
ZHUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+LutDu/YyHreNfoYd+ZtOjXsL
h67F2cmcVuBPBz+ZGDA+WpVEHrqXaZZO8acXBR5uAVfiwA1acE/kvD/CN5kAqx1VJuQ8Pvyk
iGHhUYTd27ZTliBIrptC7C/381gVwkS+a8jQFPJPO+OktZDzAYplGRY/MQCV8dIsvXUjucox
7TwTTdoLAJYRvHtfEcaCc6mO4ph6NeXQw8Grlx3IRAlTrkE5fBGyjH6R4fqnFTXRQAh1/bG+
i8hQvE6mud3mXdL2t7NP1Qxd9wW0/F/pnWY12IFP/luc3zEzIPvAe+nJluLuSEj0LZgP16mF
xBj1p+u9HPWcHRVX6q7+MQ0RWOv1AgMBAAGjMTAvMB8GA1UdEQQYMBaBFGphbHRtYW5AY29s
dW1iaWEuZWR1MAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEEBQADgYEAUDUuzxiq8bbI8vq2
swRK513RphZp+fepyKU5mwBI6aF4GcmqITQILtfTG2SXnjSeY99d+bjOdK1DJFvVh9aOy8mh
2NbEnqMnJIZtg5+eEU64DIV5bQdDRpi99H9vA0sRATIquut+3YHba+zArj0VkVof2VI+ToBu
sHdtSrZYo0gwggL6MIICY6ADAgECAgMOzawwDQYJKoZIhvcNAQEEBQAwYjELMAkGA1UEBhMC
WkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1Ro
YXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA1MDUyNzE3NDI0M1oXDTA2
MDUyNzE3NDI0M1owazEPMA0GA1UEBBMGQWx0bWFuMRUwEwYDVQQqEwxKZWZmcmV5IEVyaWMx
HDAaBgNVBAMTE0plZmZyZXkgRXJpYyBBbHRtYW4xIzAhBgkqhkiG9w0BCQEWFGphbHRtYW5A
Y29sdW1iaWEuZWR1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvi7rQ7v2Mh63
jX6GHfmbTo17C4euxdnJnFbgTwc/mRgwPlqVRB66l2mWTvGnFwUebgFX4sANWnBP5Lw/wjeZ
AKsdVSbkPD78pIhh4VGE3du2U5YgSK6bQuwv9/NYFcJEvmvI0BTyTzvjpLWQ8wGKZRkWPzEA
lfHSLL11I7nKMe08E03aCwCWEbx7XxHGgnOpjuKYejXl0MPBq5cdyEQJU65BOXwRsox+keH6
pxU10UAIdf2xvovIULxOprnd5l3S9rezT9UMXfcFtPxf6Z1mNdiBT/5bnN8xMyD7wHvpyZbi
7khI9C2YD9ephcQY9afrvRz1nB0VV+qu/jENEVjr9QIDAQABozEwLzAfBgNVHREEGDAWgRRq
YWx0bWFuQGNvbHVtYmlhLmVkdTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBAUAA4GBAFA1
Ls8YqvG2yPL6trMESudd0aYWafn3qcilOZsASOmheBnJqiE0CC7X0xtkl540nmPfXfm4znSt
QyRb1YfWjsvJodjWxJ6jJySGbYOfnhFOuAyFeW0HQ0aYvfR/bwNLEQEyKrrrft2B22vswK49
FZFaH9lSPk6AbrB3bUq2WKNIMIIDPzCCAqigAwIBAgIBDTANBgkqhkiG9w0BAQUFADCB0TEL
MAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3du
MRowGAYDVQQKExFUaGF3dGUgQ29uc3VsdGluZzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBT
ZXJ2aWNlcyBEaXZpc2lvbjEkMCIGA1UEAxMbVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIENB
MSswKQYJKoZIhvcNAQkBFhxwZXJzb25hbC1mcmVlbWFpbEB0aGF3dGUuY29tMB4XDTAzMDcx
NzAwMDAwMFoXDTEzMDcxNjIzNTk1OVowYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0
ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVl
bWFpbCBJc3N1aW5nIENBMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDEpjxVc1X7TrnK
mVoeaMB1BHCd3+n/ox7svc31W/Iadr1/DDph8r9RzgHU5VAKMNcCY1osiRVwjt3J8CuFWqo/
cVbLrzwLB+fxH5E2JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTXp6a7n2XRxSpUhQ9IBH+nttE8
YQRAHmQZcmC3+wIDAQABo4GUMIGRMBIGA1UdEwEB/wQIMAYBAf8CAQAwQwYDVR0fBDwwOjA4
oDagNIYyaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVBlcnNvbmFsRnJlZW1haWxDQS5j
cmwwCwYDVR0PBAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYDVQQDExFQcml2YXRlTGFiZWwy
LTEzODANBgkqhkiG9w0BAQUFAAOBgQBIjNFQg+oLLswNo2asZw9/r6y+whehQ5aUnX9MIbj4
Nh+qLZ82L8D0HFAgk3A8/a3hYWLD2ToZfoSxmRsAxRoLgnSeJVCUYsfbJ3FXJY3dqZw5jowg
T2Vfldr394fWxghOrvbqNOUQGls1TXfjViF4gtwhGTXeJLHTHUb/XV9lTzGCAzswggM3AgEB
MGkwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0
ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBAgMOzaww
CQYFKw4DAhoFAKCCAacwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUx
DxcNMDUwNTMwMTAzOTQ0WjAjBgkqhkiG9w0BCQQxFgQUMfFA7uD/LOKmwcsI9uDP3NMH7SIw
UgYJKoZIhvcNAQkPMUUwQzAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcN
AwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgweAYJKwYBBAGCNxAEMWswaTBiMQswCQYD
VQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UE
AxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECAw7NrDB6BgsqhkiG9w0B
CRACCzFroGkwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQ
dHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENB
AgMOzawwDQYJKoZIhvcNAQEBBQAEggEAhwCJ5IQN6JVzkUAzEzUsAfKFpvrIFh5D1yH3Q4yG
UFMdrGl5dSiZcj2CIbyyr6cHICtyYm63ZJ+GQQF4ybAr9a3hqhxuCZjmnlgOtFq6i8MowmFE
CPJpNJcFQBfAZkoLRU/bbY2R5F7f2NE0/e6UxPJ0knBtGnanvBuz21luvPm9nnForggWEq3q
KQtyco8b78EXepBu0yxOcdml0ggc6x7ZfjIwJYrOYbLfmIhKqfj0+rFxA4L4KwRIlqkxJzb1
Fs5xxrLsspl2NTpUxIWfDmGicrrFBsoG9s5Wta5QDw/gNJH65kpvxqkud7WPBmSAnBROFIJH
ApCUrCqdSq9R/QAAAAAAAA==
--------------ms080801030604030402040500--