[OpenAFS] BreakDelayedCallbacks FAILED still an issue

[Jeffrey Altman]

This is a cryptographically signed message in MIME format.

--------------ms090000010409020508010305
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Christopher D. Clausen wrote:

> Client (flexo.acm.uiuc.edu) is Mac OS X 10.3 running the 1.4.1 binary 
> release from openafs.org (previous time this happened it was 1.4.1-rc8 
> from openafs.org.)
> 
> Server (alnitak.acm.uiuc.edu) is Solaris 10 SPARC running 1.4.1-rc10 
> (previous time it was running 1.4.1-rc8, I think) that I compiled from 
> source.
> 
> The client has a hardcoded IP of: 128.174.251.23 that is on the same 
> non-firewalled subnet as the server.  The server apparently thinks that 
> the client has changed IPs (69.112.249.245) probes to find it, can't, 
> and the client marks the server down and makes all volumes on that 
> server inaccessible.
> 
> Restarting the client had no effect.  I had to restart the fs process on 
> the server to remove the error condition.
> 
> Anyone else seen this happen?  Or have a better solution than restarting 
> the fs process if it happens again?  FileLog is below:
> 
> Thu Apr 27 13:59:13 2006 MultiProbe failed to find new address for host 
> 69.112.249.245:7001
> Thu Apr 27 13:59:20 2006 CB: Call back connect back failed (in break 
> delayed) for Host 69.112.249.245:7001
> Thu Apr 27 13:59:20 2006 BreakDelayedCallbacks FAILED for host 
> 69.112.249.245:7001 which IS UP.  Connection from 128.174.251.23:7001. 
> Possible network or routing failure.
> Thu Apr 27 13:59:20 2006 MultiProbe failed to find new address for host 
> 69.112.249.245:7001
> Thu Apr 27 14:02:20 2006 CB: Call back connect back failed (in break 
> delayed) for Host 69.112.249.245:7001
> Thu Apr 27 14:02:20 2006 BreakDelayedCallbacks FAILED for host 
> 69.112.249.245:7001 which IS UP.  Connection from 128.174.251.23:7001. 
> Possible network or routing failure.
> Thu Apr 27 14:02:20 2006 MultiProbe failed to find new address for host 
> 69.112.249.245:7001
> Thu Apr 27 14:06:56 2006 CB: WhoAreYou failed for 69.112.249.245:7001, 
> error -01
> Thu Apr 27 14:07:03 2006 CB: Call back connect back failed (in break 
> delayed) for Host 69.112.249.245:7001
> Thu Apr 27 14:07:03 2006 BreakDelayedCallbacks FAILED for host 
> 69.112.249.245:7001 which IS UP.  Connection from 128.174.251.23:7001. 
> Possible network or routing failure.
> Thu Apr 27 14:07:03 2006 MultiProbe failed to find new address for host 
> 69.112.249.245:7001
> Thu Apr 27 14:09:05 2006 CB: WhoAreYou failed for 69.112.249.245:7001, 
> error -01
> Thu Apr 27 14:09:12 2006 CB: Call back connect back failed (in break 
> delayed) for Host 69.112.249.245:7001
> Thu Apr 27 14:09:12 2006 BreakDelayedCallbacks FAILED for host 
> 69.112.249.245:7001 which IS UP.  Connection from 128.174.251.23:7001. 
> Possible network or routing failure.
> 
> And yes, I am in the process of upgrading to the 1.4.1 release right now 
> on our servers.
> 
> <<CDC

Please bump the server log level to 125 for a while so we can get more
detail.  I am very interested in the fact that the file server continues
to report the same problem after a client restart.   To the best of my
knowledge the MacOSX client does not preserve the UUID between restarts.
Therefore, the old UUID and the new should be treated as separate
clients.  The only reason that a client on two addr:ports should be
considered the same is if they have the same UUID.

Jeffrey Altman

--------------ms090000010409020508010305
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJXzCC
AwowggJzoAMCAQICAw7NrTANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJaQTElMCMGA1UE
ChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNv
bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDUwNTI3MTc0NzU3WhcNMDYwNTI3MTc0NzU3
WjBzMQ8wDQYDVQQEEwZBbHRtYW4xFTATBgNVBCoTDEplZmZyZXkgRXJpYzEcMBoGA1UEAxMT
SmVmZnJleSBFcmljIEFsdG1hbjErMCkGCSqGSIb3DQEJARYcamFsdG1hbkBzZWN1cmUtZW5k
cG9pbnRzLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKjPyrF+rdjOUSK/
bWwZHdx5p1+y6iiCd4vvYEVDxouYFp5C/fZEWm5n45ubBUbMSUI1MAZN6ooEoH09UTj6BXhM
S8B987ls81dKOIUphTF2jOzq8gsFmeA15yHMRAD20LqUWeLyvYk8FCNQw+dsKMMhX+WdsxOm
RY/1jPkJL6oN8kEwoUFkOX9/OfWWh6oFnV6faiEHUKDMFubsb9X0KVD8iIeR7Cxz7i4kXqRX
wMlp2fyoxcDIJrBaTY8nA++g3p34IkWt1a5po6g683nIgSnGpwYIwuJheBqSEZfLYWa+1KdD
6Sn27Ud94GqUvPVG5jC6zVC5EJ2aWuoAu+nNuV8CAwEAAaM5MDcwJwYDVR0RBCAwHoEcamFs
dG1hbkBzZWN1cmUtZW5kcG9pbnRzLmNvbTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBAUA
A4GBADtvO//tjiAV6VJGtoNtrl34mB5jGyGTiotzw8riB6zz0GvY11bcWDmp6JKif+pVG+8L
IySDosbuva13qu2HwYUxBmWc7CoNd2k9kRlcrfbDUTTrGOZK8qyqNqT3gQZTAa9ZnUI0su9G
y/n2o5bQcaYdqR3htNrpvdLSPOWhILOXMIIDCjCCAnOgAwIBAgIDDs2tMA0GCSqGSIb3DQEB
BAUAMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBM
dGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQTAeFw0w
NTA1MjcxNzQ3NTdaFw0wNjA1MjcxNzQ3NTdaMHMxDzANBgNVBAQTBkFsdG1hbjEVMBMGA1UE
KhMMSmVmZnJleSBFcmljMRwwGgYDVQQDExNKZWZmcmV5IEVyaWMgQWx0bWFuMSswKQYJKoZI
hvcNAQkBFhxqYWx0bWFuQHNlY3VyZS1lbmRwb2ludHMuY29tMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAqM/KsX6t2M5RIr9tbBkd3HmnX7LqKIJ3i+9gRUPGi5gWnkL99kRa
bmfjm5sFRsxJQjUwBk3qigSgfT1ROPoFeExLwH3zuWzzV0o4hSmFMXaM7OryCwWZ4DXnIcxE
APbQupRZ4vK9iTwUI1DD52wowyFf5Z2zE6ZFj/WM+Qkvqg3yQTChQWQ5f3859ZaHqgWdXp9q
IQdQoMwW5uxv1fQpUPyIh5HsLHPuLiRepFfAyWnZ/KjFwMgmsFpNjycD76DenfgiRa3Vrmmj
qDrzeciBKcanBgjC4mF4GpIRl8thZr7Up0PpKfbtR33gapS89UbmMLrNULkQnZpa6gC76c25
XwIDAQABozkwNzAnBgNVHREEIDAegRxqYWx0bWFuQHNlY3VyZS1lbmRwb2ludHMuY29tMAwG
A1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEEBQADgYEAO287/+2OIBXpUka2g22uXfiYHmMbIZOK
i3PDyuIHrPPQa9jXVtxYOanokqJ/6lUb7wsjJIOixu69rXeq7YfBhTEGZZzsKg13aT2RGVyt
9sNRNOsY5kryrKo2pPeBBlMBr1mdQjSy70bL+fajltBxph2pHeG02um90tI85aEgs5cwggM/
MIICqKADAgECAgENMA0GCSqGSIb3DQEBBQUAMIHRMQswCQYDVQQGEwJaQTEVMBMGA1UECBMM
V2VzdGVybiBDYXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xGjAYBgNVBAoTEVRoYXd0ZSBDb25z
dWx0aW5nMSgwJgYDVQQLEx9DZXJ0aWZpY2F0aW9uIFNlcnZpY2VzIERpdmlzaW9uMSQwIgYD
VQQDExtUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgQ0ExKzApBgkqhkiG9w0BCQEWHHBlcnNv
bmFsLWZyZWVtYWlsQHRoYXd0ZS5jb20wHhcNMDMwNzE3MDAwMDAwWhcNMTMwNzE2MjM1OTU5
WjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRk
LjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwgZ8wDQYJ
KoZIhvcNAQEBBQADgY0AMIGJAoGBAMSmPFVzVftOucqZWh5owHUEcJ3f6f+jHuy9zfVb8hp2
vX8MOmHyv1HOAdTlUAow1wJjWiyJFXCO3cnwK4Vaqj9xVsuvPAsH5/EfkTYkKhPPK9Xzgnc9
A74r/rsYPge/QIACZNenprufZdHFKlSFD0gEf6e20TxhBEAeZBlyYLf7AgMBAAGjgZQwgZEw
EgYDVR0TAQH/BAgwBgEB/wIBADBDBgNVHR8EPDA6MDigNqA0hjJodHRwOi8vY3JsLnRoYXd0
ZS5jb20vVGhhd3RlUGVyc29uYWxGcmVlbWFpbENBLmNybDALBgNVHQ8EBAMCAQYwKQYDVR0R
BCIwIKQeMBwxGjAYBgNVBAMTEVByaXZhdGVMYWJlbDItMTM4MA0GCSqGSIb3DQEBBQUAA4GB
AEiM0VCD6gsuzA2jZqxnD3+vrL7CF6FDlpSdf0whuPg2H6otnzYvwPQcUCCTcDz9reFhYsPZ
Ohl+hLGZGwDFGguCdJ4lUJRix9sncVcljd2pnDmOjCBPZV+V2vf3h9bGCE6u9uo05RAaWzVN
d+NWIXiC3CEZNd4ksdMdRv9dX2VPMYIDOzCCAzcCAQEwaTBiMQswCQYDVQQGEwJaQTElMCMG
A1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBl
cnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECAw7NrTAJBgUrDgMCGgUAoIIBpzAYBgkqhkiG
9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0wNjA0MjcyMDAxMTBaMCMGCSqG
SIb3DQEJBDEWBBRUZV00bvNwZ0g7qq0ocbo2arlNjTBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqG
SIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG
9w0DAgIBKDB4BgkrBgEEAYI3EAQxazBpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3
dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJl
ZW1haWwgSXNzdWluZyBDQQIDDs2tMHoGCyqGSIb3DQEJEAILMWugaTBiMQswCQYDVQQGEwJa
QTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhh
d3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECAw7NrTANBgkqhkiG9w0BAQEFAASC
AQAE0KsWDoDfQ5iEIVYX1PPbFlxIV3Q+1M0T9+oXVu6uICoeLmJIKCxne5Wn0Vf319v3ldxZ
MCOnXz/rsizAypT/J16A1G+oHZ6T2cMfDwz7xxnKSHWish2VU5yBlXROmHsAbxFVJB5o1Z+O
hDbS/KKccGDr4y7qqb7oLucu2vXiMKBDuYXYVFRlbv+sZz1H3vCmkFtaeYdYM/OmPt2kZXT3
UJq/76HJOF9dZy81MTKc/CezcRFDDuF43QyNTfWF9OtVHiSk38xafuvxVyq+L7gRDUp1SazH
9OVhueUFS8+WmskW1Tuhc6LFN+XjBAiARuoJQibTyYRk0mxWeFPQA81gAAAAAAAA
--------------ms090000010409020508010305--