[OpenAFS-devel] kauth broken - problem with IOMGR in lwp?
Jeffrey Altman
jaltman@auristor.com
Wed, 13 Nov 2024 21:45:45 -0500
--Apple-Mail=_74022820-1F2B-4D05-B964-641182CF32E7
Content-Type: multipart/alternative;
boundary="Apple-Mail=_C4952796-AA36-4516-9C66-6ED2F2D17CD6"
--Apple-Mail=_C4952796-AA36-4516-9C66-6ED2F2D17CD6
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
> On Nov 13, 2024, at 8:02=E2=80=AFPM, Jeffrey Altman =
<jaltman@auristor.com> wrote:
>=20
>=20
>> On Nov 13, 2024, at 7:19=E2=80=AFPM, Ben Huntsman <ben@huntsmans.net> =
wrote:
>>=20
>> Any idea how to track it down? This is made more difficult by the =
limited debugger output.
>>=20
>=20
> git bisect from a known working version to a known broken version.
Git bisect is going to be the best option because there was five years =
of significant development between when the 1.6 series was cut from =
master and when the 1.8 series was cut from master. =20
I think the first question to answer is whether or not the failure case =
was present at the point where the openafs-stable-1_8_x branch forked =
from master. =20
commit e739eaa650ee30dcce54d05908b062839eafbf73 (tag: =
BP-openafs-stable-1_8_x)
If it was working there, then bisect along the openafs-stable_1_8_x =
branch to find the commit where things break for the first time.
If it was not working there, then bisect the master branch from the =
point where the openafs-stable-1_6_x branch forked. =20
commit c89bc695707f62e64f64b3abb15470b291f613a1 (tag: =
BP-openafs-stable-1_6_x)
LWP is hard to debug because it is a custom cooperative threading model =
which is not going to be understood by debuggers.
In OpenAFS 1.8.x the commonly used command line tools although built for =
both LWP and pthreads are going to be distributed as the pthreads =
version. There is no pthreads build of klog. It=E2=80=99s quite =
possible the breakage is to all LWP clients and not specific to kauth. =
One thought that comes to mind is whether the difference is in how =
OpenAFS is being built. For example, was the 1.6.13 build that =
succeeded 32-bit vs the 1.8 build being 64-bit?
--Apple-Mail=_C4952796-AA36-4516-9C66-6ED2F2D17CD6
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"content-type" content=3D"text/html; =
charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: =
after-white-space;"><br><div><blockquote type=3D"cite"><div>On Nov 13, =
2024, at 8:02=E2=80=AFPM, Jeffrey Altman <jaltman@auristor.com> =
wrote:</div><br class=3D"Apple-interchange-newline"><div><meta =
http-equiv=3D"content-type" content=3D"text/html; charset=3Dutf-8"><div =
style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; =
line-break: after-white-space;"><br><div><blockquote type=3D"cite"><div>On=
Nov 13, 2024, at 7:19=E2=80=AFPM, Ben Huntsman =
<ben@huntsmans.net> wrote:</div><div><div class=3D"elementToProof" =
style=3D"font-style: normal; font-variant-caps: normal; font-weight: =
400; letter-spacing: normal; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none; font-family: =
Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, =
sans-serif; font-size: 12pt;"><br></div><div class=3D"elementToProof" =
style=3D"font-style: normal; font-variant-caps: normal; font-weight: =
400; letter-spacing: normal; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none; font-family: =
Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, =
sans-serif; font-size: 12pt;"> Any idea how to track it =
down? This is made more difficult by the limited debugger =
output.</div><div class=3D"elementToProof" style=3D"font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, =
Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: =
12pt;"><br></div></div></blockquote><br></div><div>git bisect from a =
known working version to a known broken =
version.</div></div></div></blockquote><br></div><div>Git bisect is =
going to be the best option because there was five years of significant =
development between when the 1.6 series was cut from master and when the =
1.8 series was cut from master. </div><div><br></div><div>I =
think the first question to answer is whether or not the failure case =
was present at the point where the openafs-stable-1_8_x branch forked =
from master. </div><div><br></div><div> commit =
e739eaa650ee30dcce54d05908b062839eafbf73 (tag: =
BP-openafs-stable-1_8_x)</div><div><br></div><div>If it was working =
there, then bisect along the openafs-stable_1_8_x branch to find the =
commit where things break for the first time.</div><div>If it was not =
working there, then bisect the master branch from the point where the =
openafs-stable-1_6_x branch forked. =
</div><div><br></div><div> commit =
c89bc695707f62e64f64b3abb15470b291f613a1 (tag: =
BP-openafs-stable-1_6_x)</div><div><br></div><div>LWP is hard to debug =
because it is a custom cooperative threading model which is not going to =
be understood by debuggers.</div><div><br></div><div>In OpenAFS 1.8.x =
the commonly used command line tools although built for both LWP and =
pthreads are going to be distributed as the pthreads version. =
There is no pthreads build of klog. It=E2=80=99s quite =
possible the breakage is to all LWP clients and not specific to kauth. =
One thought that comes to mind is whether the difference is in how =
OpenAFS is being built. For example, was the 1.6.13 build that =
succeeded 32-bit vs the 1.8 build being =
64-bit?</div><div><br></div><div><br></div></body></html>=
--Apple-Mail=_C4952796-AA36-4516-9C66-6ED2F2D17CD6--
--Apple-Mail=_74022820-1F2B-4D05-B964-641182CF32E7
Content-Disposition: attachment;
filename=smime.p7s
Content-Type: application/pkcs7-signature;
name=smime.p7s
Content-Transfer-Encoding: base64
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCDHEw
ggXSMIIEuqADAgECAhBAAYJpmi/rPn/F0fJyDlzMMA0GCSqGSIb3DQEBCwUAMDoxCzAJBgNVBAYT
AlVTMRIwEAYDVQQKEwlJZGVuVHJ1c3QxFzAVBgNVBAMTDlRydXN0SUQgQ0EgQTEzMB4XDTIyMDgw
NDE2MDQ0OFoXDTI1MTAzMTE2MDM0OFowcDEvMC0GCgmSJomT8ixkAQETH0EwMTQxMEQwMDAwMDE4
MjY5OUEyRkQyMDAwMjMzQ0QxGTAXBgNVBAMTEEplZmZyZXkgRSBBbHRtYW4xFTATBgNVBAoTDEF1
cmlTdG9yIEluYzELMAkGA1UEBhMCVVMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCk
C7PKBBZnQqDKPtZPMLAy77zo2DPvwtGnd1hNjPvbXrpGxUb3xHZRtv179LHKAOcsY2jIctzieMxf
82OMyhpBziMPsFAG/ukihBMFj3/xEeZVso3K27pSAyyNfO/wJ0rX7G+ges22Dd7goZul8rPaTJBI
xbZDuaykJMGpNq4PQ8VPcnYZx+6b+nJwJJoJ46kIEEfNh3UKvB/vM0qtxS690iAdgmQIhTl+qfXq
4IxWB6b+3NeQxgR6KLU4P7v88/tvJTpxIKkg9xj89ruzeThyRFd2DSe3vfdnq9+g4qJSHRXyTft6
W3Lkp7UWTM4kMqOcc4VSRdufVKBQNXjGIcnhAgMBAAGjggKcMIICmDAOBgNVHQ8BAf8EBAMCBPAw
gYQGCCsGAQUFBwEBBHgwdjAwBggrBgEFBQcwAYYkaHR0cDovL2NvbW1lcmNpYWwub2NzcC5pZGVu
dHJ1c3QuY29tMEIGCCsGAQUFBzAChjZodHRwOi8vdmFsaWRhdGlvbi5pZGVudHJ1c3QuY29tL2Nl
cnRzL3RydXN0aWRjYWExMy5wN2MwHwYDVR0jBBgwFoAULbfeG1l+KpguzeHUG+PFEBJe6RQwCQYD
VR0TBAIwADCCASsGA1UdIASCASIwggEeMIIBGgYLYIZIAYb5LwAGAgEwggEJMEoGCCsGAQUFBwIB
Fj5odHRwczovL3NlY3VyZS5pZGVudHJ1c3QuY29tL2NlcnRpZmljYXRlcy9wb2xpY3kvdHMvaW5k
ZXguaHRtbDCBugYIKwYBBQUHAgIwga0MgapUaGlzIFRydXN0SUQgQ2VydGlmaWNhdGUgaGFzIGJl
ZW4gaXNzdWVkIGluIGFjY29yZGFuY2Ugd2l0aCBJZGVuVHJ1c3QncyBUcnVzdElEIENlcnRpZmlj
YXRlIFBvbGljeSBmb3VuZCBhdCBodHRwczovL3NlY3VyZS5pZGVudHJ1c3QuY29tL2NlcnRpZmlj
YXRlcy9wb2xpY3kvdHMvaW5kZXguaHRtbDBFBgNVHR8EPjA8MDqgOKA2hjRodHRwOi8vdmFsaWRh
dGlvbi5pZGVudHJ1c3QuY29tL2NybC90cnVzdGlkY2FhMTMuY3JsMB8GA1UdEQQYMBaBFGphbHRt
YW5AYXVyaXN0b3IuY29tMB0GA1UdDgQWBBQB+nzqgljLocLTsiUn2yWqEc2sgjAdBgNVHSUEFjAU
BggrBgEFBQcDAgYIKwYBBQUHAwQwDQYJKoZIhvcNAQELBQADggEBAJwVeycprp8Ox1npiTyfwc5Q
aVaqtoe8Dcg2JXZc0h4DmYGW2rRLHp8YL43snEV93rPJVk6B2v4cWLeQfaMrnyNeEuvHx/2CT44c
dLtaEk5zyqo3GYJYlLcRVz6EcSGHv1qPXgDT0xB/25etwGYqutYF4Chkxu4KzIpq90eDMw5ajkex
w+8ARQz4N5+d6NRbmMCovd7wTGi8th/BZvz8hgKUiUJoQle4wDxrdXdnIhCP7g87InXKefWgZBF4
VX21t2+hkc04qrhIJlHrocPG9mRSnnk2WpsY0MXta8ivbVKtfpY7uSNDZSKTDi1izEFH5oeQdYRk
gIGb319a7FjslV8wggaXMIIEf6ADAgECAhBAAXA7OrqBjMk8rp4OuNQSMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRIwEAYDVQQKEwlJZGVuVHJ1c3QxJzAlBgNVBAMTHklkZW5UcnVzdCBD
b21tZXJjaWFsIFJvb3QgQ0EgMTAeFw0yMDAyMTIyMTA3NDlaFw0zMDAyMTIyMTA3NDlaMDoxCzAJ
BgNVBAYTAlVTMRIwEAYDVQQKEwlJZGVuVHJ1c3QxFzAVBgNVBAMTDlRydXN0SUQgQ0EgQTEzMIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAu6sUO01SDD99PM+QdZkNxKxJNt0NgQE+Zt6i
xaNP0JKSjTd+SG5LwqxBWjnOgI/3dlwgtSNeN77AgSs+rA4bK4GJ75cUZZANUXRKw/et8pf9Qn6i
qgB63OdHxBN/15KbM3HR+PyiHXQoUVIevCKW8nnlWnnZabT1FejOhRRKVUg5HACGOTfnCOONrlxl
g+m1Vjgno1uNqNuLM/jkD1z6phNZ/G9IfZGI0ppHX5AA/bViWceX248VmefNhSR14ADZJtlAAWOi
2un03bqrBPHA9nDyXxI8rgWLfUP5rDy8jx2hEItg95+ORF5wfkGUq787HBjspE86CcaduLka/Bk2
VwIDAQABo4IChzCCAoMwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwgYkGCCsG
AQUFBwEBBH0wezAwBggrBgEFBQcwAYYkaHR0cDovL2NvbW1lcmNpYWwub2NzcC5pZGVudHJ1c3Qu
Y29tMEcGCCsGAQUFBzAChjtodHRwOi8vdmFsaWRhdGlvbi5pZGVudHJ1c3QuY29tL3Jvb3RzL2Nv
bW1lcmNpYWxyb290Y2ExLnA3YzAfBgNVHSMEGDAWgBTtRBnA0/AGi+6ke75C5yZUyI42djCCASQG
A1UdIASCARswggEXMIIBEwYEVR0gADCCAQkwSgYIKwYBBQUHAgEWPmh0dHBzOi8vc2VjdXJlLmlk
ZW50cnVzdC5jb20vY2VydGlmaWNhdGVzL3BvbGljeS90cy9pbmRleC5odG1sMIG6BggrBgEFBQcC
AjCBrQyBqlRoaXMgVHJ1c3RJRCBDZXJ0aWZpY2F0ZSBoYXMgYmVlbiBpc3N1ZWQgaW4gYWNjb3Jk
YW5jZSB3aXRoIElkZW5UcnVzdCdzIFRydXN0SUQgQ2VydGlmaWNhdGUgUG9saWN5IGZvdW5kIGF0
IGh0dHBzOi8vc2VjdXJlLmlkZW50cnVzdC5jb20vY2VydGlmaWNhdGVzL3BvbGljeS90cy9pbmRl
eC5odG1sMEoGA1UdHwRDMEEwP6A9oDuGOWh0dHA6Ly92YWxpZGF0aW9uLmlkZW50cnVzdC5jb20v
Y3JsL2NvbW1lcmNpYWxyb290Y2ExLmNybDAdBgNVHQ4EFgQULbfeG1l+KpguzeHUG+PFEBJe6RQw
HQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMA0GCSqGSIb3DQEBCwUAA4ICAQB/7BKcygLX
6Nl4a03cDHt7TLdPxCzFvDF2bkVYCFTRX47UfeomF1gBPFDee3H/IPlLRmuTPoNt0qjdpfQzmDWN
95jUXLdLPRToNxyaoB5s0hOhcV6H08u3FHACBif55i0DTDzVSaBv0AZ9h1XeuGx4Fih1Vm3Xxz24
GBqqVudvPRLyMJ7u6hvBqTIKJ53uCs3dyQLZT9DXnp+kJv8y7ZSAY+QVrI/dysT8avtn8d7k7azN
BkfnbRq+0e88QoBnel6u+fpwbd5NLRHywXeH+phbzULCa+bLPRMqJaW2lbhvSWrMHRDy3/d8Hvgn
LCBFK2s4Spns4YCN4xVcbqlGWzgolHCKUH39vpcsDo1ymZFrJ8QR6ihIn8FmJ5oKwAnnd/G6ADXF
C9budb9+532phSAXOZrrecIQn+vtP366PC+aClAPsIIDJDsotS5z4X2JUFsNIuEgXGqhiKE7SuZb
rFG9sdcLprSlJN7TsRDc0W2b9nqwD+rj/5MN0C+eKwha+8ydv0+qzTyxPP90KRgaegGowC4dUsZy
Tk2n4Z3MuAHX5nAZL/Vh/SyDj/ajorV44yqZBzQ3ChKhXbfUSwe2xMmygA2Z5DRwMRJnp/BscizY
dNk2WXJMTnH+wVLN8sLEwEtQR4eTLoFmQvrK2AMBS9kW5sBkMzINt/ZbbcZ3F+eAMDGCAqYwggKi
AgEBME4wOjELMAkGA1UEBhMCVVMxEjAQBgNVBAoTCUlkZW5UcnVzdDEXMBUGA1UEAxMOVHJ1c3RJ
RCBDQSBBMTMCEEABgmmaL+s+f8XR8nIOXMwwDQYJYIZIAWUDBAIBBQCgggEpMBgGCSqGSIb3DQEJ
AzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTI0MTExNDAyNDU0NVowLwYJKoZIhvcNAQkE
MSIEIL4QiWQUKs7SKtKL9gUpP/z4/FuUs2hPxmNapQ5vM890MF0GCSsGAQQBgjcQBDFQME4wOjEL
MAkGA1UEBhMCVVMxEjAQBgNVBAoTCUlkZW5UcnVzdDEXMBUGA1UEAxMOVHJ1c3RJRCBDQSBBMTMC
EEABgmmaL+s+f8XR8nIOXMwwXwYLKoZIhvcNAQkQAgsxUKBOMDoxCzAJBgNVBAYTAlVTMRIwEAYD
VQQKEwlJZGVuVHJ1c3QxFzAVBgNVBAMTDlRydXN0SUQgQ0EgQTEzAhBAAYJpmi/rPn/F0fJyDlzM
MA0GCSqGSIb3DQEBCwUABIIBAEjaRcPL6tEhXFMWawiti68wt+9bnsk7PYMzO5pgY1hj1dyGX4cX
6vPufvwO+9BD484YHLaGlD3HIR/1X90XuqyjqTQbehkA6MZOzgNzAd21yHikNdhBr83ZCOI47uX/
BKxJqMSXyyFneMwpPqADOjEzq1noXRDMduZySULoqhWe7/GJ7VMGsv7j6yiWHWUEXAAdMBqpe3lt
e1tKlaHfdl1v71tZTedD+r2XvqLmT5tkCEmKbPZZEklBd75VnYfQ52yopVRnQ25L5i9HLNyHR9BZ
e6+0s117qgHqaw6ulBtRZej28uK4E0wv2IqAOJCiObH1337kElgxVZ0gHIFKZ5kAAAAAAAA=
--Apple-Mail=_74022820-1F2B-4D05-B964-641182CF32E7--