From jsbillin@umich.edu Fri Mar 5 14:07:43 2021 From: jsbillin@umich.edu (Jonathan Billings) Date: Fri, 5 Mar 2021 09:07:43 -0500 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor Message-ID: --00000000000047a00905bcca9c57 Content-Type: text/plain; charset="UTF-8" Hello, Our university uses the Crowdstrike endpoint security tool, and we use OpenAFS for both our user's home directory as well as serving software to our students, faculty and researchers. Is anyone else using Crowdstrike and OpenAFS on Linux (specifically, RHEL7)? I've discovered that the Crowdstrike service (falcon-sensor) installs a linux security module which seems to interact with the OpenAFS kernel module in a bad way, causing the kernel to panic and reboot. After installing the kdump service, I'm able to capture a kernel dump and backtrace, and it is definitely something to do with how OpenAFS and the falcon lsm interact. I wasn't able to trigger it with just command-line ssh but a graphical login seems to be a reliable trigger. Specifically, it seems to be in the cache handling when it panics. Has anyone else experienced this? -- Jonathan Billings (he/his) College of Engineering - CAEN - Linux Support --00000000000047a00905bcca9c57 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

Our university uses t= he Crowdstrike endpoint security tool, and we use OpenAFS for both our user= 's home directory as well as serving software to our students, faculty = and researchers.=C2=A0 Is anyone else using Crowdstrike and OpenAFS on Linu= x (specifically, RHEL7)?

I've discovered t= hat the Crowdstrike service (falcon-sensor) installs a linux security modul= e which seems to interact with the OpenAFS kernel module in a bad way, caus= ing the kernel to panic and reboot.=C2=A0 After installing the kdump servic= e, I'm able to capture a kernel dump and backtrace, and it is definitel= y something to do with how OpenAFS and the falcon lsm interact.=C2=A0 I was= n't able to trigger it with just command-line ssh but a graphical login= seems to be a reliable trigger.=C2=A0 Specifically, it seems to be in the = cache handling when it panics.

Has anyone else= experienced this?

--
Jonathan Bi= llings <jsbillin= @umich.edu> (he/his)
College of Engineering - CAEN - Linux Suppor= t
--00000000000047a00905bcca9c57-- From kaduk@mit.edu Sun Mar 7 21:34:45 2021 From: kaduk@mit.edu (Benjamin Kaduk) Date: Sun, 7 Mar 2021 13:34:45 -0800 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor In-Reply-To: References: Message-ID: <20210307213445.GV56617@kduck.mit.edu> On Fri, Mar 05, 2021 at 09:07:43AM -0500, Jonathan Billings wrote: > Hello, > > Our university uses the Crowdstrike endpoint security tool, and we use > OpenAFS for both our user's home directory as well as serving software to > our students, faculty and researchers. Is anyone else using Crowdstrike > and OpenAFS on Linux (specifically, RHEL7)? > > I've discovered that the Crowdstrike service (falcon-sensor) installs a > linux security module which seems to interact with the OpenAFS kernel > module in a bad way, causing the kernel to panic and reboot. After > installing the kdump service, I'm able to capture a kernel dump and > backtrace, and it is definitely something to do with how OpenAFS and the > falcon lsm interact. I wasn't able to trigger it with just command-line > ssh but a graphical login seems to be a reliable trigger. Specifically, it > seems to be in the cache handling when it panics. > > Has anyone else experienced this? I don't use Crowdstrike so haven't seen it, but can you post the backtrace? Thanks, Ben From jon@csail.mit.edu Mon Mar 8 14:56:02 2021 From: jon@csail.mit.edu (Jonathan D. Proulx) Date: Mon, 8 Mar 2021 09:56:02 -0500 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor In-Reply-To: References: Message-ID: <20210308145602.yschfruedl7hx2xg@csail.mit.edu> We at MIT CSAIL stoped using crowdstrike partly becuase they refused to fix this despite us providing a patch to falcon-sensor (whcih is just a tarred pile of shell scripts). The need to excluse /afs from their scans there's several ways to do this (they use "find" internally). We found them unhelpful and very good at talkign to magnagement types and very bad at anyting actually technical. I can't say enough against them...they'll probably call our Office of General Council on me for saying anything against them they've done that before (and put more effort into "unmasking" the pseudonymous jr admin's reddit user account than the actual security issues we were paying them to look at). -Jon --- Jonathan Proulx Sr. Technical Architect MIT CSAIL On Fri, Mar 05, 2021 at 09:07:43AM -0500, Jonathan Billings wrote: :Hello, : :Our university uses the Crowdstrike endpoint security tool, and we use :OpenAFS for both our user's home directory as well as serving software to :our students, faculty and researchers. Is anyone else using Crowdstrike :and OpenAFS on Linux (specifically, RHEL7)? : :I've discovered that the Crowdstrike service (falcon-sensor) installs a :linux security module which seems to interact with the OpenAFS kernel :module in a bad way, causing the kernel to panic and reboot. After :installing the kdump service, I'm able to capture a kernel dump and :backtrace, and it is definitely something to do with how OpenAFS and the :falcon lsm interact. I wasn't able to trigger it with just command-line :ssh but a graphical login seems to be a reliable trigger. Specifically, it :seems to be in the cache handling when it panics. : :Has anyone else experienced this? : :-- :Jonathan Billings (he/his) :College of Engineering - CAEN - Linux Support -- From kenh@cmf.nrl.navy.mil Mon Mar 8 15:06:44 2021 From: kenh@cmf.nrl.navy.mil (Ken Hornstein) Date: Mon, 08 Mar 2021 10:06:44 -0500 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor In-Reply-To: <20210308145602.yschfruedl7hx2xg@csail.mit.edu> References: <20210308145602.yschfruedl7hx2xg@csail.mit.edu> Message-ID: <202103081505.128F536B012483@hedwig.cmf.nrl.navy.mil> >We at MIT CSAIL stoped using crowdstrike partly becuase they refused >to fix this despite us providing a patch to falcon-sensor (whcih is >just a tarred pile of shell scripts). > >The need to excluse /afs from their scans there's several ways to do >this (they use "find" internally). > >We found them unhelpful and very good at talkign to magnagement types >and very bad at anyting actually technical. For what it's worth ... we ran into this EXACT issue not with crowdstrike, but some other similar product (which I want to say was McAfee something or other, maybe). The situation was even more comical, because, AGAIN, all they had to do was exclude /afs, but ... as it was explained to me, the online portal to submit change requests was broken and we couldn't formally submit the change request. And the online portal was broken for ... years? Like, LITERALLY, it was down for at least a year. There were a lot of management layers between us and the people who could submit the change request, so I don't know how accurate that was. And this was a couple of years ago so maybe the situation has changed. But the general obnoxiousness of the security software vendor seems to be universal, sadly. The upside was, however, that because it ended up crashing our system we could use the legitimate excuse, "We can't run that, it crashes our systems and the support portal doesn't work, see ticket X". So ... the system works, I guess? --Ken From jon@csail.mit.edu Mon Mar 8 15:33:22 2021 From: jon@csail.mit.edu (Jonathan Proulx) Date: Mon, 8 Mar 2021 10:33:22 -0500 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor In-Reply-To: <202103081505.128F536B012483@hedwig.cmf.nrl.navy.mil> References: <20210308145602.yschfruedl7hx2xg@csail.mit.edu> <202103081505.128F536B012483@hedwig.cmf.nrl.navy.mil> Message-ID: <20210308153322.o3wnm3hgiwe2gbrq@csail.mit.edu> On Mon, Mar 08, 2021 at 10:06:44AM -0500, Ken Hornstein wrote: :>We at MIT CSAIL stoped using crowdstrike partly becuase they refused :>to fix this despite us providing a patch to falcon-sensor (whcih is :>just a tarred pile of shell scripts). :> :>The need to excluse /afs from their scans there's several ways to do :>this (they use "find" internally). :> :>We found them unhelpful and very good at talkign to magnagement types :>and very bad at anyting actually technical. : :For what it's worth ... we ran into this EXACT issue not with crowdstrike, :but some other similar product (which I want to say was McAfee something :or other, maybe). The situation was even more comical, because, AGAIN, :all they had to do was exclude /afs, but ... The find line in the crowdstrike "tool" already has multiple excludes for other filesystems types, btw. Obviously I can't distribute the "patch" which was one extra flag to an existing find line if I recall. If you have the Falconsensor it's not hard to unpack and sort out, when we were using it (spring 2019) it wasn't signed in any meaningful way (maybe a sha sum to replace but not crypto I forget) so it would happily run modified code as root. This was another reason we rejected it, but if you're in a position where you don't have that control and to have licensing to get the to tool it's easily hackable. but none of that is good. -Jon From jsbillin@umich.edu Mon Mar 8 18:04:18 2021 From: jsbillin@umich.edu (Jonathan Billings) Date: Mon, 8 Mar 2021 13:04:18 -0500 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor In-Reply-To: <20210307213445.GV56617@kduck.mit.edu> References: <20210307213445.GV56617@kduck.mit.edu> Message-ID: --000000000000e4e14605bd0a4351 Content-Type: text/plain; charset="UTF-8" On Sun, Mar 7, 2021 at 4:34 PM Benjamin Kaduk wrote: > I don't use Crowdstrike so haven't seen it, but can you post the backtrace? > Based on what I've heard from Mr. Proulx at MIT (and from others off-list), I have put in a ticket with Crowdstrike asking if I can share the kernel backtrace. I honestly feel like it should be OK but I don't want to risk my job over it. -- Jonathan Billings (he/his) College of Engineering - CAEN - Linux Support --000000000000e4e14605bd0a4351 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Sun, Mar 7, 2021 at 4:34 PM Benjamin K= aduk <kaduk@mit.edu> wrote:
<= /div>
I don't use Crowdstrike so haven't seen it, but can you post the ba= cktrace?

Based on what I've heard f= rom Mr. Proulx at MIT (and from others off-list), I have put in a ticket wi= th Crowdstrike asking if I can share the kernel backtrace.=C2=A0 I honestly= feel like it should be OK but I don't want to risk my job over it.
=

--
Jonathan Billings <jsbillin@umich.edu> (he/his)
College of Engineering -= CAEN - Linux Support
--000000000000e4e14605bd0a4351-- From martin.kelly@crowdstrike.com Mon Mar 8 19:35:19 2021 From: martin.kelly@crowdstrike.com (Martin Kelly) Date: Mon, 8 Mar 2021 19:35:19 +0000 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor Message-ID: ------=_NextPart_000_016E_01D7140F.1DA904C0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Mar 7, 2021 at 4:34 PM Benjamin Kaduk = wrote: > > I don't use Crowdstrike so haven't seen it, but can you post the = backtrace? > Based on what I've heard from Mr. Proulx at MIT (and from others = off-list), I have put in a ticket with Crowdstrike asking if I can share = the kernel backtrace. I honestly feel like it should be OK but I don't = want to risk my job over it. Hi, I=E2=80=99m an engineer at CrowdStrike. There is a known issue in which = OpenAFS can cause the CrowdStrike LSM to crash because current->fs can = be set to NULL in a certain code path in which it should not be NULL = because we=E2=80=99re in process context. I double-checked this on the = upstream LSM mailing list after looking at a stack trace. I had thought = that a bug report had gotten back to OpenAFS but it seems like that = didn=E2=80=99t happen; sorry about that! Below is the LKML LSM thread regarding this. Please let me know if you = have any other questions: https://www.spinics.net/lists/linux-security-module/msg39081.html https://www.spinics.net/lists/linux-security-module/msg39083.html ------=_NextPart_000_016E_01D7140F.1DA904C0 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIOnDCCBCow ggMSoAMCAQICBDhj3vgwDQYJKoZIhvcNAQEFBQAwgbQxFDASBgNVBAoTC0VudHJ1c3QubmV0MUAw PgYDVQQLFDd3d3cuZW50cnVzdC5uZXQvQ1BTXzIwNDggaW5jb3JwLiBieSByZWYuIChsaW1pdHMg bGlhYi4pMSUwIwYDVQQLExwoYykgMTk5OSBFbnRydXN0Lm5ldCBMaW1pdGVkMTMwMQYDVQQDEypF bnRydXN0Lm5ldCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAoMjA0OCkwHhcNOTkxMjI0MTc1MDUx WhcNMjkwNzI0MTQxNTEyWjCBtDEUMBIGA1UEChMLRW50cnVzdC5uZXQxQDA+BgNVBAsUN3d3dy5l bnRydXN0Lm5ldC9DUFNfMjA0OCBpbmNvcnAuIGJ5IHJlZi4gKGxpbWl0cyBsaWFiLikxJTAjBgNV BAsTHChjKSAxOTk5IEVudHJ1c3QubmV0IExpbWl0ZWQxMzAxBgNVBAMTKkVudHJ1c3QubmV0IENl cnRpZmljYXRpb24gQXV0aG9yaXR5ICgyMDQ4KTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC ggEBAK1NS6kShrLqoyAHFRZkKitL0b8LSk2O7YB2pWe3eEDAc0LIaMDbUyvdXrh2mDWTixqdfBM6 Dh9btx7P5SQUHrGBqY19uMxrSwPxAgzcq6VAJAB/dJShnQgps4gL9Yd3nVXN5MN+12pkq4UUhpVb lzJQbz3IumYM4/y9uEnBdolJGf3AqL2Jo2cvxp+8cRlguC3pLMmQdmZ7lOKveNZlU1081pyyzykD +S+kULLUSM4FMlWK/bJkTA7kmAd123/fuQhVYIUwKfl7SKRphuM1Px6GXXp6Fb3vAI4VIlQXAJAm k7wOSWiRv/hH052VQsEOTd9vJs/DGCFiZkNw1tXAB+ECAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEG MA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFFXkgdERgL7YibkIozH5oSQJFrlwMA0GCSqGSIb3 DQEBBQUAA4IBAQA7m49WmzDnU5l8enmnTZfXGZWQ+wYfyjN8RmOPlmYk+kAbISfK5nJz8k/+MZn9 yAxMaFPGgIITmPq2rdpdPfHObvYVEZSCDO4/la8Rqw/XL94fA49XLB7Ju5oaRJXrGE+mH819VxAv mwQJWoS1btgdOuHWntFseV55HBTF49BMkztlPO3fPb6m5ZUaw7UZw71eW7v/I+9oGcsSkydcAy1v MNAethqs3lr30aqoJ6b+eYHEeZkzV7oSsKngQmyTylbe/m2ECwiLfo3q15ghxvPnPHkvXpzRTBWN 4ewiN8yaQwuX3ICQjbNnm29ICBVWz7/xK3xemnbpWZDFfIM1EWVRMIIFFTCCA/2gAwIBAgIRAK8c BLKsjP+bAAAAAFHOGOMwDQYJKoZIhvcNAQELBQAwgbQxFDASBgNVBAoTC0VudHJ1c3QubmV0MUAw PgYDVQQLFDd3d3cuZW50cnVzdC5uZXQvQ1BTXzIwNDggaW5jb3JwLiBieSByZWYuIChsaW1pdHMg bGlhYi4pMSUwIwYDVQQLExwoYykgMTk5OSBFbnRydXN0Lm5ldCBMaW1pdGVkMTMwMQYDVQQDEypF bnRydXN0Lm5ldCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAoMjA0OCkwHhcNMjAwNzI5MTU0ODMw WhcNMjkwNjI5MTYxODMwWjCBpTELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4x OTA3BgNVBAsTMHd3dy5lbnRydXN0Lm5ldC9DUFMgaXMgaW5jb3Jwb3JhdGVkIGJ5IHJlZmVyZW5j ZTEfMB0GA1UECxMWKGMpIDIwMTAgRW50cnVzdCwgSW5jLjEiMCAGA1UEAxMZRW50cnVzdCBDbGFz cyAyIENsaWVudCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMQyjULQnhmdW5Ba EEy1EAAhuQdI3q5ugNb/FFAG6HWva0aO56VPrcOMsPp74BmR/fBjrXFJ86gcH6s0GSBOS1TpAJO+ cAgx3olTrFe8JO8qj0LU9+qVJV0UdtLNpxL6G7K0XGFAvV/dV5tEVdjFiRk8ZT256NSlLcIs0+qD MaIIPF5ZrhIuKgqMXvOzMa4KrX7ssEkJ/KcuIh5oZDSdFuOmPQMxQBb3lPZLGTTJl+YinEjeZKCD C1gFmMQiRokF/aO+9klMYQMWpPgKmRziwMZ+aQIyV5ADrwCUobnczq/v9HwYzjALyof41V8fWVHY iwu5OMZYwlN82ibU2/K9kM0CAwEAAaOCAS0wggEpMA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAU BggrBgEFBQcDBAYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAzBggrBgEFBQcBAQQnMCUw IwYIKwYBBQUHMAGGF2h0dHA6Ly9vY3NwLmVudHJ1c3QubmV0MDIGA1UdHwQrMCkwJ6AloCOGIWh0 dHA6Ly9jcmwuZW50cnVzdC5uZXQvMjA0OGNhLmNybDA7BgNVHSAENDAyMDAGBFUdIAAwKDAmBggr BgEFBQcCARYaaHR0cDovL3d3dy5lbnRydXN0Lm5ldC9ycGEwHQYDVR0OBBYEFAmRpbrp8i4qdd/N fv53yvLea5skMB8GA1UdIwQYMBaAFFXkgdERgL7YibkIozH5oSQJFrlwMA0GCSqGSIb3DQEBCwUA A4IBAQA/vekQdfNCp9FsgSahRiBXEiQVWrIMCH/dR7k/QpOkCq9MEe7MazD0tCyE3goXkPl4NK6u JkV2BTUkg8CTc5lPpXJxY7QJiBHLbG7vlJXVSTfPoQDwDUsUUUb0aHGy/mChNw8l/O8gWjPGqYfJ 6lL212lIls5azxCb9rcBwzohpchDwISdA/jFNAiHy4sKg1yqIyvp/7jep0kObTIVgTDIJ/TA/s8a dcyHu7oRoYJlUAWf80WSh6BFuBnnX/hGClvM2F1rFpFMFZVq4+T83gZ09mxU3cQl8GkW1uoOP1m+ AWL5YJ8dQLMx9xCcL/mKRGbYYAJOMRCx9peO/iCDvU1KMIIFUTCCBDmgAwIBAgIQDJP99GX/o7gA AAAATDnPWjANBgkqhkiG9w0BAQsFADCBpTELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3Qs IEluYy4xOTA3BgNVBAsTMHd3dy5lbnRydXN0Lm5ldC9DUFMgaXMgaW5jb3Jwb3JhdGVkIGJ5IHJl ZmVyZW5jZTEfMB0GA1UECxMWKGMpIDIwMTAgRW50cnVzdCwgSW5jLjEiMCAGA1UEAxMZRW50cnVz dCBDbGFzcyAyIENsaWVudCBDQTAeFw0yMDEwMTQyMjA2NDRaFw0yMjEwMTQyMjM2NDNaMIGRMQsw CQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTEPMA0GA1UEBxMGSXJ2aW5lMRowGAYDVQQK ExFDcm93ZFN0cmlrZSwgSW5jLjFAMBMGA1UEAxMMTWFydGluIEtlbGx5MCkGCSqGSIb3DQEJARYc bWFydGluLmtlbGx5QGNyb3dkc3RyaWtlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC ggEBAMT46ZuzJ/RhZGH9+qht+LO1VOIYXTlKXOYwfashRkS/Bbro15Km8ARz41X1CQ/9AmjPgPFL wFJXQlYZZybTsGl1o/R4PBmZUZ9AwfqHPXUrM3WKP3A4s5mVffPilfe6lTuSoipTzq8MoOaz2rK/ 4kffBpTqDL3MvYvJX9hva2UJ4RvhQ9kS/6NAf/2mobNyVufM2NAAdUyQlUV4A8Hgdz+GklWtDpzR jdZjGs7Cb+cylSVG/X8ntvq2idrWOUu8/2gH6aV1ZiR3G1U3I8HaYAdZK6ycXNfvdFSrePv9TKCk 7ud/0v5RwTcenut579PJ9J1ILBxxrphRD1MsulKWP4MCAwEAAaOCAY0wggGJMA4GA1UdDwEB/wQE AwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwQgYDVR0gBDswOTA3BgtghkgBhvps CgEEAjAoMCYGCCsGAQUFBwIBFhpodHRwOi8vd3d3LmVudHJ1c3QubmV0L3JwYTBqBggrBgEFBQcB AQReMFwwIwYIKwYBBQUHMAGGF2h0dHA6Ly9vY3NwLmVudHJ1c3QubmV0MDUGCCsGAQUFBzAChilo dHRwOi8vYWlhLmVudHJ1c3QubmV0LzIwNDhjbGFzczJzaGEyLmNlcjA0BgNVHR8ELTArMCmgJ6Al hiNodHRwOi8vY3JsLmVudHJ1c3QubmV0L2NsYXNzMmNhLmNybDAnBgNVHREEIDAegRxtYXJ0aW4u a2VsbHlAY3Jvd2RzdHJpa2UuY29tMB8GA1UdIwQYMBaAFAmRpbrp8i4qdd/Nfv53yvLea5skMB0G A1UdDgQWBBSlEgDqZESbubhJcEKhhhhKnl7OtDAJBgNVHRMEAjAAMA0GCSqGSIb3DQEBCwUAA4IB AQCoPnFPAX5tAP+aIr17tIgq1L2nP1F2pTeHgXd7ETwtUF1rSfq22mtKsDslQ5cRxy6l8LzH9j1h 2QmWK6dd+lSOUR/KaDSnXyhXsCHBX9455sWu6+n6Umrol/kOoRB3eFAht8wvXyIkNWoGf2M+p1kC r7NYzgTDauE40ch1P15zLw47j9BODeX1BRhzlal0mrdEbLgZmcCdThe9S3+X1WOucSUIA54mFl46 uuZWlpl2lSKCrvWZn5RcGWqrdbM27SsNipli0sz0H+jPKUtFfr3DNsPQTANJsPU3u9k9A5HropDq sZRj+3j+2z9Yiyi01g1k7bjc6LHmxXa0YjhQeMA8MYIEdzCCBHMCAQEwgbowgaUxCzAJBgNVBAYT AlVTMRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMTkwNwYDVQQLEzB3d3cuZW50cnVzdC5uZXQvQ1BT IGlzIGluY29ycG9yYXRlZCBieSByZWZlcmVuY2UxHzAdBgNVBAsTFihjKSAyMDEwIEVudHJ1c3Qs IEluYy4xIjAgBgNVBAMTGUVudHJ1c3QgQ2xhc3MgMiBDbGllbnQgQ0ECEAyT/fRl/6O4AAAAAEw5 z1owCQYFKw4DAhoFAKCCApEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUx DxcNMjEwMzA4MTkzNTE4WjAjBgkqhkiG9w0BCQQxFgQUGjRi3ux+siFRqM1n++as9YCZJn4wgZMG CSqGSIb3DQEJDzGBhTCBgjALBglghkgBZQMEASowCwYJYIZIAWUDBAEWMAoGCCqGSIb3DQMHMAsG CWCGSAFlAwQBAjAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAhowCwYJYIZI AWUDBAIDMAsGCWCGSAFlAwQCAjALBglghkgBZQMEAgEwgcsGCSsGAQQBgjcQBDGBvTCBujCBpTEL MAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xOTA3BgNVBAsTMHd3dy5lbnRydXN0 Lm5ldC9DUFMgaXMgaW5jb3Jwb3JhdGVkIGJ5IHJlZmVyZW5jZTEfMB0GA1UECxMWKGMpIDIwMTAg RW50cnVzdCwgSW5jLjEiMCAGA1UEAxMZRW50cnVzdCBDbGFzcyAyIENsaWVudCBDQQIQDJP99GX/ o7gAAAAATDnPWjCBzQYLKoZIhvcNAQkQAgsxgb2ggbowgaUxCzAJBgNVBAYTAlVTMRYwFAYDVQQK Ew1FbnRydXN0LCBJbmMuMTkwNwYDVQQLEzB3d3cuZW50cnVzdC5uZXQvQ1BTIGlzIGluY29ycG9y YXRlZCBieSByZWZlcmVuY2UxHzAdBgNVBAsTFihjKSAyMDEwIEVudHJ1c3QsIEluYy4xIjAgBgNV BAMTGUVudHJ1c3QgQ2xhc3MgMiBDbGllbnQgQ0ECEAyT/fRl/6O4AAAAAEw5z1owDQYJKoZIhvcN AQEBBQAEggEAQKvhpKwgPn6jaLmaBwVhzbf8+wvyaY8VFppWsaamNxBgmuMVxhLKTt/BN+JcBFcL +lgUFtHXHLy7cyB6wWac+b8jGdtTsQXCUVvrra+6HdGQgmXMeSjMVHQNK6zm6j5jxPvvN3yeP036 VSIOD9C+H9KcgWFRNkRemaIi8alS0BChUa9gvHzx5dxg/c4IMCtIMRHP5s4mH08Gkiq9sUJWdIl5 YW1m1VLV58vHZlvxewrWalDBfL+6xne5/+hYo1FIPYUS+kAWUT/dQCspQDN5xNTko5HPu8IqELn+ R6B8gz7Roi4rjHvRweT3vFg5BFFthJnY870lyvOO06kmz4f3aAAAAAAAAA== ------=_NextPart_000_016E_01D7140F.1DA904C0-- From jsbillin@umich.edu Mon Mar 8 19:43:47 2021 From: jsbillin@umich.edu (Jonathan Billings) Date: Mon, 8 Mar 2021 14:43:47 -0500 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor In-Reply-To: References: Message-ID: --000000000000a9523e05bd0ba793 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Do you know if it would be OK for me to share my kernel backtrace with the OpenAFS list? On Mon, Mar 8, 2021 at 2:37 PM Martin Kelly wrote: > On Sun, Mar 7, 2021 at 4:34 PM Benjamin Kaduk > wrote: > > > I don't use Crowdstrike so haven't seen it, but can you post the > backtrace? > > > Based on what I've heard from Mr. Proulx at MIT (and from others > off-list), I have put in a ticket with Crowdstrike asking if I can share > the kernel backtrace. I honestly feel like it should be OK but I don't > want to risk my job over it. > > Hi, > > I=E2=80=99m an engineer at CrowdStrike. There is a known issue in which O= penAFS > can cause the CrowdStrike LSM to crash because current->fs can be set to > NULL in a certain code path in which it should not be NULL because we=E2= =80=99re in > process context. I double-checked this on the upstream LSM mailing list > after looking at a stack trace. I had thought that a bug report had gotte= n > back to OpenAFS but it seems like that didn=E2=80=99t happen; sorry about= that! > > Below is the LKML LSM thread regarding this. Please let me know if you > have any other questions: > > https://www.spinics.net/lists/linux-security-module/msg39081.html > https://www.spinics.net/lists/linux-security-module/msg39083.html > --=20 Jonathan Billings (he/his) College of Engineering - CAEN - Linux Support --000000000000a9523e05bd0ba793 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Do you know if it would be OK for me to share my kernel ba= cktrace with the OpenAFS list?=C2=A0

On Mon, Mar 8, 2021 at 2:37 PM Mar= tin Kelly <martin.kelly@= crowdstrike.com> wrote:
On Sun, Mar 7, 2021 at 4:34 PM Benjamin Kaduk <mailto:kaduk@mit.edu> wrote= :
> > I don't use Crowdstrike so haven't seen it, but can you p= ost the backtrace?

> Based on what I've heard from Mr. Proulx at MIT (and from others o= ff-list), I have put in a ticket with Crowdstrike asking if I can share the= kernel backtrace.=C2=A0 I honestly feel like it should be OK but I don'= ;t want to risk my job over it.

Hi,

I=E2=80=99m an engineer at CrowdStrike. There is a known issue in which Ope= nAFS can cause the CrowdStrike LSM to crash because current->fs can be s= et to NULL in a certain code path in which it should not be NULL because we= =E2=80=99re in process context. I double-checked this on the upstream LSM m= ailing list after looking at a stack trace. I had thought that a bug report= had gotten back to OpenAFS but it seems like that didn=E2=80=99t happen; s= orry about that!

Below is the LKML LSM thread regarding this. Please let me know if you have= any other questions:

https://www.spinics.net/lists/linux= -security-module/msg39081.html
https://www.spinics.net/lists/linux= -security-module/msg39083.html


--
Jonathan Billings <jsbillin@umich.edu> (he/his)
= College of Engineering - CAEN - Linux Support
--000000000000a9523e05bd0ba793-- From jon@csail.mit.edu Mon Mar 8 19:53:50 2021 From: jon@csail.mit.edu (Jonathan Proulx) Date: Mon, 8 Mar 2021 14:53:50 -0500 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor In-Reply-To: <20210308145602.yschfruedl7hx2xg@csail.mit.edu> References: <20210308145602.yschfruedl7hx2xg@csail.mit.edu> Message-ID: <20210308195350.qxg34ew6ahms4zul@csail.mit.edu> On Mon, Mar 08, 2021 at 09:56:02AM -0500, Jonathan D. Proulx wrote: : :We at MIT CSAIL stoped using crowdstrike partly becuase they refused :to fix this despite us providing a patch to falcon-sensor (whcih is :just a tarred pile of shell scripts). Slight correction, the AFS portion of our problems was with a different CrowdStike tool not the "falcon-sensor" ... lowlevel trauma from the interaction crossed my circuits. -Jon From kaduk@mit.edu Tue Mar 9 00:20:53 2021 From: kaduk@mit.edu (Benjamin Kaduk) Date: Mon, 8 Mar 2021 16:20:53 -0800 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor In-Reply-To: References: Message-ID: <20210309002053.GU56617@kduck.mit.edu> Hi Martin, On Mon, Mar 08, 2021 at 07:35:19PM +0000, Martin Kelly wrote: > On Sun, Mar 7, 2021 at 4:34 PM Benjamin Kaduk wrote: > > > I don't use Crowdstrike so haven't seen it, but can you post the backtrace? > > > Based on what I've heard from Mr. Proulx at MIT (and from others off-list), I have put in a ticket with Crowdstrike asking if I can share the kernel backtrace. I honestly feel like it should be OK but I don't want to risk my job over it. > > Hi, > > I’m an engineer at CrowdStrike. There is a known issue in which OpenAFS can cause the CrowdStrike LSM to crash because current->fs can be set to NULL in a certain code path in which it should not be NULL because we’re in process context. I double-checked this on the upstream LSM mailing list after looking at a stack trace. I had thought that a bug report had gotten back to OpenAFS but it seems like that didn’t happen; sorry about that! > > Below is the LKML LSM thread regarding this. Please let me know if you have any other questions: > > https://www.spinics.net/lists/linux-security-module/msg39081.html > https://www.spinics.net/lists/linux-security-module/msg39083.html Thanks for spotting this thread and the quick follow-up. I suspect that the changes at https://gerrit.openafs.org/#/c/13751/ are going to be relevant in this space, but without seeing the stack trace of the crash in question it's hard to be sure. Can you speak to whether this is touching the "right" part of the code with respect to the crashes you were investigating? Thanks, Ben From martin.kelly@crowdstrike.com Tue Mar 9 01:04:07 2021 From: martin.kelly@crowdstrike.com (Martin Kelly) Date: Tue, 9 Mar 2021 01:04:07 +0000 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor In-Reply-To: References: Message-ID: <52aaebfde03b41a082cd9421137819dd@crowdstrike.com> PiBEbyB5b3Uga25vdyBpZiBpdCB3b3VsZCBiZSBPSyBmb3IgbWUgdG8gc2hhcmUgbXkga2VybmVs IGJhY2t0cmFjZSB3aXRoIHRoZSBPcGVuQUZTIGxpc3Q/DQoNClllcywgcGxlYXNlIGRvIQ0K From jaltman@auristor.com Tue Mar 9 01:21:55 2021 From: jaltman@auristor.com (Jeffrey E Altman) Date: Mon, 8 Mar 2021 20:21:55 -0500 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor In-Reply-To: <20210309002053.GU56617@kduck.mit.edu> References: <20210309002053.GU56617@kduck.mit.edu> Message-ID: <40352b81-652a-7ea4-dd30-30c87f5583e8@auristor.com> This is a cryptographically signed message in MIME format. --------------ms080506090207080200070007 Content-Type: multipart/mixed; boundary="------------9D01D2BAE42A55889EFA12FD" Content-Language: en-US This is a multi-part message in MIME format. --------------9D01D2BAE42A55889EFA12FD Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable On 3/8/2021 7:20 PM, Benjamin Kaduk (kaduk@mit.edu) wrote: > On Mon, Mar 08, 2021 at 07:35:19PM +0000, Martin Kelly wrote: >> Below is the LKML LSM thread regarding this. Please let me know if you= have any other questions: >> >> https://www.spinics.net/lists/linux-security-module/msg39081.html >> https://www.spinics.net/lists/linux-security-module/msg39083.html >=20 > Thanks for spotting this thread and the quick follow-up. This is the same thread that Yadav discussed with the openafs-release=20 team on 11 Dec 2020. > I suspect that the changes at https://gerrit.openafs.org/#/c/13751/ are= > going to be relevant in this space, but without seeing the stack trace = of > the crash in question it's hard to be sure. Can you speak to whether t= his > is touching the "right" part of the code with respect to the crashes yo= u > were investigating? The suggested change was cherry-picked to openafs-stable-1_8_x as https://gerrit.openafs.org/14082 and merged as=20 ee578e92d9f810d93659a9805d0c12084fe2bb95. As Jonathan wrote to IRC OpenAFS: > (4:53:15 PM) billings: I built openafs from the latest commit in > 1_8_x and crowdstrike still panics, so it doesnt look like any > merged commits there fix my issue. Martin's e-mail describes the call pattern: > - A process exits, calling task_exit(). I think Martin meant do_exit(). > - exit_fs() is called, setting current->fs =3D NULL. task_struct field struct fs_struct *fs; > - Next, exit_task_work() is called, exit_task_work() calls task_work_run() which flushes any pending works. > which calls fput(). which must have been called by a pending work. > - In response to the fput(), the filesystem opens a file disk cache > to update some metadata, calling dentry_open(). dentry_open() in turn will trigger a call to any configured LSM. If task_struct.fs is NULL, Kaboom!!! Jeffrey Altman --------------9D01D2BAE42A55889EFA12FD Content-Type: text/x-vcard; charset=utf-8; name="jaltman.vcf" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="jaltman.vcf" begin:vcard fn:Jeffrey Altman n:Altman;Jeffrey org:AuriStor, Inc. adr:;;255 W 94TH ST STE 6B;New York;NY;10025-6985;United States email;internet:jaltman@auristor.com title:CEO tel;work:+1-212-769-9018 url:https://www.linkedin.com/in/jeffreyaltman/ version:2.1 end:vcard --------------9D01D2BAE42A55889EFA12FD-- --------------ms080506090207080200070007 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC DGswggXSMIIEuqADAgECAhBAAW0B1qVVQ32wvx2EXYU6MA0GCSqGSIb3DQEBCwUAMDoxCzAJ BgNVBAYTAlVTMRIwEAYDVQQKEwlJZGVuVHJ1c3QxFzAVBgNVBAMTDlRydXN0SUQgQ0EgQTEy MB4XDTE5MDkwNTE0MzE0N1oXDTIyMTEwMTE0MzE0N1owcDEvMC0GCgmSJomT8ixkAQETH0Ew MTQxMEMwMDAwMDE2RDAxRDZBNTQwMDAwMDQ0NDcxGTAXBgNVBAMTEEplZmZyZXkgRSBBbHRt YW4xFTATBgNVBAoTDEF1cmlTdG9yIEluYzELMAkGA1UEBhMCVVMwggEiMA0GCSqGSIb3DQEB AQUAA4IBDwAwggEKAoIBAQCY1TC9QeWnUgEoJ81FcAVnhGn/AWuzvkYRUG5/ZyXDdaM212e8 ybCklgSmZweqNdrfaaHXk9vwjpvpD4YWgb07nJ1QBwlvRV/VPAaDdneIygJJWBCzaMVLttKO 0VimH/I/HUwFBQT2mrktucCEf2qogdi2P+p5nuhnhIUiyZ71Fo43gF6cuXIMV/1rBNIJDuwM Q3H8zi6GL0p4mZFZDDKtbYq2l8+MNxFvMrYcLaJqejQNQRBuZVfv0Fq9pOGwNLAk19baIw3U xdwx+bGpTtS63Py1/57MQ0W/ZXE/Ocnt1qoDLpJeZIuEBKgMcn5/iN9+Ro5zAuOBEKg34wBS 8QCTAgMBAAGjggKcMIICmDAOBgNVHQ8BAf8EBAMCBPAwgYQGCCsGAQUFBwEBBHgwdjAwBggr BgEFBQcwAYYkaHR0cDovL2NvbW1lcmNpYWwub2NzcC5pZGVudHJ1c3QuY29tMEIGCCsGAQUF BzAChjZodHRwOi8vdmFsaWRhdGlvbi5pZGVudHJ1c3QuY29tL2NlcnRzL3RydXN0aWRjYWEx Mi5wN2MwHwYDVR0jBBgwFoAUpHPa72k1inXMoBl7CDL4a4nkQuwwCQYDVR0TBAIwADCCASsG A1UdIASCASIwggEeMIIBGgYLYIZIAYb5LwAGAgEwggEJMEoGCCsGAQUFBwIBFj5odHRwczov L3NlY3VyZS5pZGVudHJ1c3QuY29tL2NlcnRpZmljYXRlcy9wb2xpY3kvdHMvaW5kZXguaHRt bDCBugYIKwYBBQUHAgIwga0MgapUaGlzIFRydXN0SUQgQ2VydGlmaWNhdGUgaGFzIGJlZW4g aXNzdWVkIGluIGFjY29yZGFuY2Ugd2l0aCBJZGVuVHJ1c3QncyBUcnVzdElEIENlcnRpZmlj YXRlIFBvbGljeSBmb3VuZCBhdCBodHRwczovL3NlY3VyZS5pZGVudHJ1c3QuY29tL2NlcnRp ZmljYXRlcy9wb2xpY3kvdHMvaW5kZXguaHRtbDBFBgNVHR8EPjA8MDqgOKA2hjRodHRwOi8v dmFsaWRhdGlvbi5pZGVudHJ1c3QuY29tL2NybC90cnVzdGlkY2FhMTIuY3JsMB8GA1UdEQQY MBaBFGphbHRtYW5AYXVyaXN0b3IuY29tMB0GA1UdDgQWBBR7pHsvL4H5GdzNToI9e5BuzV19 bzAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwDQYJKoZIhvcNAQELBQADggEBAFlm JYk4Ff1v/n0foZkv661W4LCRtroBaVykOXetrDDOQNK2N6JdTa146uIZVgBeU+S/0DLvJBKY tkUHQ9ovjXJTsuCBmhIIw3YlHoFxbku0wHEpXMdFUHV3tUodFJJKF3MbC8j7dOMkag59/Mdz Sjszdvit0av9nTxWs/tRKKtSQQlxtH34TouIke2UgP/Nn901QLOrJYJmtjzVz8DW3IYVxfci SBHhbhJTdley5cuEzphELo5NR4gFjBNlxH7G57Hno9+EWILpx302FJMwTgodIBJbXLbPMHou xQbOL2anOTUMKO8oH0QdQHCtC7hpgoQa7UJYJxDBI+PRaQ/HObkwggaRMIIEeaADAgECAhEA +d5Wf8lNDHdw+WAbUtoVOzANBgkqhkiG9w0BAQsFADBKMQswCQYDVQQGEwJVUzESMBAGA1UE ChMJSWRlblRydXN0MScwJQYDVQQDEx5JZGVuVHJ1c3QgQ29tbWVyY2lhbCBSb290IENBIDEw HhcNMTUwMjE4MjIyNTE5WhcNMjMwMjE4MjIyNTE5WjA6MQswCQYDVQQGEwJVUzESMBAGA1UE ChMJSWRlblRydXN0MRcwFQYDVQQDEw5UcnVzdElEIENBIEExMjCCASIwDQYJKoZIhvcNAQEB BQADggEPADCCAQoCggEBANGRTTzPCic0kq5L6ZrUJWt5LE/n6tbPXPhGt2Egv7plJMoEpvVJ JDqGqDYymaAsd8Hn9ZMAuKUEFdlx5PgCkfu7jL5zgiMNnAFVD9PyrsuF+poqmlxhlQ06sFY2 hbhQkVVQ00KCNgUzKcBUIvjv04w+fhNPkwGW5M7Ae5K5OGFGwOoRck9GG6MUVKvTNkBw2/vN MOd29VGVTtR0tjH5PS5yDXss48Yl1P4hDStO2L4wTsW2P37QGD27//XGN8K6amWB6F2XOgff /PmlQjQOORT95PmLkwwvma5nj0AS0CVp8kv0K2RHV7GonllKpFDMT0CkxMQKwoj+tWEWJTiD KSsCAwEAAaOCAoAwggJ8MIGJBggrBgEFBQcBAQR9MHswMAYIKwYBBQUHMAGGJGh0dHA6Ly9j b21tZXJjaWFsLm9jc3AuaWRlbnRydXN0LmNvbTBHBggrBgEFBQcwAoY7aHR0cDovL3ZhbGlk YXRpb24uaWRlbnRydXN0LmNvbS9yb290cy9jb21tZXJjaWFscm9vdGNhMS5wN2MwHwYDVR0j BBgwFoAU7UQZwNPwBovupHu+QucmVMiONnYwDwYDVR0TAQH/BAUwAwEB/zCCASAGA1UdIASC ARcwggETMIIBDwYEVR0gADCCAQUwggEBBggrBgEFBQcCAjCB9DBFFj5odHRwczovL3NlY3Vy ZS5pZGVudHJ1c3QuY29tL2NlcnRpZmljYXRlcy9wb2xpY3kvdHMvaW5kZXguaHRtbDADAgEB GoGqVGhpcyBUcnVzdElEIENlcnRpZmljYXRlIGhhcyBiZWVuIGlzc3VlZCBpbiBhY2NvcmRh bmNlIHdpdGggSWRlblRydXN0J3MgVHJ1c3RJRCBDZXJ0aWZpY2F0ZSBQb2xpY3kgZm91bmQg YXQgaHR0cHM6Ly9zZWN1cmUuaWRlbnRydXN0LmNvbS9jZXJ0aWZpY2F0ZXMvcG9saWN5L3Rz L2luZGV4Lmh0bWwwSgYDVR0fBEMwQTA/oD2gO4Y5aHR0cDovL3ZhbGlkYXRpb24uaWRlbnRy dXN0LmNvbS9jcmwvY29tbWVyY2lhbHJvb3RjYTEuY3JsMB0GA1UdJQQWMBQGCCsGAQUFBwMC BggrBgEFBQcDBDAOBgNVHQ8BAf8EBAMCAYYwHQYDVR0OBBYEFKRz2u9pNYp1zKAZewgy+GuJ 5ELsMA0GCSqGSIb3DQEBCwUAA4ICAQAN4YKu0vv062MZfg+xMSNUXYKvHwvZIk+6H1pUmivy DI4I6A3wWzxlr83ZJm0oGIF6PBsbgKJ/fhyyIzb+vAYFJmyI8I/0mGlc+nIQNuV2XY8cypPo VJKgpnzp/7cECXkX8R4NyPtEn8KecbNdGBdEaG4a7AkZ3ujlJofZqYdHxN29tZPdDlZ8fR36 /mAFeCEq0wOtOOc0Eyhs29+9MIZYjyxaPoTS+l8xLcuYX3RWlirRyH6RPfeAi5kySOEhG1qu NHe06QIwpigjyFT6v/vRqoIBr7WpDOSt1VzXPVbSj1PcWBgkwyGKHlQUOuSbHbHcjOD8w8wH SDbL+L2he8hNN54doy1e1wJHKmnfb0uBAeISoxRbJnMMWvgAlH5FVrQWlgajeH/6NbYbBSRx ALuEOqEQepmJM6qz4oD2sxdq4GMN5adAdYEswkY/o0bRKyFXTD3mdqeRXce0jYQbWm7oapqS ZBccFvUgYOrB78tB6c1bxIgaQKRShtWR1zMM0JfqUfD9u8Fg7G5SVO0IG/GcxkSvZeRjhYcb TfqF2eAgprpyzLWmdr0mou3bv1Sq4OuBhmTQCnqxAXr4yVTRYHkp5lCvRgeJAme1OTVpVPth /O7HJ7VuEP9GOr6kCXCXmjB4P3UJ2oU0NqfoQdcSSSt9hliALnExTEjii20B2nSDojGCAxQw ggMQAgEBME4wOjELMAkGA1UEBhMCVVMxEjAQBgNVBAoTCUlkZW5UcnVzdDEXMBUGA1UEAxMO VHJ1c3RJRCBDQSBBMTICEEABbQHWpVVDfbC/HYRdhTowDQYJYIZIAWUDBAIBBQCgggGXMBgG CSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTIxMDMwOTAxMjE1Nlow LwYJKoZIhvcNAQkEMSIEIOIGIDoov+KMiPMYexxqVUkuN+4ELY69NzjAMF9dLn8eMF0GCSsG AQQBgjcQBDFQME4wOjELMAkGA1UEBhMCVVMxEjAQBgNVBAoTCUlkZW5UcnVzdDEXMBUGA1UE AxMOVHJ1c3RJRCBDQSBBMTICEEABbQHWpVVDfbC/HYRdhTowXwYLKoZIhvcNAQkQAgsxUKBO MDoxCzAJBgNVBAYTAlVTMRIwEAYDVQQKEwlJZGVuVHJ1c3QxFzAVBgNVBAMTDlRydXN0SUQg Q0EgQTEyAhBAAW0B1qVVQ32wvx2EXYU6MGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEq MAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwIC AUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwDQYJKoZIhvcNAQEBBQAEggEAOq1HXONhyGC0 KONIvDqbXe5Go3CJUxCeGaXEBVuUyOcr3Ereprh/Q+Mafy7atkDcYHo5qNM6alpIKRZQA+8a 3gUPQBO/5USqWQMTkYWDpmw8vQgThNWfnf30WwWFnpdm+K6bSUGiYEwkG1FWE8cfoFli6fgF dVAAVkNX0zBDHj4spve8zf5VHGbBMYjXhoAW6NG7thvaN+ftWBm5FMY4S2TZph5uk3nWUsj5 WAaKAnu7sBv3IQuJ14BdvThGOiTddmhSuXp7dUKdY7X0y9Y4Qc0G0sbun2BzuvSfHxMQPjLO NOU8pXiT2RE1TtkTsB9uoEtHpDTWONsOL1eCJ/lHuwAAAAAAAA== --------------ms080506090207080200070007-- From jsbillin@umich.edu Tue Mar 9 13:46:20 2021 From: jsbillin@umich.edu (Jonathan Billings) Date: Tue, 9 Mar 2021 08:46:20 -0500 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor In-Reply-To: <40352b81-652a-7ea4-dd30-30c87f5583e8@auristor.com> References: <20210309002053.GU56617@kduck.mit.edu> <40352b81-652a-7ea4-dd30-30c87f5583e8@auristor.com> Message-ID: --0000000000002ab34f05bd1ac752 Content-Type: text/plain; charset="UTF-8" Basically, this is what I'm running: # git describe --abbrev=4 openafs-stable-1_8_x openafs-stable-1_8_7-109-gb7bdd # rxdebug localhost 7001 -version Trying 127.0.0.1 (port 7001): AFS version: OpenAFS 1.8.7-109-gb7bdd 2021-03-08 mockbuild@ With this kmod and the latest RHEL7 kernel, this is the kernel backtrace: [ 170.503804] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [ 170.506260] IP: [] _raw_spin_lock+0xc/0x30 [ 170.507074] PGD 0 [ 170.507824] Oops: 0002 [#1] SMP [ 170.508596] Modules linked in: cts rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_11308(E) nf_log_ipv4 nf_log_common xt_LOG ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock cachefiles fscache sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper [ 170.512708] ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport binfmt_misc openafs(POE) auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptsas ata_piix scsi_transport_sas nfit drm crct10dif_pclmul mptscsih crct10dif_common libata serio_raw crc32c_intel libnvdimm mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [ 170.516344] CPU: 6 PID: 2782 Comm: llvmpipe-8 Kdump: loaded Tainted: P OE ------------ 3.10.0-1160.15.2.el7.x86_64 #1 [ 170.517353] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018 [ 170.518320] task: ffff9c0127fe0000 ti: ffff9c0556be0000 task.ti: ffff9c0556be0000 [ 170.519315] RIP: 0010:[] [] _raw_spin_lock+0xc/0x30 [ 170.520311] RSP: 0018:ffff9c0556be3710 EFLAGS: 00010246 [ 170.521317] RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000000 [ 170.522310] RDX: 0000000000000001 RSI: ffff9c013eb85d80 RDI: 0000000000000004 [ 170.523301] RBP: ffff9c0556be3730 R08: 000000000001f060 R09: ffff9c013efc67a0 [ 170.524292] R10: ffff9c0073e7aa80 R11: ffff9c013c7ae440 R12: ffff9c0556be3740 [ 170.525298] R13: 0000000000000000 R14: 0000000000000000 R15: ffff9c050c867810 [ 170.526303] FS: 00007f5b1dd69700(0000) GS:ffff9c0569300000(0000) knlGS:0000000000000000 [ 170.527315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 170.528324] CR2: 0000000000000004 CR3: 00000004da610000 CR4: 00000000001607e0 [ 170.529415] Call Trace: [ 170.530459] [] ? 0xffffffffc0af7d9c [ 170.531482] [] _ZdlPv+0x48088/0x484f0 [falcon_lsm_serviceable] [ 170.532523] [] cshook_security_file_permission+0x9b/0x8c0 [falcon_lsm_serviceable] [ 170.533579] [] cshook_security_file_open+0xe/0x10 [falcon_lsm_serviceable] [ 170.534637] [] pinnedhook_security_file_open+0x43/0x70 [falcon_lsm_pinned_11308] [ 170.535719] [] security_file_open+0x22/0x70 [ 170.536789] [] do_dentry_open+0xc9/0x2d0 [ 170.537850] [] vfs_open+0x5a/0xb0 [ 170.538876] [] dentry_open+0x49/0xc0 [ 170.540008] [] afs_linux_raw_open+0x8f/0x140 [openafs] [ 170.541094] [] osi_UFSOpen+0xb4/0x1a0 [openafs] [ 170.542140] [] DRead+0x341/0x520 [openafs] [ 170.543225] [] FindItem+0x5e/0x1f0 [openafs] [ 170.544306] [] afs_dir_Delete+0x33/0x1a0 [openafs] [ 170.545409] [] ? RXAFS_RemoveFile+0x67/0x120 [openafs] [ 170.546510] [] ? afs_LocalHero+0x11d/0x200 [openafs] [ 170.547616] [] afsremove+0x489/0x7d0 [openafs] [ 170.548705] [] afs_remunlink+0x24a/0x2f0 [openafs] [ 170.549798] [] afs_InactiveVCache+0x7d/0x80 [openafs] [ 170.550891] [] afs_dentry_iput+0x58/0x140 [openafs] [ 170.551931] [] __dentry_kill+0x13a/0x1d0 [ 170.553003] [] dput+0xb5/0x1a0 [ 170.554060] [] __fput+0x18d/0x230 [ 170.555118] [] ____fput+0xe/0x10 [ 170.556169] [] task_work_run+0xbb/0xe0 [ 170.557191] [] do_exit+0x2d4/0xa30 [ 170.558162] [] ? futex_wait+0x11f/0x280 [ 170.559126] [] ? x2apic_send_IPI_mask+0x13/0x20 [ 170.560048] [] do_group_exit+0x3f/0xa0 [ 170.560939] [] get_signal_to_deliver+0x1ce/0x5e0 [ 170.561802] [] do_signal+0x57/0x6f0 [ 170.562626] [] ? do_futex+0x106/0x5a0 [ 170.563420] [] do_notify_resume+0x72/0xc0 [ 170.564184] [] int_signal+0x12/0x17 [ 170.564878] Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 ea 18 ff ff 5d [ 170.566509] RIP [] _raw_spin_lock+0xc/0x30 [ 170.567274] RSP [ 170.568018] CR2: 0000000000000004 On Mon, Mar 8, 2021 at 8:22 PM Jeffrey E Altman wrote: > On 3/8/2021 7:20 PM, Benjamin Kaduk (kaduk@mit.edu) wrote: > > On Mon, Mar 08, 2021 at 07:35:19PM +0000, Martin Kelly wrote: > >> Below is the LKML LSM thread regarding this. Please let me know if you > have any other questions: > >> > >> https://www.spinics.net/lists/linux-security-module/msg39081.html > >> https://www.spinics.net/lists/linux-security-module/msg39083.html > > > > Thanks for spotting this thread and the quick follow-up. > > This is the same thread that Yadav discussed with the openafs-release > team on 11 Dec 2020. > > > I suspect that the changes at https://gerrit.openafs.org/#/c/13751/ are > > going to be relevant in this space, but without seeing the stack trace of > > the crash in question it's hard to be sure. Can you speak to whether > this > > is touching the "right" part of the code with respect to the crashes you > > were investigating? > > The suggested change was cherry-picked to openafs-stable-1_8_x as > https://gerrit.openafs.org/14082 and merged as > ee578e92d9f810d93659a9805d0c12084fe2bb95. > > As Jonathan wrote to IRC OpenAFS: > > > (4:53:15 PM) billings: I built openafs from the latest commit in > > 1_8_x and crowdstrike still panics, so it doesnt look like any > > merged commits there fix my issue. > > Martin's e-mail describes the call pattern: > > > - A process exits, calling task_exit(). > > I think Martin meant do_exit(). > > > - exit_fs() is called, setting current->fs = NULL. > > task_struct field struct fs_struct *fs; > > > - Next, exit_task_work() is called, > > exit_task_work() calls task_work_run() which flushes any pending works. > > > which calls fput(). > > which must have been called by a pending work. > > > - In response to the fput(), the filesystem opens a file > > disk cache > > > to update some metadata, calling dentry_open(). > > dentry_open() in turn will trigger a call to any configured LSM. > If task_struct.fs is NULL, Kaboom!!! > > Jeffrey Altman > -- Jonathan Billings (he/his) College of Engineering - CAEN - Linux Support --0000000000002ab34f05bd1ac752 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Basically, this is what I'm running:
# git describe --abbrev=3D4 openafs-stable-1_8_x
openafs-s= table-1_8_7-109-gb7bdd
# rxdebug localhost 7001 -version
Tryin= g 127.0.0.1 (port 7001):
AFS version: OpenAFS 1.8.7-109-gb7bdd 2021-03-0= 8 mockbuild@

With this kmod and the latest RHEL7 k= ernel, this is the kernel backtrace:


[ =C2=A0170.503804] BUG: unable to handle kern= el NULL pointer dereference at 0000000000000004
[ =C2=A0170.506260] IP: = [<ffffffff8238a6ec>] _raw_spin_lock+0xc/0x30
[ =C2=A0170.507074] P= GD 0
[ =C2=A0170.507824] Oops: 0002 [#1] SMP
[ =C2=A0170.508596] Mo= dules linked in: cts rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fal= con_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_p= inned_11308(E) nf_log_ipv4 nf_log_common xt_LOG ip6t_rpfilter ip6t_REJECT n= f_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_br= oute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ip= v6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_= ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security ipta= ble_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6= table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock cache= files fscache sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_b= alloon aesni_intel lrw gf128mul glue_helper
[ =C2=A0170.512708] =C2=A0ab= lk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport bin= fmt_misc openafs(POE) auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sr_mod= cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm= _kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptsas ata_pi= ix scsi_transport_sas nfit drm crct10dif_pclmul mptscsih crct10dif_common l= ibata serio_raw crc32c_intel libnvdimm mptbase vmxnet3 drm_panel_orientatio= n_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
[ =C2=A0170.= 516344] CPU: 6 PID: 2782 Comm: llvmpipe-8 Kdump: loaded Tainted: P =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 OE =C2=A0------------ =C2=A0 3.10.0-1160.15.2.e= l7.x86_64 #1
[ =C2=A0170.517353] Hardware name: VMware, Inc. VMware Virt= ual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[ = =C2=A0170.518320] task: ffff9c0127fe0000 ti: ffff9c0556be0000 task.ti: ffff= 9c0556be0000
[ =C2=A0170.519315] RIP: 0010:[<ffffffff8238a6ec>] = =C2=A0[<ffffffff8238a6ec>] _raw_spin_lock+0xc/0x30
[ =C2=A0170.520= 311] RSP: 0018:ffff9c0556be3710 =C2=A0EFLAGS: 00010246
[ =C2=A0170.52131= 7] RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000000
[ = =C2=A0170.522310] RDX: 0000000000000001 RSI: ffff9c013eb85d80 RDI: 00000000= 00000004
[ =C2=A0170.523301] RBP: ffff9c0556be3730 R08: 000000000001f060= R09: ffff9c013efc67a0
[ =C2=A0170.524292] R10: ffff9c0073e7aa80 R11: ff= ff9c013c7ae440 R12: ffff9c0556be3740
[ =C2=A0170.525298] R13: 0000000000= 000000 R14: 0000000000000000 R15: ffff9c050c867810
[ =C2=A0170.526303] F= S: =C2=A000007f5b1dd69700(0000) GS:ffff9c0569300000(0000) knlGS:00000000000= 00000
[ =C2=A0170.527315] CS: =C2=A00010 DS: 0000 ES: 0000 CR0: 00000000= 80050033
[ =C2=A0170.528324] CR2: 0000000000000004 CR3: 00000004da610000= CR4: 00000000001607e0
[ =C2=A0170.529415] Call Trace:
[ =C2=A0170.53= 0459] =C2=A0[<ffffffffc0af7d9d>] ? 0xffffffffc0af7d9c
[ =C2=A0170.= 531482] =C2=A0[<ffffffffc0b56ab8>] _ZdlPv+0x48088/0x484f0 [falcon_lsm= _serviceable]
[ =C2=A0170.532523] =C2=A0[<ffffffffc0b5725b>] cshoo= k_security_file_permission+0x9b/0x8c0 [falcon_lsm_serviceable]
[ =C2=A01= 70.533579] =C2=A0[<ffffffffc0b5ea9e>] cshook_security_file_open+0xe/0= x10 [falcon_lsm_serviceable]
[ =C2=A0170.534637] =C2=A0[<ffffffffc0ae= 4873>] pinnedhook_security_file_open+0x43/0x70 [falcon_lsm_pinned_11308]=
[ =C2=A0170.535719] =C2=A0[<ffffffff81f08f12>] security_file_open= +0x22/0x70
[ =C2=A0170.536789] =C2=A0[<ffffffff81e4b339>] do_dentr= y_open+0xc9/0x2d0
[ =C2=A0170.537850] =C2=A0[<ffffffff81e4b5da>] v= fs_open+0x5a/0xb0
[ =C2=A0170.538876] =C2=A0[<ffffffff81e4b679>] d= entry_open+0x49/0xc0
[ =C2=A0170.540008] =C2=A0[<ffffffffc06cdb0f>= ] afs_linux_raw_open+0x8f/0x140 [openafs]
[ =C2=A0170.541094] =C2=A0[<= ;ffffffffc06cdc74>] osi_UFSOpen+0xb4/0x1a0 [openafs]
[ =C2=A0170.5421= 40] =C2=A0[<ffffffffc06567f1>] DRead+0x341/0x520 [openafs]
[ =C2= =A0170.543225] =C2=A0[<ffffffffc0669a7e>] FindItem+0x5e/0x1f0 [openaf= s]
[ =C2=A0170.544306] =C2=A0[<ffffffffc066a183>] afs_dir_Delete+0= x33/0x1a0 [openafs]
[ =C2=A0170.545409] =C2=A0[<ffffffffc06a6d17>]= ? RXAFS_RemoveFile+0x67/0x120 [openafs]
[ =C2=A0170.546510] =C2=A0[<= ffffffffc068e1ad>] ? afs_LocalHero+0x11d/0x200 [openafs]
[ =C2=A0170.= 547616] =C2=A0[<ffffffffc069abc9>] afsremove+0x489/0x7d0 [openafs][ =C2=A0170.548705] =C2=A0[<ffffffffc069beca>] afs_remunlink+0x24a/0= x2f0 [openafs]
[ =C2=A0170.549798] =C2=A0[<ffffffffc068489d>] afs_= InactiveVCache+0x7d/0x80 [openafs]
[ =C2=A0170.550891] =C2=A0[<ffffff= ffc06d28e8>] afs_dentry_iput+0x58/0x140 [openafs]
[ =C2=A0170.551931]= =C2=A0[<ffffffff81e670ca>] __dentry_kill+0x13a/0x1d0
[ =C2=A0170.= 553003] =C2=A0[<ffffffff81e67785>] dput+0xb5/0x1a0
[ =C2=A0170.554= 060] =C2=A0[<ffffffff81e500cd>] __fput+0x18d/0x230
[ =C2=A0170.555= 118] =C2=A0[<ffffffff81e5025e>] ____fput+0xe/0x10
[ =C2=A0170.5561= 69] =C2=A0[<ffffffff81cc294b>] task_work_run+0xbb/0xe0
[ =C2=A0170= .557191] =C2=A0[<ffffffff81ca1894>] do_exit+0x2d4/0xa30
[ =C2=A017= 0.558162] =C2=A0[<ffffffff81d1317f>] ? futex_wait+0x11f/0x280
[ = =C2=A0170.559126] =C2=A0[<ffffffff81c63d03>] ? x2apic_send_IPI_mask+0= x13/0x20
[ =C2=A0170.560048] =C2=A0[<ffffffff81ca206f>] do_group_e= xit+0x3f/0xa0
[ =C2=A0170.560939] =C2=A0[<ffffffff81cb323e>] get_s= ignal_to_deliver+0x1ce/0x5e0
[ =C2=A0170.561802] =C2=A0[<ffffffff81c2= c527>] do_signal+0x57/0x6f0
[ =C2=A0170.562626] =C2=A0[<ffffffff81= d14f46>] ? do_futex+0x106/0x5a0
[ =C2=A0170.563420] =C2=A0[<ffffff= ff81c2cc32>] do_notify_resume+0x72/0xc0
[ =C2=A0170.564184] =C2=A0[&l= t;ffffffff823952ef>] int_signal+0x12/0x17
[ =C2=A0170.564878] Code: 5= d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 0= 0 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85= c0 75 01 c3 55 89 c6 48 89 e5 e8 ea 18 ff ff 5d
[ =C2=A0170.566509] RI= P =C2=A0[<ffffffff8238a6ec>] _raw_spin_lock+0xc/0x30
[ =C2=A0170.5= 67274] =C2=A0RSP <ffff9c0556be3710>
[ =C2=A0170.568018] CR2: 00000= 00000000004


On Mon, Mar 8, 2021 at 8:22 PM J= effrey E Altman <jaltman@auristo= r.com> wrote:
On 3/8/2021 7:20 PM, Benjamin Kaduk (kaduk@mit.edu) wrote:
> On Mon, Mar 08, 2021 at 07:35:19PM +0000, Martin Kelly wrote:
>> Below is the LKML LSM thread regarding this. Please let me know if= you have any other questions:
>>
>> https://www.spinics.net/li= sts/linux-security-module/msg39081.html
>> https://www.spinics.net/li= sts/linux-security-module/msg39083.html
>
> Thanks for spotting this thread and the quick follow-up.

This is the same thread that Yadav discussed with the openafs-release
team on 11 Dec 2020.

> I suspect that the changes at https://gerrit.openafs.org/#= /c/13751/ are
> going to be relevant in this space, but without seeing the stack trace= of
> the crash in question it's hard to be sure.=C2=A0 Can you speak to= whether this
> is touching the "right" part of the code with respect to the= crashes you
> were investigating?

The suggested change was cherry-picked to openafs-stable-1_8_x as
https://gerrit.openafs.org/14082 and merged as
ee578e92d9f810d93659a9805d0c12084fe2bb95.

As Jonathan wrote to IRC OpenAFS:

=C2=A0> (4:53:15 PM) billings: I built openafs from the latest commit in=
=C2=A0> 1_8_x and crowdstrike still panics, so it doesnt look like any =C2=A0> merged commits there fix my issue.

Martin's e-mail describes the call pattern:

=C2=A0> - A process exits, calling task_exit().

I think Martin meant do_exit().

=C2=A0> - exit_fs() is called, setting current->fs =3D NULL.

task_struct field struct fs_struct *fs;

=C2=A0> - Next, exit_task_work() is called,

exit_task_work() calls task_work_run() which flushes any pending works.

=C2=A0> which calls fput().

which must have been called by a pending work.

=C2=A0> - In response to the fput(), the filesystem opens a file

disk cache

=C2=A0>=C2=A0 =C2=A0to update some metadata, calling dentry_open().

dentry_open() in turn will trigger a call to any configured LSM.
If task_struct.fs is NULL, Kaboom!!!

Jeffrey Altman


--
Jonathan Billings <jsbillin@umich.edu> (he/his)
= College of Engineering - CAEN - Linux Support
--0000000000002ab34f05bd1ac752-- From martin.kelly@crowdstrike.com Tue Mar 9 18:28:38 2021 From: martin.kelly@crowdstrike.com (Martin Kelly) Date: Tue, 9 Mar 2021 18:28:38 +0000 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor Message-ID: <7f47744595b34e8383c719309420584c@crowdstrike.com> ------=_NextPart_000_002A_01D714CE.968C6200 Content-Type: multipart/alternative; boundary="----=_NextPart_001_002B_01D714CE.968C6200" ------=_NextPart_001_002B_01D714CE.968C6200 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit This is exactly the issue referenced in the LKML thread, and the one that Jeffrey Altman analyzed, so it seems the issue has not yet been fixed. Unfortunately, I don't think the patches dealing with credentials will fix this because dentry_open calls security_file_open regardless of credentials, and security_file_open expects process context. I believe the way to fix this is that OpenAFS needs to open this file in a way that does not go through the LSM, because there is no process context here. From: Jonathan Billings Sent: Tuesday, March 9, 2021 5:46 AM To: openafs-info@openafs.org Cc: Martin Kelly Subject: [External] Re: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor Basically, this is what I'm running: # git describe --abbrev=4 openafs-stable-1_8_x openafs-stable-1_8_7-109-gb7bdd # rxdebug localhost 7001 -version Trying 127.0.0.1 [127.0.0.1] (port 7001): AFS version: OpenAFS 1.8.7-109-gb7bdd 2021-03-08 mockbuild@ With this kmod and the latest RHEL7 kernel, this is the kernel backtrace: [ 170.503804] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [ 170.506260] IP: [] _raw_spin_lock+0xc/0x30 [ 170.507074] PGD 0 [ 170.507824] Oops: 0002 [#1] SMP [ 170.508596] Modules linked in: cts rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_11308(E) nf_log_ipv4 nf_log_common xt_LOG ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock cachefiles fscache sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper [ 170.512708] ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport binfmt_misc openafs(POE) auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptsas ata_piix scsi_transport_sas nfit drm crct10dif_pclmul mptscsih crct10dif_common libata serio_raw crc32c_intel libnvdimm mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [ 170.516344] CPU: 6 PID: 2782 Comm: llvmpipe-8 Kdump: loaded Tainted: P OE ------------ 3.10.0-1160.15.2.el7.x86_64 #1 [ 170.517353] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018 [ 170.518320] task: ffff9c0127fe0000 ti: ffff9c0556be0000 task.ti: ffff9c0556be0000 [ 170.519315] RIP: 0010:[] [] _raw_spin_lock+0xc/0x30 [ 170.520311] RSP: 0018:ffff9c0556be3710 EFLAGS: 00010246 [ 170.521317] RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000000 [ 170.522310] RDX: 0000000000000001 RSI: ffff9c013eb85d80 RDI: 0000000000000004 [ 170.523301] RBP: ffff9c0556be3730 R08: 000000000001f060 R09: ffff9c013efc67a0 [ 170.524292] R10: ffff9c0073e7aa80 R11: ffff9c013c7ae440 R12: ffff9c0556be3740 [ 170.525298] R13: 0000000000000000 R14: 0000000000000000 R15: ffff9c050c867810 [ 170.526303] FS: 00007f5b1dd69700(0000) GS:ffff9c0569300000(0000) knlGS:0000000000000000 [ 170.527315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 170.528324] CR2: 0000000000000004 CR3: 00000004da610000 CR4: 00000000001607e0 [ 170.529415] Call Trace: [ 170.530459] [] ? 0xffffffffc0af7d9c [ 170.531482] [] _ZdlPv+0x48088/0x484f0 [falcon_lsm_serviceable] [ 170.532523] [] cshook_security_file_permission+0x9b/0x8c0 [falcon_lsm_serviceable] [ 170.533579] [] cshook_security_file_open+0xe/0x10 [falcon_lsm_serviceable] [ 170.534637] [] pinnedhook_security_file_open+0x43/0x70 [falcon_lsm_pinned_11308] [ 170.535719] [] security_file_open+0x22/0x70 [ 170.536789] [] do_dentry_open+0xc9/0x2d0 [ 170.537850] [] vfs_open+0x5a/0xb0 [ 170.538876] [] dentry_open+0x49/0xc0 [ 170.540008] [] afs_linux_raw_open+0x8f/0x140 [openafs] [ 170.541094] [] osi_UFSOpen+0xb4/0x1a0 [openafs] [ 170.542140] [] DRead+0x341/0x520 [openafs] [ 170.543225] [] FindItem+0x5e/0x1f0 [openafs] [ 170.544306] [] afs_dir_Delete+0x33/0x1a0 [openafs] [ 170.545409] [] ? RXAFS_RemoveFile+0x67/0x120 [openafs] [ 170.546510] [] ? afs_LocalHero+0x11d/0x200 [openafs] [ 170.547616] [] afsremove+0x489/0x7d0 [openafs] [ 170.548705] [] afs_remunlink+0x24a/0x2f0 [openafs] [ 170.549798] [] afs_InactiveVCache+0x7d/0x80 [openafs] [ 170.550891] [] afs_dentry_iput+0x58/0x140 [openafs] [ 170.551931] [] __dentry_kill+0x13a/0x1d0 [ 170.553003] [] dput+0xb5/0x1a0 [ 170.554060] [] __fput+0x18d/0x230 [ 170.555118] [] ____fput+0xe/0x10 [ 170.556169] [] task_work_run+0xbb/0xe0 [ 170.557191] [] do_exit+0x2d4/0xa30 [ 170.558162] [] ? futex_wait+0x11f/0x280 [ 170.559126] [] ? x2apic_send_IPI_mask+0x13/0x20 [ 170.560048] [] do_group_exit+0x3f/0xa0 [ 170.560939] [] get_signal_to_deliver+0x1ce/0x5e0 [ 170.561802] [] do_signal+0x57/0x6f0 [ 170.562626] [] ? do_futex+0x106/0x5a0 [ 170.563420] [] do_notify_resume+0x72/0xc0 [ 170.564184] [] int_signal+0x12/0x17 [ 170.564878] Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 ea 18 ff ff 5d [ 170.566509] RIP [] _raw_spin_lock+0xc/0x30 [ 170.567274] RSP [ 170.568018] CR2: 0000000000000004 On Mon, Mar 8, 2021 at 8:22 PM Jeffrey E Altman > wrote: On 3/8/2021 7:20 PM, Benjamin Kaduk (kaduk@mit.edu ) wrote: > On Mon, Mar 08, 2021 at 07:35:19PM +0000, Martin Kelly wrote: >> Below is the LKML LSM thread regarding this. Please let me know if you have any other questions: >> >> https://www.spinics.net/lists/linux-security-module/msg39081.html [spinics.net] >> https://www.spinics.net/lists/linux-security-module/msg39083.html [spinics.net] > > Thanks for spotting this thread and the quick follow-up. This is the same thread that Yadav discussed with the openafs-release team on 11 Dec 2020. > I suspect that the changes at https://gerrit.openafs.org/#/c/13751/ [gerrit.openafs.org] are > going to be relevant in this space, but without seeing the stack trace of > the crash in question it's hard to be sure. Can you speak to whether this > is touching the "right" part of the code with respect to the crashes you > were investigating? The suggested change was cherry-picked to openafs-stable-1_8_x as https://gerrit.openafs.org/14082 [gerrit.openafs.org] and merged as ee578e92d9f810d93659a9805d0c12084fe2bb95. As Jonathan wrote to IRC OpenAFS: > (4:53:15 PM) billings: I built openafs from the latest commit in > 1_8_x and crowdstrike still panics, so it doesnt look like any > merged commits there fix my issue. Martin's e-mail describes the call pattern: > - A process exits, calling task_exit(). I think Martin meant do_exit(). > - exit_fs() is called, setting current->fs = NULL. task_struct field struct fs_struct *fs; > - Next, exit_task_work() is called, exit_task_work() calls task_work_run() which flushes any pending works. > which calls fput(). which must have been called by a pending work. > - In response to the fput(), the filesystem opens a file disk cache > to update some metadata, calling dentry_open(). dentry_open() in turn will trigger a call to any configured LSM. If task_struct.fs is NULL, Kaboom!!! Jeffrey Altman -- Jonathan Billings > (he/his) College of Engineering - CAEN - Linux Support ------=_NextPart_001_002B_01D714CE.968C6200 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

This is exactly the issue = referenced in the LKML thread, and the one that Jeffrey Altman analyzed, = so it seems the issue has not yet been fixed. Unfortunately, I don't = think the patches dealing with credentials will fix this because = dentry_open calls security_file_open regardless of credentials, and = security_file_open expects process context.

 

I believe = the way to fix this is that OpenAFS needs to open this file in a way = that does not go through the LSM, because there is no process context = here.

 

From: Jonathan Billings = <jsbillin@umich.edu>
Sent: Tuesday, March 9, 2021 5:46 = AM
To: openafs-info@openafs.org
Cc: Martin Kelly = <martin.kelly@crowdstrike.com>
Subject: [External] Re: = [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike = falcon-sensor

 

Basically, this is what I'm = running:

 

# = git describe --abbrev=3D4 openafs-stable-1_8_x =
openafs-stable-1_8_7-109-gb7bdd

# rxdebug localhost 7001 -version
Trying 127.0.0.1 [127.0.0.1] (port 7001):
AFS version: = OpenAFS 1.8.7-109-gb7bdd 2021-03-08 = mockbuild@

 

With this kmod and the latest RHEL7 kernel, this is = the kernel backtrace:

 

 

[  170.503804] BUG: unable to = handle kernel NULL pointer dereference at 0000000000000004
[ =  170.506260] IP: [<ffffffff8238a6ec>] = _raw_spin_lock+0xc/0x30
[  170.507074] PGD 0
[ =  170.507824] Oops: 0002 [#1] SMP
[  170.508596] Modules = linked in: cts rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace = falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) = falcon_lsm_pinned_11308(E) nf_log_ipv4 nf_log_common xt_LOG = ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 = xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat = nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle = ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 = nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security = iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter = ebtables ip6table_filter ip6_tables iptable_filter = vmw_vsock_vmci_transport vsock cachefiles fscache sb_edac ppdev iosf_mbi = crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul = glue_helper
[  170.512708]  ablk_helper cryptd pcspkr = joydev sg vmw_vmci i2c_piix4 parport_pc parport binfmt_misc openafs(POE) = auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic = pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper = syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptsas ata_piix = scsi_transport_sas nfit drm crct10dif_pclmul mptscsih crct10dif_common = libata serio_raw crc32c_intel libnvdimm mptbase vmxnet3 = drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log = dm_mod fuse
[  170.516344] CPU: 6 PID: 2782 Comm: llvmpipe-8 = Kdump: loaded Tainted: P           OE =  ------------   3.10.0-1160.15.2.el7.x86_64 #1
[ =  170.517353] Hardware name: VMware, Inc. VMware Virtual = Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[ =  170.518320] task: ffff9c0127fe0000 ti: ffff9c0556be0000 task.ti: = ffff9c0556be0000
[  170.519315] RIP: = 0010:[<ffffffff8238a6ec>]  [<ffffffff8238a6ec>] = _raw_spin_lock+0xc/0x30
[  170.520311] RSP: = 0018:ffff9c0556be3710  EFLAGS: 00010246
[  170.521317] RAX: = 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000000
[ =  170.522310] RDX: 0000000000000001 RSI: ffff9c013eb85d80 RDI: = 0000000000000004
[  170.523301] RBP: ffff9c0556be3730 R08: = 000000000001f060 R09: ffff9c013efc67a0
[  170.524292] R10: = ffff9c0073e7aa80 R11: ffff9c013c7ae440 R12: ffff9c0556be3740
[ =  170.525298] R13: 0000000000000000 R14: 0000000000000000 R15: = ffff9c050c867810
[  170.526303] FS:  00007f5b1dd69700(0000) = GS:ffff9c0569300000(0000) knlGS:0000000000000000
[  170.527315] = CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ =  170.528324] CR2: 0000000000000004 CR3: 00000004da610000 CR4: = 00000000001607e0
[  170.529415] Call Trace:
[ =  170.530459]  [<ffffffffc0af7d9d>] ? = 0xffffffffc0af7d9c
[  170.531482] =  [<ffffffffc0b56ab8>] _ZdlPv+0x48088/0x484f0 = [falcon_lsm_serviceable]
[  170.532523] =  [<ffffffffc0b5725b>] = cshook_security_file_permission+0x9b/0x8c0 [falcon_lsm_serviceable]
[ =  170.533579]  [<ffffffffc0b5ea9e>] = cshook_security_file_open+0xe/0x10 [falcon_lsm_serviceable]
[ =  170.534637]  [<ffffffffc0ae4873>] = pinnedhook_security_file_open+0x43/0x70 [falcon_lsm_pinned_11308]
[ =  170.535719]  [<ffffffff81f08f12>] = security_file_open+0x22/0x70
[  170.536789] =  [<ffffffff81e4b339>] do_dentry_open+0xc9/0x2d0
[ =  170.537850]  [<ffffffff81e4b5da>] = vfs_open+0x5a/0xb0
[  170.538876] =  [<ffffffff81e4b679>] dentry_open+0x49/0xc0
[ =  170.540008]  [<ffffffffc06cdb0f>] = afs_linux_raw_open+0x8f/0x140 [openafs]
[  170.541094] =  [<ffffffffc06cdc74>] osi_UFSOpen+0xb4/0x1a0 [openafs]
[ =  170.542140]  [<ffffffffc06567f1>] DRead+0x341/0x520 = [openafs]
[  170.543225]  [<ffffffffc0669a7e>] = FindItem+0x5e/0x1f0 [openafs]
[  170.544306] =  [<ffffffffc066a183>] afs_dir_Delete+0x33/0x1a0 = [openafs]
[  170.545409]  [<ffffffffc06a6d17>] ? = RXAFS_RemoveFile+0x67/0x120 [openafs]
[  170.546510] =  [<ffffffffc068e1ad>] ? afs_LocalHero+0x11d/0x200 = [openafs]
[  170.547616]  [<ffffffffc069abc9>] = afsremove+0x489/0x7d0 [openafs]
[  170.548705] =  [<ffffffffc069beca>] afs_remunlink+0x24a/0x2f0 = [openafs]
[  170.549798]  [<ffffffffc068489d>] = afs_InactiveVCache+0x7d/0x80 [openafs]
[  170.550891] =  [<ffffffffc06d28e8>] afs_dentry_iput+0x58/0x140 = [openafs]
[  170.551931]  [<ffffffff81e670ca>] = __dentry_kill+0x13a/0x1d0
[  170.553003] =  [<ffffffff81e67785>] dput+0xb5/0x1a0
[  170.554060] =  [<ffffffff81e500cd>] __fput+0x18d/0x230
[ =  170.555118]  [<ffffffff81e5025e>] = ____fput+0xe/0x10
[  170.556169] =  [<ffffffff81cc294b>] task_work_run+0xbb/0xe0
[ =  170.557191]  [<ffffffff81ca1894>] = do_exit+0x2d4/0xa30
[  170.558162] =  [<ffffffff81d1317f>] ? futex_wait+0x11f/0x280
[ =  170.559126]  [<ffffffff81c63d03>] ? = x2apic_send_IPI_mask+0x13/0x20
[  170.560048] =  [<ffffffff81ca206f>] do_group_exit+0x3f/0xa0
[ =  170.560939]  [<ffffffff81cb323e>] = get_signal_to_deliver+0x1ce/0x5e0
[  170.561802] =  [<ffffffff81c2c527>] do_signal+0x57/0x6f0
[ =  170.562626]  [<ffffffff81d14f46>] ? = do_futex+0x106/0x5a0
[  170.563420] =  [<ffffffff81c2cc32>] do_notify_resume+0x72/0xc0
[ =  170.564184]  [<ffffffff823952ef>] = int_signal+0x12/0x17
[  170.564878] Code: 5d c3 0f 1f 44 00 00 = 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 = 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 = 55 89 c6 48 89 e5 e8 ea 18 ff ff 5d
[  170.566509] RIP =  [<ffffffff8238a6ec>] _raw_spin_lock+0xc/0x30
[ =  170.567274]  RSP <ffff9c0556be3710>
[ =  170.568018] CR2: = 0000000000000004

 

 

On = Mon, Mar 8, 2021 at 8:22 PM Jeffrey E Altman <jaltman@auristor.com> = wrote:

On 3/8/2021 7:20 PM, Benjamin Kaduk (kaduk@mit.edu) = wrote:
> On Mon, Mar 08, 2021 at 07:35:19PM +0000, Martin Kelly = wrote:
>> Below is the LKML LSM thread regarding this. Please = let me know if you have any other questions:
>>
>> https://www.spinics.net/lists/linux-security-module/msg= 39081.html [spinics.net]
>> https://www.spinics.net/lists/linux-security-module/msg= 39083.html [spinics.net]
>
> Thanks for spotting this = thread and the quick follow-up.

This is the same thread that = Yadav discussed with the openafs-release
team on 11 Dec = 2020.

> I suspect that the changes at https://gerrit.openafs.org/#/c/13751/ = [gerrit.openafs.org] are
> going to be relevant in this space, = but without seeing the stack trace of
> the crash in question it's = hard to be sure.  Can you speak to whether this
> is touching = the "right" part of the code with respect to the crashes = you
> were investigating?

The suggested change was = cherry-picked to openafs-stable-1_8_x as
https://gerrit.openafs.org/14082 = [gerrit.openafs.org] and merged as =
ee578e92d9f810d93659a9805d0c12084fe2bb95.

As Jonathan wrote = to IRC OpenAFS:

 > (4:53:15 PM) billings: I built openafs = from the latest commit in
 > 1_8_x and crowdstrike still = panics, so it doesnt look like any
 > merged commits there = fix my issue.

Martin's e-mail describes the call = pattern:

 > - A process exits, calling = task_exit().

I think Martin meant do_exit().

 > - = exit_fs() is called, setting current->fs =3D NULL.

task_struct = field struct fs_struct *fs;

 > - Next, exit_task_work() = is called,

exit_task_work() calls task_work_run() which flushes = any pending works.

 > which calls fput().

which = must have been called by a pending work.

 > - In response = to the fput(), the filesystem opens a file

disk = cache

 >   to update some metadata, calling = dentry_open().

dentry_open() in turn will trigger a call to any = configured LSM.
If task_struct.fs is NULL, Kaboom!!!

Jeffrey = Altman



--

Jonathan Billings <jsbillin@umich.edu> (he/his)
College of = Engineering - CAEN - Linux = Support

------=_NextPart_001_002B_01D714CE.968C6200-- ------=_NextPart_000_002A_01D714CE.968C6200 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIOnDCCBCow ggMSoAMCAQICBDhj3vgwDQYJKoZIhvcNAQEFBQAwgbQxFDASBgNVBAoTC0VudHJ1c3QubmV0MUAw PgYDVQQLFDd3d3cuZW50cnVzdC5uZXQvQ1BTXzIwNDggaW5jb3JwLiBieSByZWYuIChsaW1pdHMg bGlhYi4pMSUwIwYDVQQLExwoYykgMTk5OSBFbnRydXN0Lm5ldCBMaW1pdGVkMTMwMQYDVQQDEypF bnRydXN0Lm5ldCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAoMjA0OCkwHhcNOTkxMjI0MTc1MDUx WhcNMjkwNzI0MTQxNTEyWjCBtDEUMBIGA1UEChMLRW50cnVzdC5uZXQxQDA+BgNVBAsUN3d3dy5l bnRydXN0Lm5ldC9DUFNfMjA0OCBpbmNvcnAuIGJ5IHJlZi4gKGxpbWl0cyBsaWFiLikxJTAjBgNV BAsTHChjKSAxOTk5IEVudHJ1c3QubmV0IExpbWl0ZWQxMzAxBgNVBAMTKkVudHJ1c3QubmV0IENl cnRpZmljYXRpb24gQXV0aG9yaXR5ICgyMDQ4KTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC ggEBAK1NS6kShrLqoyAHFRZkKitL0b8LSk2O7YB2pWe3eEDAc0LIaMDbUyvdXrh2mDWTixqdfBM6 Dh9btx7P5SQUHrGBqY19uMxrSwPxAgzcq6VAJAB/dJShnQgps4gL9Yd3nVXN5MN+12pkq4UUhpVb lzJQbz3IumYM4/y9uEnBdolJGf3AqL2Jo2cvxp+8cRlguC3pLMmQdmZ7lOKveNZlU1081pyyzykD +S+kULLUSM4FMlWK/bJkTA7kmAd123/fuQhVYIUwKfl7SKRphuM1Px6GXXp6Fb3vAI4VIlQXAJAm k7wOSWiRv/hH052VQsEOTd9vJs/DGCFiZkNw1tXAB+ECAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEG MA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFFXkgdERgL7YibkIozH5oSQJFrlwMA0GCSqGSIb3 DQEBBQUAA4IBAQA7m49WmzDnU5l8enmnTZfXGZWQ+wYfyjN8RmOPlmYk+kAbISfK5nJz8k/+MZn9 yAxMaFPGgIITmPq2rdpdPfHObvYVEZSCDO4/la8Rqw/XL94fA49XLB7Ju5oaRJXrGE+mH819VxAv mwQJWoS1btgdOuHWntFseV55HBTF49BMkztlPO3fPb6m5ZUaw7UZw71eW7v/I+9oGcsSkydcAy1v MNAethqs3lr30aqoJ6b+eYHEeZkzV7oSsKngQmyTylbe/m2ECwiLfo3q15ghxvPnPHkvXpzRTBWN 4ewiN8yaQwuX3ICQjbNnm29ICBVWz7/xK3xemnbpWZDFfIM1EWVRMIIFFTCCA/2gAwIBAgIRAK8c BLKsjP+bAAAAAFHOGOMwDQYJKoZIhvcNAQELBQAwgbQxFDASBgNVBAoTC0VudHJ1c3QubmV0MUAw PgYDVQQLFDd3d3cuZW50cnVzdC5uZXQvQ1BTXzIwNDggaW5jb3JwLiBieSByZWYuIChsaW1pdHMg bGlhYi4pMSUwIwYDVQQLExwoYykgMTk5OSBFbnRydXN0Lm5ldCBMaW1pdGVkMTMwMQYDVQQDEypF bnRydXN0Lm5ldCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAoMjA0OCkwHhcNMjAwNzI5MTU0ODMw WhcNMjkwNjI5MTYxODMwWjCBpTELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4x OTA3BgNVBAsTMHd3dy5lbnRydXN0Lm5ldC9DUFMgaXMgaW5jb3Jwb3JhdGVkIGJ5IHJlZmVyZW5j ZTEfMB0GA1UECxMWKGMpIDIwMTAgRW50cnVzdCwgSW5jLjEiMCAGA1UEAxMZRW50cnVzdCBDbGFz cyAyIENsaWVudCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMQyjULQnhmdW5Ba EEy1EAAhuQdI3q5ugNb/FFAG6HWva0aO56VPrcOMsPp74BmR/fBjrXFJ86gcH6s0GSBOS1TpAJO+ cAgx3olTrFe8JO8qj0LU9+qVJV0UdtLNpxL6G7K0XGFAvV/dV5tEVdjFiRk8ZT256NSlLcIs0+qD MaIIPF5ZrhIuKgqMXvOzMa4KrX7ssEkJ/KcuIh5oZDSdFuOmPQMxQBb3lPZLGTTJl+YinEjeZKCD C1gFmMQiRokF/aO+9klMYQMWpPgKmRziwMZ+aQIyV5ADrwCUobnczq/v9HwYzjALyof41V8fWVHY iwu5OMZYwlN82ibU2/K9kM0CAwEAAaOCAS0wggEpMA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAU BggrBgEFBQcDBAYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAzBggrBgEFBQcBAQQnMCUw IwYIKwYBBQUHMAGGF2h0dHA6Ly9vY3NwLmVudHJ1c3QubmV0MDIGA1UdHwQrMCkwJ6AloCOGIWh0 dHA6Ly9jcmwuZW50cnVzdC5uZXQvMjA0OGNhLmNybDA7BgNVHSAENDAyMDAGBFUdIAAwKDAmBggr BgEFBQcCARYaaHR0cDovL3d3dy5lbnRydXN0Lm5ldC9ycGEwHQYDVR0OBBYEFAmRpbrp8i4qdd/N fv53yvLea5skMB8GA1UdIwQYMBaAFFXkgdERgL7YibkIozH5oSQJFrlwMA0GCSqGSIb3DQEBCwUA A4IBAQA/vekQdfNCp9FsgSahRiBXEiQVWrIMCH/dR7k/QpOkCq9MEe7MazD0tCyE3goXkPl4NK6u JkV2BTUkg8CTc5lPpXJxY7QJiBHLbG7vlJXVSTfPoQDwDUsUUUb0aHGy/mChNw8l/O8gWjPGqYfJ 6lL212lIls5azxCb9rcBwzohpchDwISdA/jFNAiHy4sKg1yqIyvp/7jep0kObTIVgTDIJ/TA/s8a dcyHu7oRoYJlUAWf80WSh6BFuBnnX/hGClvM2F1rFpFMFZVq4+T83gZ09mxU3cQl8GkW1uoOP1m+ AWL5YJ8dQLMx9xCcL/mKRGbYYAJOMRCx9peO/iCDvU1KMIIFUTCCBDmgAwIBAgIQDJP99GX/o7gA AAAATDnPWjANBgkqhkiG9w0BAQsFADCBpTELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3Qs IEluYy4xOTA3BgNVBAsTMHd3dy5lbnRydXN0Lm5ldC9DUFMgaXMgaW5jb3Jwb3JhdGVkIGJ5IHJl ZmVyZW5jZTEfMB0GA1UECxMWKGMpIDIwMTAgRW50cnVzdCwgSW5jLjEiMCAGA1UEAxMZRW50cnVz dCBDbGFzcyAyIENsaWVudCBDQTAeFw0yMDEwMTQyMjA2NDRaFw0yMjEwMTQyMjM2NDNaMIGRMQsw CQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTEPMA0GA1UEBxMGSXJ2aW5lMRowGAYDVQQK ExFDcm93ZFN0cmlrZSwgSW5jLjFAMBMGA1UEAxMMTWFydGluIEtlbGx5MCkGCSqGSIb3DQEJARYc bWFydGluLmtlbGx5QGNyb3dkc3RyaWtlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC ggEBAMT46ZuzJ/RhZGH9+qht+LO1VOIYXTlKXOYwfashRkS/Bbro15Km8ARz41X1CQ/9AmjPgPFL wFJXQlYZZybTsGl1o/R4PBmZUZ9AwfqHPXUrM3WKP3A4s5mVffPilfe6lTuSoipTzq8MoOaz2rK/ 4kffBpTqDL3MvYvJX9hva2UJ4RvhQ9kS/6NAf/2mobNyVufM2NAAdUyQlUV4A8Hgdz+GklWtDpzR jdZjGs7Cb+cylSVG/X8ntvq2idrWOUu8/2gH6aV1ZiR3G1U3I8HaYAdZK6ycXNfvdFSrePv9TKCk 7ud/0v5RwTcenut579PJ9J1ILBxxrphRD1MsulKWP4MCAwEAAaOCAY0wggGJMA4GA1UdDwEB/wQE AwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwQgYDVR0gBDswOTA3BgtghkgBhvps CgEEAjAoMCYGCCsGAQUFBwIBFhpodHRwOi8vd3d3LmVudHJ1c3QubmV0L3JwYTBqBggrBgEFBQcB AQReMFwwIwYIKwYBBQUHMAGGF2h0dHA6Ly9vY3NwLmVudHJ1c3QubmV0MDUGCCsGAQUFBzAChilo dHRwOi8vYWlhLmVudHJ1c3QubmV0LzIwNDhjbGFzczJzaGEyLmNlcjA0BgNVHR8ELTArMCmgJ6Al hiNodHRwOi8vY3JsLmVudHJ1c3QubmV0L2NsYXNzMmNhLmNybDAnBgNVHREEIDAegRxtYXJ0aW4u a2VsbHlAY3Jvd2RzdHJpa2UuY29tMB8GA1UdIwQYMBaAFAmRpbrp8i4qdd/Nfv53yvLea5skMB0G A1UdDgQWBBSlEgDqZESbubhJcEKhhhhKnl7OtDAJBgNVHRMEAjAAMA0GCSqGSIb3DQEBCwUAA4IB AQCoPnFPAX5tAP+aIr17tIgq1L2nP1F2pTeHgXd7ETwtUF1rSfq22mtKsDslQ5cRxy6l8LzH9j1h 2QmWK6dd+lSOUR/KaDSnXyhXsCHBX9455sWu6+n6Umrol/kOoRB3eFAht8wvXyIkNWoGf2M+p1kC r7NYzgTDauE40ch1P15zLw47j9BODeX1BRhzlal0mrdEbLgZmcCdThe9S3+X1WOucSUIA54mFl46 uuZWlpl2lSKCrvWZn5RcGWqrdbM27SsNipli0sz0H+jPKUtFfr3DNsPQTANJsPU3u9k9A5HropDq sZRj+3j+2z9Yiyi01g1k7bjc6LHmxXa0YjhQeMA8MYIEdzCCBHMCAQEwgbowgaUxCzAJBgNVBAYT AlVTMRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMTkwNwYDVQQLEzB3d3cuZW50cnVzdC5uZXQvQ1BT IGlzIGluY29ycG9yYXRlZCBieSByZWZlcmVuY2UxHzAdBgNVBAsTFihjKSAyMDEwIEVudHJ1c3Qs IEluYy4xIjAgBgNVBAMTGUVudHJ1c3QgQ2xhc3MgMiBDbGllbnQgQ0ECEAyT/fRl/6O4AAAAAEw5 z1owCQYFKw4DAhoFAKCCApEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUx DxcNMjEwMzA5MTgyNTU1WjAjBgkqhkiG9w0BCQQxFgQU4GSZ22a8ExdGmiMlgytG+wR/MzswgZMG CSqGSIb3DQEJDzGBhTCBgjALBglghkgBZQMEASowCwYJYIZIAWUDBAEWMAoGCCqGSIb3DQMHMAsG CWCGSAFlAwQBAjAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAhowCwYJYIZI AWUDBAIDMAsGCWCGSAFlAwQCAjALBglghkgBZQMEAgEwgcsGCSsGAQQBgjcQBDGBvTCBujCBpTEL MAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xOTA3BgNVBAsTMHd3dy5lbnRydXN0 Lm5ldC9DUFMgaXMgaW5jb3Jwb3JhdGVkIGJ5IHJlZmVyZW5jZTEfMB0GA1UECxMWKGMpIDIwMTAg RW50cnVzdCwgSW5jLjEiMCAGA1UEAxMZRW50cnVzdCBDbGFzcyAyIENsaWVudCBDQQIQDJP99GX/ o7gAAAAATDnPWjCBzQYLKoZIhvcNAQkQAgsxgb2ggbowgaUxCzAJBgNVBAYTAlVTMRYwFAYDVQQK Ew1FbnRydXN0LCBJbmMuMTkwNwYDVQQLEzB3d3cuZW50cnVzdC5uZXQvQ1BTIGlzIGluY29ycG9y YXRlZCBieSByZWZlcmVuY2UxHzAdBgNVBAsTFihjKSAyMDEwIEVudHJ1c3QsIEluYy4xIjAgBgNV BAMTGUVudHJ1c3QgQ2xhc3MgMiBDbGllbnQgQ0ECEAyT/fRl/6O4AAAAAEw5z1owDQYJKoZIhvcN AQEBBQAEggEABgPiRFnd+M/QkapnRPdpPLRIptSIevqY22gph0xq+0JUfLsJuDNLzGsKpDEYqcqP jRJYJvIBU+/hMbPx9oxv+tY/IG3IMCO4GrrKcET/uebjtYaABseJ3lFTB6mmWBzoYXMqNh+hUiMm zPgoR8H7i2bMGWtW/tsx9SkIHggN0ohXj2goXMTPyia0V5tm5Na2Ea81s8QflAPYV7TwPJ7srAjG 1yQFgTEMnYY0P96k541Ebzp7xn9KA69yE7S5sZiAPxrZDjMotIm85FkDToVw0yDeqcy74we64mhc sfexDodm4oFmkYewtK/kSec5gwQs4EQts74DllgzO7XCPxcGCAAAAAAAAA== ------=_NextPart_000_002A_01D714CE.968C6200-- From cgrundman@gmail.com Thu Mar 11 20:02:59 2021 From: cgrundman@gmail.com (Chaskiel Grundman) Date: Thu, 11 Mar 2021 15:02:59 -0500 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor In-Reply-To: References: <20210309002053.GU56617@kduck.mit.edu> <40352b81-652a-7ea4-dd30-30c87f5583e8@auristor.com> Message-ID: --000000000000ebe15a05bd484584 Content-Type: text/plain; charset="UTF-8" The bad news is, override_creds isn't going to fix this, because current->fs isn't part of the creds. It's still going to be null (h/t jhutz) The less bad news is, I think this will only affect closing a deleted file. Other operations should not trigger cache I/O after a flush. (exit_files, which occurs before exit_fs, called flush and fput, but fput these days doesn't do the work immediately, it punts it to task_work). So if we punt the unlink of a sillyrenamed file to a kthread, we should be able to avoid this problem. But we may have other issues. If crowdstrike is validating dentry_open against current->fs->root, it might deny cache I/O done on behalf of a chroot'd (or namespaced) process. We can't solve that without punting ALL cache I/O to a kthread. --000000000000ebe15a05bd484584 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
The bad news is, override_creds isn't going to fix thi= s, because current->fs isn't part of the creds. It's still going= to be null (h/t jhutz)
The less bad news is, I think this will only af= fect closing a deleted file. Other operations should not trigger cache I/O = after a flush. (exit_files, which occurs before exit_fs, called flush and f= put, but fput these days doesn't do the work immediately, it punts it t= o task_work). So if we punt the unlink of a sillyrenamed file to a kthread,= we should be able to avoid this problem.

But we m= ay have other issues.
If crowdstrike is validating dentry_open ag= ainst current->fs->root, it might deny cache I/O done on behalf of a = chroot'd (or namespaced) process. We can't solve that without punti= ng ALL cache I/O to a kthread.
--000000000000ebe15a05bd484584-- From martin.kelly@crowdstrike.com Thu Mar 11 22:11:00 2021 From: martin.kelly@crowdstrike.com (Martin Kelly) Date: Thu, 11 Mar 2021 22:11:00 +0000 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor Message-ID: <13a618e4c63c4150b3e6af5cda033a7e@crowdstrike.com> PiBUaGUgYmFkIG5ld3MgaXMsIG92ZXJyaWRlX2NyZWRzIGlzbid0IGdvaW5nIHRvIGZpeCB0aGlz LCBiZWNhdXNlIGN1cnJlbnQtPmZzIGlzbid0IHBhcnQgb2YgdGhlIGNyZWRzLiBJdCdzIHN0aWxs IGdvaW5nIHRvIGJlIG51bGwgKGgvdCBqaHV0eikNCj4gVGhlIGxlc3MgYmFkIG5ld3MgaXMsIEkg dGhpbmsgdGhpcyB3aWxsIG9ubHkgYWZmZWN0IGNsb3NpbmcgYSBkZWxldGVkIGZpbGUuIE90aGVy IG9wZXJhdGlvbnMgc2hvdWxkIG5vdCB0cmlnZ2VyIGNhY2hlIEkvTyBhZnRlciBhIGZsdXNoLiAo ZXhpdF9maWxlcywgd2hpY2ggb2NjdXJzIGJlZm9yZSBleGl0X2ZzLCBjYWxsZWQgZmx1c2ggYW5k IGZwdXQsIGJ1dCBmcHV0IHRoZXNlIGRheXMgZG9lc24ndCBkbyB0aGUgd29yayBpbW1lZGlhdGVs eSwgaXQgcHVudHMgaXQgdG8gdGFza193b3JrKS4gU28gaWYgd2UgcHVudCB0aGUgdW5saW5rIG9m IGEgc2lsbHlyZW5hbWVkIGZpbGUgdG8gYSBrdGhyZWFkLCB3ZSBzaG91bGQgYmUgYWJsZSB0byBh dm9pZCB0aGlzIHByb2JsZW0uDQo+DQo+IEJ1dCB3ZSBtYXkgaGF2ZSBvdGhlciBpc3N1ZXMuDQo+ IElmIGNyb3dkc3RyaWtlIGlzIHZhbGlkYXRpbmcgZGVudHJ5X29wZW4gYWdhaW5zdCBjdXJyZW50 LT5mcy0+cm9vdCwgaXQgbWlnaHQgZGVueSBjYWNoZSBJL08gZG9uZSBvbiBiZWhhbGYgb2YgYSBj aHJvb3QnZCAob3IgbmFtZXNwYWNlZCkgcHJvY2Vzcy4gV2UgY2FuJ3Qgc29sdmUgdGhhdCB3aXRo b3V0IHB1bnRpbmcgQUxMIGNhY2hlIEkvTyB0byBhIGt0aHJlYWQuDQoNCkkgdGhpbmsgdGhlIGtl eSB0aGluZyB0byBwcmV2ZW50IHRoaXMgY3Jhc2ggaXMganVzdCB0byBhdm9pZCB0cmlnZ2VyaW5n IGFuIExTTSBob29rIGR1cmluZyBwcm9jZXNzIGNvbnRleHQuIElmIHlvdSBzZWUgb3RoZXIgaXNz dWVzIHdpdGggZmFsY29uIGJ1dCBhcmUgZm9sbG93aW5nIGFsbCB0aGUgdXBzdHJlYW0gTGludXgg a2VybmVsIGRyaXZlciBndWlkYW5jZSwgcGxlYXNlIGNvbnRhY3QgbWUgKG9yIENyb3dkU3RyaWtl IG1vcmUgZ2VuZXJhbGx5KSBhbmQgd2UnbGwgYmUgaGFwcHkgdG8gd29yayB3aXRoIHlvdS4NCg== From martin.kelly@crowdstrike.com Thu Mar 11 22:30:04 2021 From: martin.kelly@crowdstrike.com (Martin Kelly) Date: Thu, 11 Mar 2021 22:30:04 +0000 Subject: [OpenAFS] OpenAFS 1.8.7 on Linux systems running Crowdstrike falcon-sensor In-Reply-To: References: <20210309002053.GU56617@kduck.mit.edu> <40352b81-652a-7ea4-dd30-30c87f5583e8@auristor.com> Message-ID: ------=_NextPart_000_002F_01D71682.BD763840 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit > The bad news is, override_creds isn't going to fix this, because current->fs > isn't part of the creds. It's still going to be null (h/t jhutz) > The less bad news is, I think this will only affect closing a deleted file. > Other operations should not trigger cache I/O after a flush. (exit_files, > which occurs before exit_fs, called flush and fput, but fput these days > doesn't do the work immediately, it punts it to task_work). So if we punt > the unlink of a sillyrenamed file to a kthread, we should be able to avoid > this problem. > > But we may have other issues. > If crowdstrike is validating dentry_open against current->fs->root, it might > deny cache I/O done on behalf of a chroot'd (or namespaced) process. We > can't solve that without punting ALL cache I/O to a kthread. [Resending, as originally this sent with uuencoded, which the mail list archives don't handle well. Hopefully it's OK this time.] I think the key thing to prevent this crash is just to avoid triggering an LSM hook during process context. If you see other issues with falcon but are following all the upstream Linux kernel driver guidance, please contact me (or CrowdStrike more generally) and we'll be happy to work with you. ------=_NextPart_000_002F_01D71682.BD763840 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIOnDCCBCow ggMSoAMCAQICBDhj3vgwDQYJKoZIhvcNAQEFBQAwgbQxFDASBgNVBAoTC0VudHJ1c3QubmV0MUAw PgYDVQQLFDd3d3cuZW50cnVzdC5uZXQvQ1BTXzIwNDggaW5jb3JwLiBieSByZWYuIChsaW1pdHMg bGlhYi4pMSUwIwYDVQQLExwoYykgMTk5OSBFbnRydXN0Lm5ldCBMaW1pdGVkMTMwMQYDVQQDEypF bnRydXN0Lm5ldCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAoMjA0OCkwHhcNOTkxMjI0MTc1MDUx WhcNMjkwNzI0MTQxNTEyWjCBtDEUMBIGA1UEChMLRW50cnVzdC5uZXQxQDA+BgNVBAsUN3d3dy5l bnRydXN0Lm5ldC9DUFNfMjA0OCBpbmNvcnAuIGJ5IHJlZi4gKGxpbWl0cyBsaWFiLikxJTAjBgNV BAsTHChjKSAxOTk5IEVudHJ1c3QubmV0IExpbWl0ZWQxMzAxBgNVBAMTKkVudHJ1c3QubmV0IENl cnRpZmljYXRpb24gQXV0aG9yaXR5ICgyMDQ4KTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC ggEBAK1NS6kShrLqoyAHFRZkKitL0b8LSk2O7YB2pWe3eEDAc0LIaMDbUyvdXrh2mDWTixqdfBM6 Dh9btx7P5SQUHrGBqY19uMxrSwPxAgzcq6VAJAB/dJShnQgps4gL9Yd3nVXN5MN+12pkq4UUhpVb lzJQbz3IumYM4/y9uEnBdolJGf3AqL2Jo2cvxp+8cRlguC3pLMmQdmZ7lOKveNZlU1081pyyzykD +S+kULLUSM4FMlWK/bJkTA7kmAd123/fuQhVYIUwKfl7SKRphuM1Px6GXXp6Fb3vAI4VIlQXAJAm k7wOSWiRv/hH052VQsEOTd9vJs/DGCFiZkNw1tXAB+ECAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEG MA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFFXkgdERgL7YibkIozH5oSQJFrlwMA0GCSqGSIb3 DQEBBQUAA4IBAQA7m49WmzDnU5l8enmnTZfXGZWQ+wYfyjN8RmOPlmYk+kAbISfK5nJz8k/+MZn9 yAxMaFPGgIITmPq2rdpdPfHObvYVEZSCDO4/la8Rqw/XL94fA49XLB7Ju5oaRJXrGE+mH819VxAv mwQJWoS1btgdOuHWntFseV55HBTF49BMkztlPO3fPb6m5ZUaw7UZw71eW7v/I+9oGcsSkydcAy1v MNAethqs3lr30aqoJ6b+eYHEeZkzV7oSsKngQmyTylbe/m2ECwiLfo3q15ghxvPnPHkvXpzRTBWN 4ewiN8yaQwuX3ICQjbNnm29ICBVWz7/xK3xemnbpWZDFfIM1EWVRMIIFFTCCA/2gAwIBAgIRAK8c BLKsjP+bAAAAAFHOGOMwDQYJKoZIhvcNAQELBQAwgbQxFDASBgNVBAoTC0VudHJ1c3QubmV0MUAw PgYDVQQLFDd3d3cuZW50cnVzdC5uZXQvQ1BTXzIwNDggaW5jb3JwLiBieSByZWYuIChsaW1pdHMg bGlhYi4pMSUwIwYDVQQLExwoYykgMTk5OSBFbnRydXN0Lm5ldCBMaW1pdGVkMTMwMQYDVQQDEypF bnRydXN0Lm5ldCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eSAoMjA0OCkwHhcNMjAwNzI5MTU0ODMw WhcNMjkwNjI5MTYxODMwWjCBpTELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4x OTA3BgNVBAsTMHd3dy5lbnRydXN0Lm5ldC9DUFMgaXMgaW5jb3Jwb3JhdGVkIGJ5IHJlZmVyZW5j ZTEfMB0GA1UECxMWKGMpIDIwMTAgRW50cnVzdCwgSW5jLjEiMCAGA1UEAxMZRW50cnVzdCBDbGFz cyAyIENsaWVudCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMQyjULQnhmdW5Ba EEy1EAAhuQdI3q5ugNb/FFAG6HWva0aO56VPrcOMsPp74BmR/fBjrXFJ86gcH6s0GSBOS1TpAJO+ cAgx3olTrFe8JO8qj0LU9+qVJV0UdtLNpxL6G7K0XGFAvV/dV5tEVdjFiRk8ZT256NSlLcIs0+qD MaIIPF5ZrhIuKgqMXvOzMa4KrX7ssEkJ/KcuIh5oZDSdFuOmPQMxQBb3lPZLGTTJl+YinEjeZKCD C1gFmMQiRokF/aO+9klMYQMWpPgKmRziwMZ+aQIyV5ADrwCUobnczq/v9HwYzjALyof41V8fWVHY iwu5OMZYwlN82ibU2/K9kM0CAwEAAaOCAS0wggEpMA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAU BggrBgEFBQcDBAYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAzBggrBgEFBQcBAQQnMCUw IwYIKwYBBQUHMAGGF2h0dHA6Ly9vY3NwLmVudHJ1c3QubmV0MDIGA1UdHwQrMCkwJ6AloCOGIWh0 dHA6Ly9jcmwuZW50cnVzdC5uZXQvMjA0OGNhLmNybDA7BgNVHSAENDAyMDAGBFUdIAAwKDAmBggr BgEFBQcCARYaaHR0cDovL3d3dy5lbnRydXN0Lm5ldC9ycGEwHQYDVR0OBBYEFAmRpbrp8i4qdd/N fv53yvLea5skMB8GA1UdIwQYMBaAFFXkgdERgL7YibkIozH5oSQJFrlwMA0GCSqGSIb3DQEBCwUA A4IBAQA/vekQdfNCp9FsgSahRiBXEiQVWrIMCH/dR7k/QpOkCq9MEe7MazD0tCyE3goXkPl4NK6u JkV2BTUkg8CTc5lPpXJxY7QJiBHLbG7vlJXVSTfPoQDwDUsUUUb0aHGy/mChNw8l/O8gWjPGqYfJ 6lL212lIls5azxCb9rcBwzohpchDwISdA/jFNAiHy4sKg1yqIyvp/7jep0kObTIVgTDIJ/TA/s8a dcyHu7oRoYJlUAWf80WSh6BFuBnnX/hGClvM2F1rFpFMFZVq4+T83gZ09mxU3cQl8GkW1uoOP1m+ AWL5YJ8dQLMx9xCcL/mKRGbYYAJOMRCx9peO/iCDvU1KMIIFUTCCBDmgAwIBAgIQDJP99GX/o7gA AAAATDnPWjANBgkqhkiG9w0BAQsFADCBpTELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3Qs IEluYy4xOTA3BgNVBAsTMHd3dy5lbnRydXN0Lm5ldC9DUFMgaXMgaW5jb3Jwb3JhdGVkIGJ5IHJl ZmVyZW5jZTEfMB0GA1UECxMWKGMpIDIwMTAgRW50cnVzdCwgSW5jLjEiMCAGA1UEAxMZRW50cnVz dCBDbGFzcyAyIENsaWVudCBDQTAeFw0yMDEwMTQyMjA2NDRaFw0yMjEwMTQyMjM2NDNaMIGRMQsw CQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTEPMA0GA1UEBxMGSXJ2aW5lMRowGAYDVQQK ExFDcm93ZFN0cmlrZSwgSW5jLjFAMBMGA1UEAxMMTWFydGluIEtlbGx5MCkGCSqGSIb3DQEJARYc bWFydGluLmtlbGx5QGNyb3dkc3RyaWtlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC ggEBAMT46ZuzJ/RhZGH9+qht+LO1VOIYXTlKXOYwfashRkS/Bbro15Km8ARz41X1CQ/9AmjPgPFL wFJXQlYZZybTsGl1o/R4PBmZUZ9AwfqHPXUrM3WKP3A4s5mVffPilfe6lTuSoipTzq8MoOaz2rK/ 4kffBpTqDL3MvYvJX9hva2UJ4RvhQ9kS/6NAf/2mobNyVufM2NAAdUyQlUV4A8Hgdz+GklWtDpzR jdZjGs7Cb+cylSVG/X8ntvq2idrWOUu8/2gH6aV1ZiR3G1U3I8HaYAdZK6ycXNfvdFSrePv9TKCk 7ud/0v5RwTcenut579PJ9J1ILBxxrphRD1MsulKWP4MCAwEAAaOCAY0wggGJMA4GA1UdDwEB/wQE AwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwQgYDVR0gBDswOTA3BgtghkgBhvps CgEEAjAoMCYGCCsGAQUFBwIBFhpodHRwOi8vd3d3LmVudHJ1c3QubmV0L3JwYTBqBggrBgEFBQcB AQReMFwwIwYIKwYBBQUHMAGGF2h0dHA6Ly9vY3NwLmVudHJ1c3QubmV0MDUGCCsGAQUFBzAChilo dHRwOi8vYWlhLmVudHJ1c3QubmV0LzIwNDhjbGFzczJzaGEyLmNlcjA0BgNVHR8ELTArMCmgJ6Al hiNodHRwOi8vY3JsLmVudHJ1c3QubmV0L2NsYXNzMmNhLmNybDAnBgNVHREEIDAegRxtYXJ0aW4u a2VsbHlAY3Jvd2RzdHJpa2UuY29tMB8GA1UdIwQYMBaAFAmRpbrp8i4qdd/Nfv53yvLea5skMB0G A1UdDgQWBBSlEgDqZESbubhJcEKhhhhKnl7OtDAJBgNVHRMEAjAAMA0GCSqGSIb3DQEBCwUAA4IB AQCoPnFPAX5tAP+aIr17tIgq1L2nP1F2pTeHgXd7ETwtUF1rSfq22mtKsDslQ5cRxy6l8LzH9j1h 2QmWK6dd+lSOUR/KaDSnXyhXsCHBX9455sWu6+n6Umrol/kOoRB3eFAht8wvXyIkNWoGf2M+p1kC r7NYzgTDauE40ch1P15zLw47j9BODeX1BRhzlal0mrdEbLgZmcCdThe9S3+X1WOucSUIA54mFl46 uuZWlpl2lSKCrvWZn5RcGWqrdbM27SsNipli0sz0H+jPKUtFfr3DNsPQTANJsPU3u9k9A5HropDq sZRj+3j+2z9Yiyi01g1k7bjc6LHmxXa0YjhQeMA8MYIEdzCCBHMCAQEwgbowgaUxCzAJBgNVBAYT AlVTMRYwFAYDVQQKEw1FbnRydXN0LCBJbmMuMTkwNwYDVQQLEzB3d3cuZW50cnVzdC5uZXQvQ1BT IGlzIGluY29ycG9yYXRlZCBieSByZWZlcmVuY2UxHzAdBgNVBAsTFihjKSAyMDEwIEVudHJ1c3Qs IEluYy4xIjAgBgNVBAMTGUVudHJ1c3QgQ2xhc3MgMiBDbGllbnQgQ0ECEAyT/fRl/6O4AAAAAEw5 z1owCQYFKw4DAhoFAKCCApEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUx DxcNMjEwMzExMjIyODAxWjAjBgkqhkiG9w0BCQQxFgQUfm/K5UftoKMmSElGBg93EHp2GeQwgZMG CSqGSIb3DQEJDzGBhTCBgjALBglghkgBZQMEASowCwYJYIZIAWUDBAEWMAoGCCqGSIb3DQMHMAsG CWCGSAFlAwQBAjAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAhowCwYJYIZI AWUDBAIDMAsGCWCGSAFlAwQCAjALBglghkgBZQMEAgEwgcsGCSsGAQQBgjcQBDGBvTCBujCBpTEL MAkGA1UEBhMCVVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xOTA3BgNVBAsTMHd3dy5lbnRydXN0 Lm5ldC9DUFMgaXMgaW5jb3Jwb3JhdGVkIGJ5IHJlZmVyZW5jZTEfMB0GA1UECxMWKGMpIDIwMTAg RW50cnVzdCwgSW5jLjEiMCAGA1UEAxMZRW50cnVzdCBDbGFzcyAyIENsaWVudCBDQQIQDJP99GX/ o7gAAAAATDnPWjCBzQYLKoZIhvcNAQkQAgsxgb2ggbowgaUxCzAJBgNVBAYTAlVTMRYwFAYDVQQK Ew1FbnRydXN0LCBJbmMuMTkwNwYDVQQLEzB3d3cuZW50cnVzdC5uZXQvQ1BTIGlzIGluY29ycG9y YXRlZCBieSByZWZlcmVuY2UxHzAdBgNVBAsTFihjKSAyMDEwIEVudHJ1c3QsIEluYy4xIjAgBgNV BAMTGUVudHJ1c3QgQ2xhc3MgMiBDbGllbnQgQ0ECEAyT/fRl/6O4AAAAAEw5z1owDQYJKoZIhvcN AQEBBQAEggEAr6LDYb4nyp8osvOBa/rDQ+a0dXki5qK1eMbtv8ADVzafV11VWeskGPWr+ZcdqzR0 oSiTEqNhwugyrAyhs9NrKs2IZ9F0wMCPe/8DNMjGznBdSqneoqqsPeuY8/E6IIgeDpDomuB9zrnp YSqgIBdkvUUeol6ep7ZbJEhFQuVl0ZIdxjS/1UPINC6pDjiUAyErRGYpB0dNnCV5lEgFyeyYda/r uV7LriqroXiDI791znQMnaNbuJkAninElgJKEaV3llFu2/NRPupHjbcgg8K6nUtuoo/NrvND2rBf SAl4EHnwXCpczGBpzRmwrEG8v9TxoWUZJHEoXtDwdOKlUTAAUwAAAAAAAA== ------=_NextPart_000_002F_01D71682.BD763840-- From iwienand@redhat.com Mon Mar 29 05:23:42 2021 From: iwienand@redhat.com (Ian Wienand) Date: Mon, 29 Mar 2021 15:23:42 +1100 Subject: [OpenAFS] Occasional "VLDB: no permission access for call" Message-ID: Hello, A new thing I've noticed after we have upgraded everything to 1.8.6 is like the following: ~$ vos remove -server afs01.dfw.openstack.org -id 536870937 -partition a Could not lock VLDB entry for the volume 536870937 VLDB: no permission access for call VLDB: no permission access for call Error in vos remove command. VLDB: no permission access for call $ vos remove -server afs01.dfw.openstack.org -id 536870937 -partition a Volume 536870937 on partition /vicepa server afs01.dfw.openstack.org deleted The first failed, I literally pressed "up and enter" to try again and it then worked. In a similar fashion I can run a loop of "vos release" and have random volumes fail to release, then work just fine on another try. None of our afsdb servers have anything in VLLog or PtLog (nothing at all since they started). All my tokens are granted and valid, etc. -- I mean it works one time and not another doing nothing at all in between. I have strace's of a failing and good "vos remove" attempt, if it would help. Any ideas? Any suggestions what else to trace to see what's going on here? -i From kaduk@mit.edu Mon Mar 29 06:27:35 2021 From: kaduk@mit.edu (Benjamin Kaduk) Date: Sun, 28 Mar 2021 22:27:35 -0700 Subject: [OpenAFS] Occasional "VLDB: no permission access for call" In-Reply-To: References: Message-ID: <20210329052735.GU79563@kduck.mit.edu> On Mon, Mar 29, 2021 at 03:23:42PM +1100, Ian Wienand wrote: > Hello, > > A new thing I've noticed after we have upgraded everything to 1.8.6 is > like the following: What were you upgrading from? (And I assume you mean 1.8.7?) > ~$ vos remove -server afs01.dfw.openstack.org -id 536870937 -partition a > > Could not lock VLDB entry for the volume 536870937 > VLDB: no permission access for call > > > VLDB: no permission access for call > Error in vos remove command. > VLDB: no permission access for call This only happens if the code thinks that your authenticated identity is not in the UserList on the server in question. I find it unlikely that the view of the file on disk is changing, so assume that something goes wrong on the way to that. (It looks like the SetLock operation is what fails rather than the Delete itself, for what it's worth.) > $ vos remove -server afs01.dfw.openstack.org -id 536870937 -partition a > Volume 536870937 on partition /vicepa server afs01.dfw.openstack.org deleted > > The first failed, I literally pressed "up and enter" to try again and > it then worked. In a similar fashion I can run a loop of "vos > release" and have random volumes fail to release, then work just fine > on another try. > > None of our afsdb servers have anything in VLLog or PtLog (nothing at > all since they started). All my tokens are granted and valid, etc. -- > I mean it works one time and not another doing nothing at all in > between. > > I have strace's of a failing and good "vos remove" attempt, if it > would help. > > Any ideas? Any suggestions what else to trace to see what's going on > here? strace on the client is unlikely to help. A packet capture of the rx challenge/response might, and an audit log from the vlserver might as well. -Ben From cgrundman@gmail.com Mon Mar 29 14:11:28 2021 From: cgrundman@gmail.com (Chaskiel Grundman) Date: Mon, 29 Mar 2021 09:11:28 -0400 Subject: [OpenAFS] Any xstat tools? Message-ID: --000000000000513d9905beac9f3c Content-Type: text/plain; charset="UTF-8" Does anyone have tools for capturing, aggregating, analyzing or displaying info from xstats? Not so much operation timing, though that may also be helpful. --000000000000513d9905beac9f3c Content-Type: text/html; charset="UTF-8"
Does anyone have tools for capturing, aggregating, analyzing or displaying info from xstats? Not so much operation timing, though that may also be helpful.
--000000000000513d9905beac9f3c-- From mvitale@sinenomine.net Mon Mar 29 16:08:32 2021 From: mvitale@sinenomine.net (Mark Vitale) Date: Mon, 29 Mar 2021 15:08:32 +0000 Subject: [OpenAFS] Any xstat tools? In-Reply-To: References: Message-ID: <758480ED-ADDD-4997-957D-7AE790AAC1A8@sinenomine.net> Chaskiel, > On Mar 29, 2021, at 9:11 AM, Chaskiel Grundman wrot= e: >=20 > Does anyone have tools for capturing, aggregating, analyzing or displayin= g info from xstats? Not so much operation timing, though that may also be h= elpful. There are some patches under review on gerrit to make OpenAFS metrics more = machine-readable: https://gerrit.openafs.org/#/c/14359/ xstat: Add the xstat_fs_test -forma= t option https://gerrit.openafs.org/#/c/14358/ rxdebug: Add rxdebug -raw option I am using these patches to collect OpenAFS cell metrics into collectd and = display charts with graphite. collectd works fine for the scale I need, but it does seem to drop an obser= vation once in a while. It can handle aggregation automatically if you configure it to do that. graphite works fine too, but it is a little long in the tooth; if I were do= ing this today I would try grafana or some other alternatives. =20 Other sites have had success using Splunk to display their charts. I've used Splunk to troubleshoot OpenAFS performance problems and it was=20 extremely useful. But I have not set it up myself, so I can't give any guidance on how difficult it is to set up compared to graphite or grafana. I am currently pulling my stats remotely by running all my xstats and rxdeb= ug scripts from a central collector machine, but this is not ideal.=20 I would recommend that if you are doing this at any kind of scale, you should try to collect the stats locally (e.g. xstat_fs_test localhost) = on each OpenAFS machine - either via cron job or a bos bnode script - and=20 then push the stats to your collector. Be aware that xstat_fs_test uses a new ephemeral port on each invocation; this in turn results in a new peer on the fileserver. It's not horrible ov= erhead, even if you are collecting every 60 seconds; but if you are concerned about= it, you can reduce this impact by allowing xstat_fs_test to run continuously. That is, intead of invoking it periodically with the -onceonly option, invo= ke it once and specify a -frequency and a -period. In this way it will reuse the same ephemeral port over and over, thus creating only a single peer on = the fileserver. The rx_peer issue is moot for the rxdebug command; although it also uses a new ephemeral port for each invocation, the rxdebug packets are handled b= y the rx stack directly, without requiring an rx_call, rx_connection, or rx_p= eer. Regards, -- Mark Vitale mvitale@sinenomine.net From jaltman@auristor.com Mon Mar 29 16:29:27 2021 From: jaltman@auristor.com (Jeffrey E Altman) Date: Mon, 29 Mar 2021 11:29:27 -0400 Subject: [OpenAFS] Occasional "VLDB: no permission access for call" In-Reply-To: References: Message-ID: <1b86bff8-2980-4500-7ca1-cbc9f56c89d5@auristor.com> This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Ohs5DN0AHUT6Lg2lZLSIfrxMP1R3pCwK3 Content-Type: multipart/mixed; boundary="stiTzuLDuIXdDagOzfQiapGMBsbvopDx8"; protected-headers="v1" From: Jeffrey E Altman To: Ian Wienand Cc: openafs-info@openafs.org Message-ID: <1b86bff8-2980-4500-7ca1-cbc9f56c89d5@auristor.com> Subject: Re: [OpenAFS] Occasional "VLDB: no permission access for call" References: In-Reply-To: --stiTzuLDuIXdDagOzfQiapGMBsbvopDx8 Content-Type: multipart/mixed; boundary="------------3FA84F811DC40885B55B2E2D" Content-Language: en-US This is a multi-part message in MIME format. --------------3FA84F811DC40885B55B2E2D Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable On 3/29/2021 12:23 AM, Ian Wienand (iwienand@redhat.com) wrote: > A new thing I've noticed after we have upgraded everything to 1.8.6 openstack.org also deployed a new database server and this problem is=20 most likely due to a failure to synchronize the super-user list onto the = new vlserver. If the vos command randomly chooses to speak with the new = vlserver instance first, it receives a permission denied error and does=20 not contact any other servers. Jeffrey Altman --------------3FA84F811DC40885B55B2E2D Content-Type: text/x-vcard; charset=utf-8; name="jaltman.vcf" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="jaltman.vcf" begin:vcard fn:Jeffrey Altman n:Altman;Jeffrey org:AuriStor, Inc. adr:;;255 W 94TH ST STE 6B;New York;NY;10025-6985;United States email;internet:jaltman@auristor.com title:CEO tel;work:+1-212-769-9018 url:https://www.linkedin.com/in/jeffreyaltman/ version:2.1 end:vcard --------------3FA84F811DC40885B55B2E2D-- --stiTzuLDuIXdDagOzfQiapGMBsbvopDx8-- --Ohs5DN0AHUT6Lg2lZLSIfrxMP1R3pCwK3 Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEE+kRK8Zf0SbJM8+aZ93pzVZK2mgQFAmBh8lcFAwAAAAAACgkQ93pzVZK2mgSU ZQ//c/5b3dpSyPPzWxJJ8v4Kv6GZGqyu2qnFX1E9vSl6EEij1yNOBR3xAPn6BHJkgaFyrSGe7YKj MmW+CJyfbk0Vpb3nx8Ca3+T24CNWXH9qOGRJi3Ut/d6cIjub4WJK1LnJWHsV+8wL4amNzVdf3WIT SwqK0XlZ9Lwu1VawxHtTqVN6WcEkDwFxJOlEus1uRO9Cmst6H0FxPkw1iSzUToRQCEBmkoD2X9I4 v171O7DV4k81O70sfjOePC2aE8I7xEXe7Wo+lpJVihsYpe9gOtA2a5ZT4ZbQo+ZiyYKotr3HX7RV csQBtK0KFBGm/Zp3aYgAbxN+CzJ9d1QsastSzFWPKjEllPS0c9HZBCzAi98mVoo6r/Qeb08HZDQ/ ItD3bVpJUpcAOPTuw0KxsGafp1PyqL8DuxBSU5aCTt6sjfst39tgFp8hW46MAC0XepBSrnvsrSUa 5kjJi2JDei9NO7Zm9CdDCUHF/yjute8mvhVEI9KKPc0gwMEM9yxiv0QWag1wGPabbXibDuWJspJi sC4gDC6U6GyGSTyOLpwSJfmUTw89Pj3mR9Qi+12ApGFwY0UM6ecrpnwdRNpR1gRSNM6xETV8HHGB poXEy9KDjKWkpWxuq4So2waJjKAYUQe+M8s3G53CN1+9Ck/Lf3L2q2Oe8Ry5/v5F40RCucI7qN6m 59Q= =19WW -----END PGP SIGNATURE----- --Ohs5DN0AHUT6Lg2lZLSIfrxMP1R3pCwK3-- From mvitale@sinenomine.net Mon Mar 29 16:31:28 2021 From: mvitale@sinenomine.net (Mark Vitale) Date: Mon, 29 Mar 2021 15:31:28 +0000 Subject: [OpenAFS] Any xstat tools? In-Reply-To: References: Message-ID: Chaskiel, > On Mar 29, 2021, at 9:11 AM, Chaskiel Grundman wrot= e: >=20 > Does anyone have tools for capturing, aggregating, analyzing or displayin= g info from xstats? Not so much operation timing, though that may also be h= elpful. I forgot to mention that if you are interested in RPC operation counts and = times, the xstat_fs_test RPC metrics (-collID 2) are adequate and work out of the = box, but they are only available for fileservers. Similarly, xstat_cm_test also reports RPC operation counts and times, but only for cache managers. These metrics have some other limitations as well. The lesser-known Rx RPC statistics (aka RPCSTATS, RXSTATS) have to be enabl= ed, but they are easy to enable (via the --enable_processs_stats option or rxstat_enable= _process command). I consider them superior to what's available via xstat_fs_test:=20 - they are universal (work _almost_ the same for all OpenAFS components, wi= th some cache manager quirks) - they are more accurate (they include some rx and dispatch overhead that x= stat_* can't capture) - they record RPC metrics from both the server (RPCs received) and client (= RPCs issued) perspective - they are remotely resettable (via rxstat_clear_process).=20 - they support both process (server) and peer (client) granularity =20 Regards, -- Mark Vitale mvitale@sinenomine.net From iwienand@redhat.com Tue Mar 30 04:48:31 2021 From: iwienand@redhat.com (Ian Wienand) Date: Tue, 30 Mar 2021 14:48:31 +1100 Subject: [OpenAFS] Occasional "VLDB: no permission access for call" In-Reply-To: <1b86bff8-2980-4500-7ca1-cbc9f56c89d5@auristor.com> References: <1b86bff8-2980-4500-7ca1-cbc9f56c89d5@auristor.com> Message-ID: On Mon, Mar 29, 2021 at 11:29:27AM -0400, Jeffrey E Altman wrote: > openstack.org also deployed a new database server and this problem is most > likely due to a failure to synchronize the super-user list onto the new > vlserver. If the vos command randomly chooses to speak with the new > vlserver instance first, it receives a permission denied error and does not > contact any other servers. Thank you, an astute observation as usual :) Ensuring all afsdb servers have the same /etc/openafs/server/UserList via our configuration deployment has solved this and I can "vos release" in a loop as much as I want. Many thanks, -i