[OpenAFS] best practice for a service to access a user AFS token?
and why ruid instead of euid?
Jeffrey Altman
jaltman@auristor.com
Fri, 18 Nov 2016 00:59:23 -0500
This is a cryptographically signed message in MIME format.
--------------ms070803040801090608090303
Content-Type: multipart/mixed;
boundary="------------67499816D6C2E0D0C0FD661D"
This is a multi-part message in MIME format.
--------------67499816D6C2E0D0C0FD661D
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Hi Todd,
Thanks for your note.
On 11/17/2016 2:27 PM, Todd Tannenbaum wrote:
> Hi OpenAFS gurus, I am in desperate need of your advice!
>=20
> We are adding OpenAFS support to HTCondor (http://htcondor.org).
>=20
> I read in the docs that the Cache Manager identifies token by either th=
e
> user's UNIX UID or by a process authentication group (PAG). In the
> non-PAG case, I expected the cache manager to identify the token by
> _effective_ uid. But my testing implies that the token is identified b=
y
> the _real_ uid. Is there a way to change this behavior?
Your description is close but not quite correct. Tokens are always
tracked by a Process Authentication Group. The distinction is whether
the PAG identifier is a unique identifier independent of the user id or
is the user id. It is an explicit security property of a PAG that one
user (including root) cannot switch a process into a PAG. To change
this behavior would break the security guarantees that PAGs provide.
> My situation is hopefully not new or unique : I have a long-running
> service (HTCondor in this case) running as root that is implemented via=
> a cooperating set of processes, and this service needs to impersonate
> different users in order read/write to the filesystem on their behalf.
The behavior of HTCondor is similar to that of IBM's Platform Load
Sharing Facility (LSF) which distributes jobs across an HPC environment
where the jobs have the need to authenticate to and access resources
over a network. LSF supports the ability to forward Kerberos v5 ticket
granting tickets with the jobs, renew them on the individual nodes. LSF
has the ability to create a PAG but does not have any specific knowledge
of how to obtain afs (or AuriStorFS) tokens. Those tokens are obtained
by executing a shell script that can be configured with the job.
When jobs will require weeks or months to complete it is critical from a
security perspective to distribute Kerberos tickets that have a
short-lifetime but can be renewed for the length of the job provided
that the represented principal has not been disabled.
> Ideally I'd like this service to be as ignorant of AFS is possible. It=
> currently performs impersonation by changing out the effective UID,
> which of course works fine for the local filesystem.
>
> Meanwhile, I have another service running (a home-brewed "credential
> manager") on the machine that keeps AFS tokens refreshed for all users
> of the service by doing an "aklog" as each user. Both HTCondor and the=
> credential manager are not running in a PAG, as my understanding is a
> PAG can only hold one user token per cell.
It is correct that a cache manager can only associate one token (or
identity) per cell in each PAG. If a PAG stored more than one identity
per cell there would be an identity selection problem.
There is nothing that prevents a user-land application from storing more
than one token per cell and setting the desired token into the cache
manager prior to the file system requests that must use it are issued.
> The problem is the HTCondor service cannot access AFS on behalf of the
> user being impersonated by simply switching effective UID; it apparentl=
y
> needs to switch over the real UID as well. Thanks to the saved UID and=
> the setresuid() syscall, it is trivial to change HTCondor to switch bot=
h
> the read and effective UID over to the user being impersonated, and
> still switch back to root afterwards. Doing so would likely make
> everything work great with AFS. But the problem is changing the
> real-uid is a potential security hole, as now HTCondor service processe=
s
> could receive signals (i.e. SIGKILL!) from the unprivileged users it is=
> simply trying to impersonate. This is why OpenAFS's reliance on using=
> the real UID for identifying tokens seems very broken to me; seems like=
> it should be euid (or specifically on Linux clients, the file-system
> uid) like every other similar system...
The security hole that you want to avoid introducing into HTCondor is
exactly the security hole that would be introduced into every
application if PAGs followed the effective uid.
>=20
> Any thoughts/advice?
>=20
> One idea is I could have all the HTCondor service processes run in thei=
r
> own PAG and perform some sort of very lightweight "aklog" whenever
> impersonation is required. But I need this "aklog" to be very
> fast/lightweight; I don't want to be blocked on network communication t=
o
> some KDC or even file I/O if I can help it. The service would have
> access to a KRB5 credential cache for the user that has an AFS service
> principal; given that, is there some lightweight "aklog"-like in-proces=
s
> library call I can use (that avoids I/O)? Something the HTCondor
> service could dlopen()/dlsym() (so I don't introduce a pile of
> dependencies) ?
There are four steps that need to be performed:
1. obtain and/or renew Kerberos v5 tickets
2. obtain the afs (or AuriStorFS token)
3. create the PAG for the process
4. set the token for the PAG
Step 3 needs to be performed once per process and does not involve the
network.
Step 4 needs to be performed each time the token is replaced due to
renewal. It does not require network.
Steps 1 and 2 do require network but can be performed in the background
after the initial acquisition.
>=20
> Another idea, as I only care about supporting Linux, would be to
> leverage Linux kernel keyrings. I am thinking perhaps my credential
> manager could link the "afs_pag: _pag" key to the user keyring, and the=
n
> the HTCondor service could link this key into its session keyring when
> impersonating. Does anyone think that would work? Or is there more to=
> swapping PAGs in and out (i.e. besides the key on the keyring, pagsh
> seems to do some magic with groups as "/bin/id" shows some magic groups=
> when in a PAG...) ? Is the keyring-based idea crazy talk or a good ide=
a
> to pursue if Linux is my only target?
The groups are not required on Linux. The cache manager uses keyrings
internally to track the PAG.
The behavior you are seeking with user-land manipulation of keyrings to
store afs tokens is the one that David Howell's kafs client will provide
when it is complete.
https://www.infradead.org/~dhowells/kafs/
> I've seen the great lengths (i.e. immense amount of code, security
> side-steps of creating their own krb4 tickets) that Samba has done to
> support AFS; I am hoping there is an easier way.
Samba had a different problem than HTCondor. An SMB 1.1 server used to
authenticate a client by the client sending a username and password over
the wire to the server which could then be used with kinit (or
equivalent library calls) to obtain a Kerberos TGT. In the 90s this was
more often than not Kerberos v4.
In SMB 1.2 NTLM authentication was introduced and later GSS-Kerberos v5
authentication. Both avoided the transmission of the user's password.
Therefore, Samba had to obtain a Kerberos v5 ticket granting ticket or
an AFS token some other way. This typically made use of an
impersonation service to acquire a TGT or Token after asserting it had
authenticated the user identity. Many single sign-on web authentication
services utilize a similar model.
> Your suggestions greatly appreciated.
The best approach in my opinion is to follow the LSF model.
Jeffrey Altman
AuriStor Inc.
--------------67499816D6C2E0D0C0FD661D
Content-Type: text/x-vcard; charset=utf-8;
name="jaltman.vcf"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
filename="jaltman.vcf"
begin:vcard
fn:Jeffrey Altman
n:Altman;Jeffrey
org:AuriStor, Inc.
adr:Suite 6B;;255 West 94Th Street;New York;New York;10025-6985;United St=
ates
email;internet:jaltman@auristor.com
title:Founder and CEO
tel;work:+1-212-769-9018
note;quoted-printable:LinkedIn: https://www.linkedin.com/in/jeffreyaltman=
=3D0D=3D0A=3D
Skype: jeffrey.e.altman=3D0D=3D0A=3D
=09
url:https://www.auristor.com/
version:2.1
end:vcard
--------------67499816D6C2E0D0C0FD661D--
--------------ms070803040801090608090303
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
DIMwggXqMIIE0qADAgECAhBAAVgjDHzXCy5hFo6GsQuLMA0GCSqGSIb3DQEBCwUAMDoxCzAJ
BgNVBAYTAlVTMRIwEAYDVQQKEwlJZGVuVHJ1c3QxFzAVBgNVBAMTDlRydXN0SUQgQ0EgQTEy
MB4XDTE2MTEwMjAzMTkzMFoXDTE3MTEwMjAzMTkzMFowgYYxLTArBgNVBAsMJFZlcmlmaWVk
IEVtYWlsOiBqYWx0bWFuQGF1cmlzdG9yLmNvbTEjMCEGCSqGSIb3DQEJARYUamFsdG1hbkBh
dXJpc3Rvci5jb20xMDAuBgoJkiaJk/IsZAEBEyA3RjAwMDAwMTAwMDAwMTU4MjMwQzdDQTcw
MDAwMDdCMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALtO/7veyX1tdA4aDvsO
c0gS/fICCTPlhS34dpRNmZYs2mvWb/xGRCKeflva8uZLFqsCE6ybMgz2IaB9rpowRcGSr/R7
CyY6HEBl3Rw0OkkslV7HQTxgp1fBXbGGoS8gOp6/ML64d5UJsMUM6NM59SzC77v8bi0sYkaT
q31SoiJrCcPiI0F2wZiESf1tguQONl3Emt9fhEokmlLRHAlv1DkRw2XeUOyUwVHIVgF1wnzR
4Ao86aYmym62Z669a6NPla3hqLmYk9w+ydTnzcWj0X6nhvaMxb7fhOUyj06Y5Mxhqye+dJg4
bPj/G+8OoNVY9l5h/qbJaWpSbDog4I0LxaUCAwEAAaOCAp0wggKZMA4GA1UdDwEB/wQEAwIF
oDCBhAYIKwYBBQUHAQEEeDB2MDAGCCsGAQUFBzABhiRodHRwOi8vY29tbWVyY2lhbC5vY3Nw
LmlkZW50cnVzdC5jb20wQgYIKwYBBQUHMAKGNmh0dHA6Ly92YWxpZGF0aW9uLmlkZW50cnVz
dC5jb20vY2VydHMvdHJ1c3RpZGNhYTEyLnA3YzAfBgNVHSMEGDAWgBSkc9rvaTWKdcygGXsI
MvhrieRC7DAJBgNVHRMEAjAAMIIBLAYDVR0gBIIBIzCCAR8wggEbBgtghkgBhvkvAAYLATCC
AQowSgYIKwYBBQUHAgEWPmh0dHBzOi8vc2VjdXJlLmlkZW50cnVzdC5jb20vY2VydGlmaWNh
dGVzL3BvbGljeS90cy9pbmRleC5odG1sMIG7BggrBgEFBQcCAjCBrhqBq1RoaXMgVHJ1c3RJ
RCBDZXJ0aWZpY2F0ZSBoYXMgYmVlbiBpc3N1ZWQgaW4gYWNjb3JkYW5jZSB3aXRoIApJZGVu
VHJ1c3QncyBUcnVzdElEIENlcnRpZmljYXRlIFBvbGljeSBmb3VuZCBhdCBodHRwczovL3Nl
Y3VyZS5pZGVudHJ1c3QuY29tL2NlcnRpZmljYXRlcy9wb2xpY3kvdHMvaW5kZXguaHRtbDBF
BgNVHR8EPjA8MDqgOKA2hjRodHRwOi8vdmFsaWRhdGlvbi5pZGVudHJ1c3QuY29tL2NybC90
cnVzdGlkY2FhMTIuY3JsMB8GA1UdEQQYMBaBFGphbHRtYW5AYXVyaXN0b3IuY29tMB0GA1Ud
DgQWBBT6okll+NyYQTyBKCigoq5jFBzw3DAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH
AwQwDQYJKoZIhvcNAQELBQADggEBABF9+YNU05P6tFiN8wvh52K+dgiUsQijFqN/tmwxqWHj
3TKxLwl8ZtK0F44FYcfeRXgE59bAxG8tZVRYBD7jY3qYEreri0x2DgJcA0oPfV1NHxM51t6h
ST09jpXEcthB4FJu3xWg8O6pB7Oq9Q9kkTjHfVLLru9B3BxrY+1NKZaSkaTBb0JGw6kCN2MM
rXrwPAdLlAWn5ywPRMqTIlgvhxtIQMv+dxpXhco9O5+ckMIdp1uBMJ82Hslfy3K9KCZvm6Cr
BG8C/fBo5TefVGuxKyOwROvup2gx1/KZWoIO0hyMNGzFPrijNZD5fAlZ11FcAJ9Why+pd5FY
mmJwln/BGPcwggaRMIIEeaADAgECAhEA+d5Wf8lNDHdw+WAbUtoVOzANBgkqhkiG9w0BAQsF
ADBKMQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScwJQYDVQQDEx5JZGVuVHJ1
c3QgQ29tbWVyY2lhbCBSb290IENBIDEwHhcNMTUwMjE4MjIyNTE5WhcNMjMwMjE4MjIyNTE5
WjA6MQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MRcwFQYDVQQDEw5UcnVzdElE
IENBIEExMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANGRTTzPCic0kq5L6ZrU
JWt5LE/n6tbPXPhGt2Egv7plJMoEpvVJJDqGqDYymaAsd8Hn9ZMAuKUEFdlx5PgCkfu7jL5z
giMNnAFVD9PyrsuF+poqmlxhlQ06sFY2hbhQkVVQ00KCNgUzKcBUIvjv04w+fhNPkwGW5M7A
e5K5OGFGwOoRck9GG6MUVKvTNkBw2/vNMOd29VGVTtR0tjH5PS5yDXss48Yl1P4hDStO2L4w
TsW2P37QGD27//XGN8K6amWB6F2XOgff/PmlQjQOORT95PmLkwwvma5nj0AS0CVp8kv0K2RH
V7GonllKpFDMT0CkxMQKwoj+tWEWJTiDKSsCAwEAAaOCAoAwggJ8MIGJBggrBgEFBQcBAQR9
MHswMAYIKwYBBQUHMAGGJGh0dHA6Ly9jb21tZXJjaWFsLm9jc3AuaWRlbnRydXN0LmNvbTBH
BggrBgEFBQcwAoY7aHR0cDovL3ZhbGlkYXRpb24uaWRlbnRydXN0LmNvbS9yb290cy9jb21t
ZXJjaWFscm9vdGNhMS5wN2MwHwYDVR0jBBgwFoAU7UQZwNPwBovupHu+QucmVMiONnYwDwYD
VR0TAQH/BAUwAwEB/zCCASAGA1UdIASCARcwggETMIIBDwYEVR0gADCCAQUwggEBBggrBgEF
BQcCAjCB9DBFFj5odHRwczovL3NlY3VyZS5pZGVudHJ1c3QuY29tL2NlcnRpZmljYXRlcy9w
b2xpY3kvdHMvaW5kZXguaHRtbDADAgEBGoGqVGhpcyBUcnVzdElEIENlcnRpZmljYXRlIGhh
cyBiZWVuIGlzc3VlZCBpbiBhY2NvcmRhbmNlIHdpdGggSWRlblRydXN0J3MgVHJ1c3RJRCBD
ZXJ0aWZpY2F0ZSBQb2xpY3kgZm91bmQgYXQgaHR0cHM6Ly9zZWN1cmUuaWRlbnRydXN0LmNv
bS9jZXJ0aWZpY2F0ZXMvcG9saWN5L3RzL2luZGV4Lmh0bWwwSgYDVR0fBEMwQTA/oD2gO4Y5
aHR0cDovL3ZhbGlkYXRpb24uaWRlbnRydXN0LmNvbS9jcmwvY29tbWVyY2lhbHJvb3RjYTEu
Y3JsMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDAOBgNVHQ8BAf8EBAMCAYYwHQYD
VR0OBBYEFKRz2u9pNYp1zKAZewgy+GuJ5ELsMA0GCSqGSIb3DQEBCwUAA4ICAQAN4YKu0vv0
62MZfg+xMSNUXYKvHwvZIk+6H1pUmivyDI4I6A3wWzxlr83ZJm0oGIF6PBsbgKJ/fhyyIzb+
vAYFJmyI8I/0mGlc+nIQNuV2XY8cypPoVJKgpnzp/7cECXkX8R4NyPtEn8KecbNdGBdEaG4a
7AkZ3ujlJofZqYdHxN29tZPdDlZ8fR36/mAFeCEq0wOtOOc0Eyhs29+9MIZYjyxaPoTS+l8x
LcuYX3RWlirRyH6RPfeAi5kySOEhG1quNHe06QIwpigjyFT6v/vRqoIBr7WpDOSt1VzXPVbS
j1PcWBgkwyGKHlQUOuSbHbHcjOD8w8wHSDbL+L2he8hNN54doy1e1wJHKmnfb0uBAeISoxRb
JnMMWvgAlH5FVrQWlgajeH/6NbYbBSRxALuEOqEQepmJM6qz4oD2sxdq4GMN5adAdYEswkY/
o0bRKyFXTD3mdqeRXce0jYQbWm7oapqSZBccFvUgYOrB78tB6c1bxIgaQKRShtWR1zMM0Jfq
UfD9u8Fg7G5SVO0IG/GcxkSvZeRjhYcbTfqF2eAgprpyzLWmdr0mou3bv1Sq4OuBhmTQCnqx
AXr4yVTRYHkp5lCvRgeJAme1OTVpVPth/O7HJ7VuEP9GOr6kCXCXmjB4P3UJ2oU0NqfoQdcS
SSt9hliALnExTEjii20B2nSDojGCAxQwggMQAgEBME4wOjELMAkGA1UEBhMCVVMxEjAQBgNV
BAoTCUlkZW5UcnVzdDEXMBUGA1UEAxMOVHJ1c3RJRCBDQSBBMTICEEABWCMMfNcLLmEWjoax
C4swDQYJYIZIAWUDBAIBBQCgggGXMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZI
hvcNAQkFMQ8XDTE2MTExODA1NTkyM1owLwYJKoZIhvcNAQkEMSIEIO2qHu//77m8fTsrou4U
TYiA4TXXko0v60NPfBIWWKhjMF0GCSsGAQQBgjcQBDFQME4wOjELMAkGA1UEBhMCVVMxEjAQ
BgNVBAoTCUlkZW5UcnVzdDEXMBUGA1UEAxMOVHJ1c3RJRCBDQSBBMTICEEABWCMMfNcLLmEW
joaxC4swXwYLKoZIhvcNAQkQAgsxUKBOMDoxCzAJBgNVBAYTAlVTMRIwEAYDVQQKEwlJZGVu
VHJ1c3QxFzAVBgNVBAMTDlRydXN0SUQgQ0EgQTEyAhBAAVgjDHzXCy5hFo6GsQuLMGwGCSqG
SIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggq
hkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwDQYJ
KoZIhvcNAQEBBQAEggEAAIHMrTmyCXXJotOrYMHht7GwJyjGhlsgh1XptChrfaxcgU95g4mi
UIgqK0Kke9rw57eCPSKuKh6W3OU8CZkKu3GRkYPc6bLJnygCcsfYA12XUNlOw5DXbV9ODYxF
4zRT8twMvt8pRaThDspXIgXXmojhJZyDBJnl6NbAVnUxPAjql8IB4zwmafFbiD/D1pmUa62c
9JByyO5lfXgAOAgC1n/zMGALW4JTUveq9wOYEFpJt8WmLilsHsK7YXYiD/95R7wxA4TTQYEN
nYowTlfYSGPc5q4PBuDr8UAzOpGQbLkJ/ff8c4iBsO4BCjtCW5VZOe+n6F/8rZZ8jbqvhRSj
dwAAAAAAAA==
--------------ms070803040801090608090303--