[OpenAFS] KeyFile issues upgrading servers from 1.4 to 1.6
Jeffrey Altman
jaltman@auristor.com
Fri, 22 Dec 2017 16:25:51 -0500
This is a cryptographically signed message in MIME format.
--------------ms040509000200000201060800
Content-Type: multipart/mixed;
boundary="------------DF56B97E2E4FC259150E03C5"
Content-Language: en-US
This is a multi-part message in MIME format.
--------------DF56B97E2E4FC259150E03C5
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
On 12/20/2017 2:27 PM, Anjana Kar wrote:
> Currently we have 4 servers running openafs-server-1.4.14-el5.1.1,
> 2 of them being database servers.
According to your database servers the psc.edu cell has three database
servers:
>vos eachvl -cell psc.edu -noauth -execute \
-format "rxdebug %s 7003 -version"
Trying all endpoints for daphne.psc.edu:
Version: OpenAFS 1.4.14 built 2010-12-27
Trying all endpoints for velma.psc.edu:
Version: OpenAFS 1.4.14 built 2010-12-27
Trying all endpoints for shaggy.psc.edu:
Version: OpenAFS 1.4.14 built 2010-12-27
>udebug daphne.psc.edu -coord
[128.182.66.185]:7003 is not the coordinator
Contacting coordinator at [128.182.66.184]:7003
First response received from [128.182.66.184]:7003
Host's addresses are: 128.182.66.184 10.32.5.186
Host's time is Fri Dec 22 15:23:31 2017
Local time is Fri Dec 22 15:23:31 2017 (time differential 0 secs)
Last yes vote for 128.182.66.184 was 3 secs ago (coordinator);
Last vote started 3 secs ago (at Fri Dec 22 15:23:28 2017)
Local db version is 1502154074.168010
I am coordinator until 56 secs from now (at Fri Dec 22 15:24:27 2017) (3
servers)
Recovery state 1f (have best db version; sync complete; db modified)
The last trans I handled was 1502154074.28237654
Coordinator's db version is 1502154074.168010
0 locked pages, 0 of them for write
Last time a new db version was labelled was:
11820137 secs ago (at Mon Aug 07 21:01:14 2017)
Server (128.182.66.185 10.32.5.185): (db 1502154074.168010)
last vote rcvd 4 secs ago (at Fri Dec 22 15:23:27 2017),
last beacon sent 3 secs ago (at Fri Dec 22 15:23:28 2017), last vote
was yes
dbcurrent=3D1, up=3D1 beaconSince=3D1
Server (128.182.59.182): (db 1502154074.168010)
last vote rcvd 3 secs ago (at Fri Dec 22 15:23:28 2017),
last beacon sent 3 secs ago (at Fri Dec 22 15:23:28 2017), last vote
was yes
dbcurrent=3D1, up=3D1 beaconSince=3D1
Each of these servers is also a fileserver.
There is one more server, 128.182.59.77, which is only a fileserver.
As these systems are 1.4.14 they could not be rekeyed to support AES
Kerberos service keys in order to address
"Brute force DES attack permits compromise of AFS cell"
http://www.openafs.org/pages/security/#OPENAFS-SA-2013-003
> The new VM servers have openafs-server-1.6.16-1.el7.centos.x86_64,
> and we'd like to configure them so they can replace the 1.4 servers.
Clients and admin tools find the psc.edu database servers through the
CellServDB file published by grand.central.org and distributed with OpenA=
FS:
>psc.edu #PSC (Pittsburgh Supercomputing Center)
128.182.59.182 #shaggy.psc.edu
128.182.66.184 #velma.psc.edu
128.182.66.185 #daphne.psc.edu
DNS SRV records:
_afs3-vlserver._udp.psc.edu SRV service location:
priority =3D 0
weight =3D 0
port =3D 7003
svr hostname =3D daphne.psc.edu
_afs3-vlserver._udp.psc.edu SRV service location:
priority =3D 0
weight =3D 0
port =3D 7003
svr hostname =3D shaggy.psc.edu
_afs3-vlserver._udp.psc.edu SRV service location:
priority =3D 0
weight =3D 0
port =3D 7003
svr hostname =3D velma.psc.edu
DNS AFSDB records:
psc.edu AFSDB subtype =3D 1, AFS db server =3D shaggy.psc.edu
psc.edu AFSDB subtype =3D 1, AFS db server =3D velma.psc.edu
psc.edu AFSDB subtype =3D 1, AFS db server =3D daphne.psc.edu
in that order.
When using CellServDB file data the OpenAFS UNIX clients only use the IP
addresses and the Windows clients only use the DNS host names.
Clients will remember the resolved IP addresses until either:
1. they are restarted
2. "fs newcell" is issued to update the address list for the cell
As such, when replacing database servers it is very important that the
host names and IP addresses be preserved across the replacement.
Otherwise, there will be an impact on the clients.
> The first problem we've run into is with the KeyFile. "bos create" also=
> gives the same message.
>
> [root@afs-vma etc]# bos listkeys afs-vma.psc.edu -localauth
> bos: ticket contained unknown key version number error encountered whil=
e
> listing keys
>=20
> Question is do we need to create a new KeyFile?
The "KeyFile" from the existing servers should be copied to the new
servers as is.
> Are there any documentation or steps we can follow for this migration?
Due to bugs in the 1.4.14 database server implementation it is important
that a database server remain off for at least five minutes before it is
restarted each time the server is shutdown. Failure to do so can result
in corruption of the replicated database.
The 1.6.16 release (as are all versions before 1.6.22) is vulnerable to
a remote denial of service attack that can result in server panics. I
strongly advise deploying 1.6.22 or later.
Once all of the servers have been upgraded to at least 1.6.22 it is
critical that the DES cell key be replaced with an
AES256-CTS-HMAC-SHA1-96 Kerberos service key. Failure to do so leaves
the cell vulnerable to brute force attacks.
AuriStor provides professional OpenAFS support services to assist
organizations such as PSC when upgrading cells.
https://www.auristor.com/openafs/
Jeffrey Altman
--------------DF56B97E2E4FC259150E03C5
Content-Type: text/x-vcard; charset=utf-8;
name="jaltman.vcf"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
filename="jaltman.vcf"
begin:vcard
fn:Jeffrey Altman
n:Altman;Jeffrey
org:AuriStor, Inc.
adr:Suite 6B;;255 West 94Th Street;New York;New York;10025-6985;United St=
ates
email;internet:jaltman@auristor.com
title:Founder and CEO
tel;work:+1-212-769-9018
note;quoted-printable:LinkedIn: https://www.linkedin.com/in/jeffreyaltman=
=3D0D=3D0A=3D
Skype: jeffrey.e.altman=3D0D=3D0A=3D
=09
url:https://www.auristor.com/
version:2.1
end:vcard
--------------DF56B97E2E4FC259150E03C5--
--------------ms040509000200000201060800
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
DIIwggXpMIIE0aADAgECAhBAAV7gPRitcrlGsJTzkwjvMA0GCSqGSIb3DQEBCwUAMDoxCzAJ
BgNVBAYTAlVTMRIwEAYDVQQKEwlJZGVuVHJ1c3QxFzAVBgNVBAMTDlRydXN0SUQgQ0EgQTEy
MB4XDTE3MTAwMzAzMTczM1oXDTE4MTEwMzAzMTczM1owgYUxLTArBgNVBAsMJFZlcmlmaWVk
IEVtYWlsOiBqYWx0bWFuQGF1cmlzdG9yLmNvbTEjMCEGCSqGSIb3DQEJARYUamFsdG1hbkBh
dXJpc3Rvci5jb20xLzAtBgoJkiaJk/IsZAEBEx9BMDE0MjdFMDAwMDAxNUVFMDNEMTg3QTAw
MDA0QUE1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqqJC89ZA1DSS7t/Ug8Dd
BQv5nBDumInWtFvHwVCORitVCvlkX4SfqKpERATq0eHOSc0zEz1PUjhAT8lgbNj8Bs92pL9t
DW/VHHpq11w06rCEmZJNxgErAIvMpRuAhGrzvBpQBLj8nDArHWw+5nRn/KnK7ZO81LEEj4TG
w0PEKGSa0aFA+JdRTJ6BZSDP2o/8AHx+Bw4JgW8VppAe4IuY/F+JoYtyQDL+fm1YMnFMtf1A
6IvlGXD7gMksPRbVIfD+QpHZbQvNXZAVVDaCWZuWQq46Vl4lSlkmW9yMlGddvFGl2zSMK7ny
f0kbWJLw9lZxXDegY0/ciJPACPsyBwuyLwIDAQABo4ICnTCCApkwDgYDVR0PAQH/BAQDAgWg
MIGEBggrBgEFBQcBAQR4MHYwMAYIKwYBBQUHMAGGJGh0dHA6Ly9jb21tZXJjaWFsLm9jc3Au
aWRlbnRydXN0LmNvbTBCBggrBgEFBQcwAoY2aHR0cDovL3ZhbGlkYXRpb24uaWRlbnRydXN0
LmNvbS9jZXJ0cy90cnVzdGlkY2FhMTIucDdjMB8GA1UdIwQYMBaAFKRz2u9pNYp1zKAZewgy
+GuJ5ELsMAkGA1UdEwQCMAAwggEsBgNVHSAEggEjMIIBHzCCARsGC2CGSAGG+S8ABgsBMIIB
CjBKBggrBgEFBQcCARY+aHR0cHM6Ly9zZWN1cmUuaWRlbnRydXN0LmNvbS9jZXJ0aWZpY2F0
ZXMvcG9saWN5L3RzL2luZGV4Lmh0bWwwgbsGCCsGAQUFBwICMIGuGoGrVGhpcyBUcnVzdElE
IENlcnRpZmljYXRlIGhhcyBiZWVuIGlzc3VlZCBpbiBhY2NvcmRhbmNlIHdpdGggCklkZW5U
cnVzdCdzIFRydXN0SUQgQ2VydGlmaWNhdGUgUG9saWN5IGZvdW5kIGF0IGh0dHBzOi8vc2Vj
dXJlLmlkZW50cnVzdC5jb20vY2VydGlmaWNhdGVzL3BvbGljeS90cy9pbmRleC5odG1sMEUG
A1UdHwQ+MDwwOqA4oDaGNGh0dHA6Ly92YWxpZGF0aW9uLmlkZW50cnVzdC5jb20vY3JsL3Ry
dXN0aWRjYWExMi5jcmwwHwYDVR0RBBgwFoEUamFsdG1hbkBhdXJpc3Rvci5jb20wHQYDVR0O
BBYEFNefZrPaqPUvaS6V6kAmHDwFhoDiMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcD
BDANBgkqhkiG9w0BAQsFAAOCAQEAKlssrfOJ5+WwHyhFSeSsioN0qpg2QDX/uvodF38JbquO
1U0my0j3Cc/bwk48++bjzp0Fvk/Kkcmss5/6zzJMjr9rf12QCQfKkbO9nMm8Bg6IP3pYgk0W
/F1h3ZQF3OgBn3zZoOd3f1a6dF6z12MqKA/2g5GKrQFxkdzTGrNw6ISE9uY8ysvc3i2N2kas
HNi5Etk7StZ1jvFX5sQMIeNdlF+z+BU/AyT7NoBS4gCH+ggF+DG7fAYywvy42Lfu8p6kopKT
5JZpYce1cNjnOaDhzhgeR+oXxoDbekF27JinXHQSKjBxhujcZu5leAkpctFpZxnIKZJZUBiu
31Nm7xYaijCCBpEwggR5oAMCAQICEQD53lZ/yU0Md3D5YBtS2hU7MA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRIwEAYDVQQKEwlJZGVuVHJ1c3QxJzAlBgNVBAMTHklkZW5UcnVz
dCBDb21tZXJjaWFsIFJvb3QgQ0EgMTAeFw0xNTAyMTgyMjI1MTlaFw0yMzAyMTgyMjI1MTla
MDoxCzAJBgNVBAYTAlVTMRIwEAYDVQQKEwlJZGVuVHJ1c3QxFzAVBgNVBAMTDlRydXN0SUQg
Q0EgQTEyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0ZFNPM8KJzSSrkvpmtQl
a3ksT+fq1s9c+Ea3YSC/umUkygSm9UkkOoaoNjKZoCx3wef1kwC4pQQV2XHk+AKR+7uMvnOC
Iw2cAVUP0/Kuy4X6miqaXGGVDTqwVjaFuFCRVVDTQoI2BTMpwFQi+O/TjD5+E0+TAZbkzsB7
krk4YUbA6hFyT0YboxRUq9M2QHDb+80w53b1UZVO1HS2Mfk9LnINeyzjxiXU/iENK07YvjBO
xbY/ftAYPbv/9cY3wrpqZYHoXZc6B9/8+aVCNA45FP3k+YuTDC+ZrmePQBLQJWnyS/QrZEdX
saieWUqkUMxPQKTExArCiP61YRYlOIMpKwIDAQABo4ICgDCCAnwwgYkGCCsGAQUFBwEBBH0w
ezAwBggrBgEFBQcwAYYkaHR0cDovL2NvbW1lcmNpYWwub2NzcC5pZGVudHJ1c3QuY29tMEcG
CCsGAQUFBzAChjtodHRwOi8vdmFsaWRhdGlvbi5pZGVudHJ1c3QuY29tL3Jvb3RzL2NvbW1l
cmNpYWxyb290Y2ExLnA3YzAfBgNVHSMEGDAWgBTtRBnA0/AGi+6ke75C5yZUyI42djAPBgNV
HRMBAf8EBTADAQH/MIIBIAYDVR0gBIIBFzCCARMwggEPBgRVHSAAMIIBBTCCAQEGCCsGAQUF
BwICMIH0MEUWPmh0dHBzOi8vc2VjdXJlLmlkZW50cnVzdC5jb20vY2VydGlmaWNhdGVzL3Bv
bGljeS90cy9pbmRleC5odG1sMAMCAQEagapUaGlzIFRydXN0SUQgQ2VydGlmaWNhdGUgaGFz
IGJlZW4gaXNzdWVkIGluIGFjY29yZGFuY2Ugd2l0aCBJZGVuVHJ1c3QncyBUcnVzdElEIENl
cnRpZmljYXRlIFBvbGljeSBmb3VuZCBhdCBodHRwczovL3NlY3VyZS5pZGVudHJ1c3QuY29t
L2NlcnRpZmljYXRlcy9wb2xpY3kvdHMvaW5kZXguaHRtbDBKBgNVHR8EQzBBMD+gPaA7hjlo
dHRwOi8vdmFsaWRhdGlvbi5pZGVudHJ1c3QuY29tL2NybC9jb21tZXJjaWFscm9vdGNhMS5j
cmwwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMA4GA1UdDwEB/wQEAwIBhjAdBgNV
HQ4EFgQUpHPa72k1inXMoBl7CDL4a4nkQuwwDQYJKoZIhvcNAQELBQADggIBAA3hgq7S+/Tr
Yxl+D7ExI1Rdgq8fC9kiT7ofWlSaK/IMjgjoDfBbPGWvzdkmbSgYgXo8GxuAon9+HLIjNv68
BgUmbIjwj/SYaVz6chA25XZdjxzKk+hUkqCmfOn/twQJeRfxHg3I+0Sfwp5xs10YF0Robhrs
CRne6OUmh9mph0fE3b21k90OVnx9Hfr+YAV4ISrTA6045zQTKGzb370whliPLFo+hNL6XzEt
y5hfdFaWKtHIfpE994CLmTJI4SEbWq40d7TpAjCmKCPIVPq/+9GqggGvtakM5K3VXNc9VtKP
U9xYGCTDIYoeVBQ65JsdsdyM4PzDzAdINsv4vaF7yE03nh2jLV7XAkcqad9vS4EB4hKjFFsm
cwxa+ACUfkVWtBaWBqN4f/o1thsFJHEAu4Q6oRB6mYkzqrPigPazF2rgYw3lp0B1gSzCRj+j
RtErIVdMPeZ2p5Fdx7SNhBtabuhqmpJkFxwW9SBg6sHvy0HpzVvEiBpApFKG1ZHXMwzQl+pR
8P27wWDsblJU7Qgb8ZzGRK9l5GOFhxtN+oXZ4CCmunLMtaZ2vSai7du/VKrg64GGZNAKerEB
evjJVNFgeSnmUK9GB4kCZ7U5NWlU+2H87scntW4Q/0Y6vqQJcJeaMHg/dQnahTQ2p+hB1xJJ
K32GWIAucTFMSOKLbQHadIOiMYIDFDCCAxACAQEwTjA6MQswCQYDVQQGEwJVUzESMBAGA1UE
ChMJSWRlblRydXN0MRcwFQYDVQQDEw5UcnVzdElEIENBIEExMgIQQAFe4D0YrXK5RrCU85MI
7zANBglghkgBZQMEAgEFAKCCAZcwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG
9w0BCQUxDxcNMTcxMjIyMjEyNTUxWjAvBgkqhkiG9w0BCQQxIgQg4WN49q/5ftkbPDUCWjpy
J/zD0SHHJ+0mhTNaL2UnQV8wXQYJKwYBBAGCNxAEMVAwTjA6MQswCQYDVQQGEwJVUzESMBAG
A1UEChMJSWRlblRydXN0MRcwFQYDVQQDEw5UcnVzdElEIENBIEExMgIQQAFe4D0YrXK5RrCU
85MI7zBfBgsqhkiG9w0BCRACCzFQoE4wOjELMAkGA1UEBhMCVVMxEjAQBgNVBAoTCUlkZW5U
cnVzdDEXMBUGA1UEAxMOVHJ1c3RJRCBDQSBBMTICEEABXuA9GK1yuUawlPOTCO8wbAYJKoZI
hvcNAQkPMV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqG
SIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDANBgkq
hkiG9w0BAQEFAASCAQBKDXBdBrGWYiuQPMgo4VmmQaaWK4za36Rbwn8WrrY1SZRtqATIKxA7
i9xi1yj7SyvHBM624PvKo6ryx15hxx6qyFagBhCoI9maiL91/Rj0SLZuUYubdCXfx0Nih4BP
skowFkz3nPk53CypjY7W3joE1/EPwuT7oE8RneEzWW9B3FOaakkvRRHwRqeT40je98jFZJnc
Dpl0zWZr/u5zUZRQFv/G7oEKSoW5TFFfau/n01aZP+WWb65lHqRg8AM32jDhyafeKjlpWjQ7
CD+5wLvPpuHAGvZJSctDe8QmnOgmWxBjblkeKGMEypl35+AO2jweOEWJ2huT7tPYtQwpmyUM
AAAAAAAA
--------------ms040509000200000201060800--