[OpenAFS] Procedure for changing database server IP addresses

[Jeffrey Altman]

This is a cryptographically signed message in MIME format.

--------------ms060507010909040509040502
Content-Type: multipart/mixed;
 boundary="------------99C064DED93110759CE94AE1"

This is a multi-part message in MIME format.
--------------99C064DED93110759CE94AE1
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 1/17/2017 3:45 PM, Stephen Joyce wrote:
> I know the current best-practice for changing the IP addresses of AFS
> database servers is don't do it.
>=20
> But assuming that I want/need to change IPs and have available hardware=
,
> is the use of clone dbservers the preferred method? I can tolerate shor=
t
> service interruptions of up to a few minutes as long as they're planned=

> for low-utilization times.

um, not really.

> Initial condition is 3 dbservers ("OLD") located via AFSDB & SRV,

I assume these servers are who, what and when as listed in the
CellServDB file distributed from

  http://www.central.org/csdb.html

and included in every OpenAFS distribution.

> running 1.6.x. Desired final condition is 3 dbservers ("NEW") with
> different IP addresses, also running 1.6.x (for now).

The first thing to be aware of is that any entries in the CellServDB
file take precedence over information provided via DNS.  For recent
OpenAFS releases the precedence order is

 * CellServDB file
 * DNS SRV
 * DNS AFSDB

The Unix cache manager only uses the IPv4 addresses that are provided in
the CellServDB file.  Whereas the Windows cache manager only uses the
host name and performs a DNS A query on the name to obtain the IP
address to use.

The CellServDB file contains entries for physics.unc.edu but not
cas.unc.edu.  Although physics.unc.edu lists the same DB servers as
cas.unc.edu.

The second thing to be aware of is that a UBIK quorum is defined by the
set of dbservers that share a common configuration.  Running OpenAFS
UBIK servers with a mixture of configurations can lead to more than one
dbserver believe it is the master.

The UBIK clone servers are interesting because they are documented as
being non-voting.  That isn't exactly true.  All UBIK dbservers must
maintain connectivity with every other UBIK dbserver in its
configuration.  What is special about clones is not that they don't vote
but that

 1. they cannot vote for themselves
 2. their vote for other servers are received and then discarded
 3. a clone cannot be the source of the best database.

Many sites have experienced problems with UBIK quorums consisting of
more than 3 servers.  Some sites have successfully run with as many as 5
servers.  It really depends on the number of number of clients and the
average rate of application RPCs (VL, PT, ...).

The primary benefit of using clones in OpenAFS is when you wish to
prevent a server with a low IPv4 address from being elected the
coordinator (aka sync site).

> I'm roughing out a procedure, but my current thinking involves..
>
>  add 3 NEW dbservers as r/o clones (restarting db procs)

I don't believe that using clones at this stage is helpful.

Also, you should leave all of the DB servers shutdown for at least 90
seconds when modifying the configuration.

>  modify DNS to show all 6 IPs.
>  'fs newcell' or restart all afsd's (including on servers)

You will also need to update the configuration and restart the
fileservers.  The fileservers are clients of the PT and VL servers but
use the server CellServDB file for their server info.

>  swap clone/non-clone roles so that NEW dbservers are r/w and OLD
> dbservers are r/o clones (restarting db procs). At this point, sync mus=
t
> be a non-clone, r/w "NEW" server.=20

Using clones to prevent the old servers from becoming coordinator is the
proper use.  You might want to consider only leaving one of the old
servers running at this point.  Be sure to shutdown all dbservers when
the configuration is changed.

> Verify with udebug. Any client afsd's
> not restarted/newcell'ed won't be able to make pt/vl changes.

The fileservers when started modify their VL entry. If their CellServDB
files are not updated as well, then they won't be able to registered.

>  modify DNS to show only 3 NEW IPs
>  'fs newcell' or restart of all afsd's (including on servers)
>=20
>  remove 3 OLD dbservers which must be r/o clones (restarting db procs).=

> Any client afsd's not restarted/newcell'ed won't be able to query
> pt/vlservers.

correct.

> Because it could take some time to restart/newcell all clients, I'm
> thinking of doing the clone addition/dns steps then waiting some time
> (week+) before doing the role swap and second dns change. Then waiting
> another period of time (week+) before doing the last removal.
>=20
> I'm assuming that I can use -auditlog (or even a packet sniffer) to see=

> what clients might still be using the OLD dbservers prior to the final
> decommissioning.

rxdebug <dbserver> <port> -peer

> Seems a bit too simple. What am I missing?

Good luck.

Jeffrey Altman


--------------99C064DED93110759CE94AE1
Content-Type: text/x-vcard; charset=utf-8;
 name="jaltman.vcf"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
 filename="jaltman.vcf"

begin:vcard
fn:Jeffrey Altman
n:Altman;Jeffrey
org:AuriStor, Inc.
adr:Suite 6B;;255 West 94Th Street;New York;New York;10025-6985;United St=
ates
email;internet:jaltman@auristor.com
title:Founder and CEO
tel;work:+1-212-769-9018
note;quoted-printable:LinkedIn: https://www.linkedin.com/in/jeffreyaltman=
=3D0D=3D0A=3D
	Skype: jeffrey.e.altman=3D0D=3D0A=3D
=09
url:https://www.auristor.com/
version:2.1
end:vcard


--------------99C064DED93110759CE94AE1--

--------------ms060507010909040509040502
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
DIMwggXqMIIE0qADAgECAhBAAVgjDHzXCy5hFo6GsQuLMA0GCSqGSIb3DQEBCwUAMDoxCzAJ
BgNVBAYTAlVTMRIwEAYDVQQKEwlJZGVuVHJ1c3QxFzAVBgNVBAMTDlRydXN0SUQgQ0EgQTEy
MB4XDTE2MTEwMjAzMTkzMFoXDTE3MTEwMjAzMTkzMFowgYYxLTArBgNVBAsMJFZlcmlmaWVk
IEVtYWlsOiBqYWx0bWFuQGF1cmlzdG9yLmNvbTEjMCEGCSqGSIb3DQEJARYUamFsdG1hbkBh
dXJpc3Rvci5jb20xMDAuBgoJkiaJk/IsZAEBEyA3RjAwMDAwMTAwMDAwMTU4MjMwQzdDQTcw
MDAwMDdCMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALtO/7veyX1tdA4aDvsO
c0gS/fICCTPlhS34dpRNmZYs2mvWb/xGRCKeflva8uZLFqsCE6ybMgz2IaB9rpowRcGSr/R7
CyY6HEBl3Rw0OkkslV7HQTxgp1fBXbGGoS8gOp6/ML64d5UJsMUM6NM59SzC77v8bi0sYkaT
q31SoiJrCcPiI0F2wZiESf1tguQONl3Emt9fhEokmlLRHAlv1DkRw2XeUOyUwVHIVgF1wnzR
4Ao86aYmym62Z669a6NPla3hqLmYk9w+ydTnzcWj0X6nhvaMxb7fhOUyj06Y5Mxhqye+dJg4
bPj/G+8OoNVY9l5h/qbJaWpSbDog4I0LxaUCAwEAAaOCAp0wggKZMA4GA1UdDwEB/wQEAwIF
oDCBhAYIKwYBBQUHAQEEeDB2MDAGCCsGAQUFBzABhiRodHRwOi8vY29tbWVyY2lhbC5vY3Nw
LmlkZW50cnVzdC5jb20wQgYIKwYBBQUHMAKGNmh0dHA6Ly92YWxpZGF0aW9uLmlkZW50cnVz
dC5jb20vY2VydHMvdHJ1c3RpZGNhYTEyLnA3YzAfBgNVHSMEGDAWgBSkc9rvaTWKdcygGXsI
MvhrieRC7DAJBgNVHRMEAjAAMIIBLAYDVR0gBIIBIzCCAR8wggEbBgtghkgBhvkvAAYLATCC
AQowSgYIKwYBBQUHAgEWPmh0dHBzOi8vc2VjdXJlLmlkZW50cnVzdC5jb20vY2VydGlmaWNh
dGVzL3BvbGljeS90cy9pbmRleC5odG1sMIG7BggrBgEFBQcCAjCBrhqBq1RoaXMgVHJ1c3RJ
RCBDZXJ0aWZpY2F0ZSBoYXMgYmVlbiBpc3N1ZWQgaW4gYWNjb3JkYW5jZSB3aXRoIApJZGVu
VHJ1c3QncyBUcnVzdElEIENlcnRpZmljYXRlIFBvbGljeSBmb3VuZCBhdCBodHRwczovL3Nl
Y3VyZS5pZGVudHJ1c3QuY29tL2NlcnRpZmljYXRlcy9wb2xpY3kvdHMvaW5kZXguaHRtbDBF
BgNVHR8EPjA8MDqgOKA2hjRodHRwOi8vdmFsaWRhdGlvbi5pZGVudHJ1c3QuY29tL2NybC90
cnVzdGlkY2FhMTIuY3JsMB8GA1UdEQQYMBaBFGphbHRtYW5AYXVyaXN0b3IuY29tMB0GA1Ud
DgQWBBT6okll+NyYQTyBKCigoq5jFBzw3DAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH
AwQwDQYJKoZIhvcNAQELBQADggEBABF9+YNU05P6tFiN8wvh52K+dgiUsQijFqN/tmwxqWHj
3TKxLwl8ZtK0F44FYcfeRXgE59bAxG8tZVRYBD7jY3qYEreri0x2DgJcA0oPfV1NHxM51t6h
ST09jpXEcthB4FJu3xWg8O6pB7Oq9Q9kkTjHfVLLru9B3BxrY+1NKZaSkaTBb0JGw6kCN2MM
rXrwPAdLlAWn5ywPRMqTIlgvhxtIQMv+dxpXhco9O5+ckMIdp1uBMJ82Hslfy3K9KCZvm6Cr
BG8C/fBo5TefVGuxKyOwROvup2gx1/KZWoIO0hyMNGzFPrijNZD5fAlZ11FcAJ9Why+pd5FY
mmJwln/BGPcwggaRMIIEeaADAgECAhEA+d5Wf8lNDHdw+WAbUtoVOzANBgkqhkiG9w0BAQsF
ADBKMQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MScwJQYDVQQDEx5JZGVuVHJ1
c3QgQ29tbWVyY2lhbCBSb290IENBIDEwHhcNMTUwMjE4MjIyNTE5WhcNMjMwMjE4MjIyNTE5
WjA6MQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MRcwFQYDVQQDEw5UcnVzdElE
IENBIEExMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANGRTTzPCic0kq5L6ZrU
JWt5LE/n6tbPXPhGt2Egv7plJMoEpvVJJDqGqDYymaAsd8Hn9ZMAuKUEFdlx5PgCkfu7jL5z
giMNnAFVD9PyrsuF+poqmlxhlQ06sFY2hbhQkVVQ00KCNgUzKcBUIvjv04w+fhNPkwGW5M7A
e5K5OGFGwOoRck9GG6MUVKvTNkBw2/vNMOd29VGVTtR0tjH5PS5yDXss48Yl1P4hDStO2L4w
TsW2P37QGD27//XGN8K6amWB6F2XOgff/PmlQjQOORT95PmLkwwvma5nj0AS0CVp8kv0K2RH
V7GonllKpFDMT0CkxMQKwoj+tWEWJTiDKSsCAwEAAaOCAoAwggJ8MIGJBggrBgEFBQcBAQR9
MHswMAYIKwYBBQUHMAGGJGh0dHA6Ly9jb21tZXJjaWFsLm9jc3AuaWRlbnRydXN0LmNvbTBH
BggrBgEFBQcwAoY7aHR0cDovL3ZhbGlkYXRpb24uaWRlbnRydXN0LmNvbS9yb290cy9jb21t
ZXJjaWFscm9vdGNhMS5wN2MwHwYDVR0jBBgwFoAU7UQZwNPwBovupHu+QucmVMiONnYwDwYD
VR0TAQH/BAUwAwEB/zCCASAGA1UdIASCARcwggETMIIBDwYEVR0gADCCAQUwggEBBggrBgEF
BQcCAjCB9DBFFj5odHRwczovL3NlY3VyZS5pZGVudHJ1c3QuY29tL2NlcnRpZmljYXRlcy9w
b2xpY3kvdHMvaW5kZXguaHRtbDADAgEBGoGqVGhpcyBUcnVzdElEIENlcnRpZmljYXRlIGhh
cyBiZWVuIGlzc3VlZCBpbiBhY2NvcmRhbmNlIHdpdGggSWRlblRydXN0J3MgVHJ1c3RJRCBD
ZXJ0aWZpY2F0ZSBQb2xpY3kgZm91bmQgYXQgaHR0cHM6Ly9zZWN1cmUuaWRlbnRydXN0LmNv
bS9jZXJ0aWZpY2F0ZXMvcG9saWN5L3RzL2luZGV4Lmh0bWwwSgYDVR0fBEMwQTA/oD2gO4Y5
aHR0cDovL3ZhbGlkYXRpb24uaWRlbnRydXN0LmNvbS9jcmwvY29tbWVyY2lhbHJvb3RjYTEu
Y3JsMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDAOBgNVHQ8BAf8EBAMCAYYwHQYD
VR0OBBYEFKRz2u9pNYp1zKAZewgy+GuJ5ELsMA0GCSqGSIb3DQEBCwUAA4ICAQAN4YKu0vv0
62MZfg+xMSNUXYKvHwvZIk+6H1pUmivyDI4I6A3wWzxlr83ZJm0oGIF6PBsbgKJ/fhyyIzb+
vAYFJmyI8I/0mGlc+nIQNuV2XY8cypPoVJKgpnzp/7cECXkX8R4NyPtEn8KecbNdGBdEaG4a
7AkZ3ujlJofZqYdHxN29tZPdDlZ8fR36/mAFeCEq0wOtOOc0Eyhs29+9MIZYjyxaPoTS+l8x
LcuYX3RWlirRyH6RPfeAi5kySOEhG1quNHe06QIwpigjyFT6v/vRqoIBr7WpDOSt1VzXPVbS
j1PcWBgkwyGKHlQUOuSbHbHcjOD8w8wHSDbL+L2he8hNN54doy1e1wJHKmnfb0uBAeISoxRb
JnMMWvgAlH5FVrQWlgajeH/6NbYbBSRxALuEOqEQepmJM6qz4oD2sxdq4GMN5adAdYEswkY/
o0bRKyFXTD3mdqeRXce0jYQbWm7oapqSZBccFvUgYOrB78tB6c1bxIgaQKRShtWR1zMM0Jfq
UfD9u8Fg7G5SVO0IG/GcxkSvZeRjhYcbTfqF2eAgprpyzLWmdr0mou3bv1Sq4OuBhmTQCnqx
AXr4yVTRYHkp5lCvRgeJAme1OTVpVPth/O7HJ7VuEP9GOr6kCXCXmjB4P3UJ2oU0NqfoQdcS
SSt9hliALnExTEjii20B2nSDojGCAxQwggMQAgEBME4wOjELMAkGA1UEBhMCVVMxEjAQBgNV
BAoTCUlkZW5UcnVzdDEXMBUGA1UEAxMOVHJ1c3RJRCBDQSBBMTICEEABWCMMfNcLLmEWjoax
C4swDQYJYIZIAWUDBAIBBQCgggGXMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZI
hvcNAQkFMQ8XDTE3MDExODA3MzkwOFowLwYJKoZIhvcNAQkEMSIEIJngbw6cuKXR/M2/AYtU
/7/N9j5WwUgL9OI0BStNO/1NMF0GCSsGAQQBgjcQBDFQME4wOjELMAkGA1UEBhMCVVMxEjAQ
BgNVBAoTCUlkZW5UcnVzdDEXMBUGA1UEAxMOVHJ1c3RJRCBDQSBBMTICEEABWCMMfNcLLmEW
joaxC4swXwYLKoZIhvcNAQkQAgsxUKBOMDoxCzAJBgNVBAYTAlVTMRIwEAYDVQQKEwlJZGVu
VHJ1c3QxFzAVBgNVBAMTDlRydXN0SUQgQ0EgQTEyAhBAAVgjDHzXCy5hFo6GsQuLMGwGCSqG
SIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggq
hkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwDQYJ
KoZIhvcNAQEBBQAEggEAkPmZnDmyJvLlf4BeAFoIPW+RnOUfpinEfXgvlLRwGkl1bXzUpDHd
xzm1Oyg1OkjaHxcL0P8XfdegIaTpVEOWTbDkUs0rzHc3ngfAtX2i7oiGEV8NN1An4POlJEFu
kGmvy2Vfv+KoQG3q5SHuYFOhFoSBEZfXJAgGs75O+avuLgfN3y5Z57cRSyVaQVESNafB8p+m
pv0BRUyE41LkLs/LaqgACIhV4JBKjqzljHKo/ZinLuifdR90LgZ6/Vwo4iv2IrmIXi6F/hwP
XRy4uToagMRDRBYC8RC1q1EVWQ3LqKNk/FDQmNgVT726lazzJBLQySMoqAmLVSUe4v/hKATn
9QAAAAAAAA==
--------------ms060507010909040509040502--