[OpenAFS] DB quorum problems on 1.4/1.6 mixed cell
Arne Wiebalck
Arne.Wiebalck@cern.ch
Tue, 11 Feb 2014 17:25:18 +0000
--Apple-Mail=_7C5A9ACA-C33A-4EEE-8BE6-75CA563B40E3
Content-Type: multipart/alternative;
boundary="Apple-Mail=_5C05B5A2-9FD1-4C27-BE89-F1C81279AC1C"
--Apple-Mail=_5C05B5A2-9FD1-4C27-BE89-F1C81279AC1C
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=windows-1252
Thanks Andrew and Derrick!
We've seen the "major synchronisation error" as well when trying to =
provoke that problem.
This and the fact that we have the very same quorum issue about one year =
ago when
restarting the VLDB servers made the problem go away for some time seem =
to indicate
it's indeed the issue you mention. This was when we first added 1.6 =
servers to our cell, btw.
Apparently, we're pretty lucky ;)
I'll restart our VLDB servers =85
Thanks!
Arne
On Feb 11, 2014, at 5:48 PM, D Brashear <shadow@gmail.com>
wrote:
> The 1.4/1.6 issue is surely a red herring. You hit the nail when you =
mentioned negative transaction IDs. There was a bugfix early in the 1.6 =
series which handled that; you probably want to just restart all your =
dbservers so you can start counting up to rollover again, until you get =
to the point of updating them.
>=20
>=20
> On Tue, Feb 11, 2014 at 4:50 AM, Arne Wiebalck <Arne.Wiebalck@cern.ch> =
wrote:
> Hi,
>=20
> We've recently added some 1.6.6 servers into our cell which is mainly =
on 1.4.15 (i.e. most of the file servers
> and the DB servers). We now encounter quorum problems with our VLDB =
servers.
>=20
> The primary symptom is that releases fail with "u: no quorum elected".
>=20
> VLLog on the sync site shows at that moment:
> -->
> Tue Feb 11 09:51:47 2014 ubik:server 137.138.246.51 is back up: will =
be contacted through 137.138.246.51
> Tue Feb 11 09:51:47 2014 ubik:server 137.138.246.50 is back up: will =
be contacted through 137.138.246.50
> <--
> where 137.138.246.50 and 137.138.246.51 are the non-sync sites.
>=20
> We can relatively easy trigger this problem by moving volumes between =
1.4 and 1.6 based servers (1.4/1.4
> and 1.6/1.6 transfers seems OK): the move gets stuck and udebug shows=20=
>=20
> -->
> Host's addresses are: 137.138.128.148
> Host's 137.138.128.148 time is Tue Feb 11 09:38:02 2014
> Local time is Tue Feb 11 09:38:02 2014 (time differential 0 secs)
> Last yes vote for 137.138.128.148 was 5 secs ago (sync site);
> Last vote started 5 secs ago (at Tue Feb 11 09:37:57 2014)
> Local db version is 1392106724.154
> I am sync site until 54 secs from now (at Tue Feb 11 09:38:56 2014) (3 =
servers)
> Recovery state f
> I am currently managing write trans 1392106724.-1892301558
> Sync site's db version is 1392106724.154
> 0 locked pages, 0 of them for write
> There are write locks held
> Last time a new db version was labelled was:
> 1158 secs ago (at Tue Feb 11 09:18:44 2014)
>=20
> Server (137.138.246.51): (db 1392106724.153)
> last vote rcvd 20 secs ago (at Tue Feb 11 09:37:42 2014),
> last beacon sent 20 secs ago (at Tue Feb 11 09:37:42 2014), last =
vote was yes
> dbcurrent=3D0, up=3D0 beaconSince=3D0
>=20
> Server (137.138.246.50): (db 1392106724.154)
> last vote rcvd 6 secs ago (at Tue Feb 11 09:37:56 2014),
> last beacon sent 5 secs ago (at Tue Feb 11 09:37:57 2014), last =
vote was yes
> dbcurrent=3D1, up=3D1 beaconSince=3D1
> <--
>=20
> Note that the sync site has gone to Recovery state f and that the time =
at which the last vote was received
> on the other two servers has quite a time gap which gets larger with =
time. Is the negative trans ID ok?
>=20
> At some point the sync site loses its sync site state:
>=20
> -->
> Host's addresses are: 137.138.128.148
> Host's 137.138.128.148 time is Tue Feb 11 09:41:01 2014
> Local time is Tue Feb 11 09:41:02 2014 (time differential 1 secs)
> Last yes vote for 137.138.128.148 was 4 secs ago (sync site);
> Last vote started 4 secs ago (at Tue Feb 11 09:40:58 2014)
> Local db version is 1392106724.206
> I am not sync site
> Lowest host 137.138.128.148 was set 4 secs ago
> Sync host 137.138.128.148 was set 4 secs ago
> I am currently managing write trans 1392106724.-1892283756
> Sync site's db version is 1392106724.206
> 0 locked pages, 0 of them for write
> There are write locks held
> Last time a new db version was labelled was:
> 1337 secs ago (at Tue Feb 11 09:18:45 2014)
> <--
>=20
> so there is no sync site any longer and the vos command gets a no =
quorum error.
>=20
> As this also happens when we do not move volumes around (like at 3am), =
but other operations such
> as the backup touch the volumes, I would suspect that VLDB operations =
in general can trigger this.
>=20
> Is this a known issue?=20
>=20
> I had understood that it should be OK to run 1.4 and 1.6 file servers =
in parallel and that the DB servers
> could be updated after the file servers, but maybe that is not =
correct?
>=20
> Thanks!
> Arne
>=20
>=20
> --
> Arne Wiebalck
> CERN IT
>=20
>=20
>=20
>=20
> --=20
> D
--Apple-Mail=_5C05B5A2-9FD1-4C27-BE89-F1C81279AC1C
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=windows-1252
<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dwindows-1252"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><div>Thanks Andrew and Derrick!</div><div><br></div><div>We've seen =
the "major synchronisation error" as well when trying to provoke that =
problem.</div><div>This and the fact that we have the very same quorum =
issue about one year ago when</div><div>restarting the VLDB servers =
made the problem go away for some time seem to =
indicate</div><div>it's indeed the issue you mention. This was when we =
first added 1.6 servers to our cell, btw.</div><div>Apparently, we're =
pretty lucky ;)</div><div><br></div><div>I'll restart our VLDB servers =
=85</div><div><br></div><div>Thanks!</div><div> Arne</div><div><br></=
div><br><div><div>On Feb 11, 2014, at 5:48 PM, D Brashear <<a =
href=3D"mailto:shadow@gmail.com">shadow@gmail.com</a>></div><div> =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><meta http-equiv=3D"Content-Type" content=3D"text/html; =
charset=3Diso-8859-1"><div dir=3D"ltr">The 1.4/1.6 issue is surely a red =
herring. You hit the nail when you mentioned negative transaction IDs. =
There was a bugfix early in the 1.6 series which handled that; you =
probably want to just restart all your dbservers so you can start =
counting up to rollover again, until you get to the point of updating =
them.<br>
</div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On =
Tue, Feb 11, 2014 at 4:50 AM, Arne Wiebalck <span dir=3D"ltr"><<a =
href=3D"mailto:Arne.Wiebalck@cern.ch" =
target=3D"_blank">Arne.Wiebalck@cern.ch</a>></span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div =
style=3D"word-wrap:break-word">Hi,<div><br></div><div>We've recently =
added some 1.6.6 servers into our cell which is mainly on 1.4.15 (i.e. =
most of the file servers</div>
<div>and the DB servers). We now encounter quorum problems with our VLDB =
servers.</div><div><br></div><div>The primary symptom is that releases =
fail with "u: no quorum elected".</div><div><br></div><div>VLLog on the =
sync site shows at that moment:</div>
<div>--></div><div><div>Tue Feb 11 09:51:47 2014 ubik:server =
137.138.246.51 is back up: will be contacted through =
137.138.246.51</div><div>Tue Feb 11 09:51:47 2014 ubik:server =
137.138.246.50 is back up: will be contacted through =
137.138.246.50</div>
</div><div><--</div><div>where 137.138.246.50 =
and 137.138.246.51 are the non-sync =
sites.</div><div><br></div><div>We can relatively easy trigger this =
problem by moving volumes between 1.4 and 1.6 based servers =
(1.4/1.4</div>
<div>and 1.6/1.6 transfers seems OK): the move gets stuck and udebug =
shows </div><div><br></div><div>--></div><div><div>Host's =
addresses are: 137.138.128.148</div><div>Host's 137.138.128.148 time is =
Tue Feb 11 09:38:02 2014</div>
<div>Local time is Tue Feb 11 09:38:02 2014 (time differential 0 =
secs)</div><div>Last yes vote for 137.138.128.148 was 5 secs ago (sync =
site);</div><div>Last vote started 5 secs ago (at Tue Feb 11 09:37:57 =
2014)</div><div>
Local db version is 1392106724.154</div><div>I am sync site until 54 =
secs from now (at Tue Feb 11 09:38:56 2014) (3 =
servers)</div><div>Recovery state f</div><div>I am currently managing =
write trans 1392106724.-1892301558</div>
<div>Sync site's db version is 1392106724.154</div><div>0 locked pages, =
0 of them for write</div><div>There are write locks held</div><div>Last =
time a new db version was labelled was:</div><div> =
1158 secs ago (at Tue Feb 11 09:18:44 2014)</div>
<div><br></div><div>Server (137.138.246.51): (db =
1392106724.153)</div><div> last vote rcvd 20 secs ago (at =
Tue Feb 11 09:37:42 2014),</div><div> last beacon sent 20 =
secs ago (at Tue Feb 11 09:37:42 2014), last vote was yes</div>
<div> dbcurrent=3D0, up=3D0 =
beaconSince=3D0</div><div><br></div><div>Server (137.138.246.50): (db =
1392106724.154)</div><div> last vote rcvd 6 secs ago (at =
Tue Feb 11 09:37:56 2014),</div><div> last beacon sent 5 =
secs ago (at Tue Feb 11 09:37:57 2014), last vote was yes</div>
<div> dbcurrent=3D1, up=3D1 =
beaconSince=3D1</div></div><div><--</div><div><br></div><div>Note =
that the sync site has gone to Recovery state f and that the time at =
which the last vote was received</div><div>on the other two servers has =
quite a time gap which gets larger with time. Is the negative trans ID =
ok?</div>
<div><br></div><div>At some point the sync site loses its sync site =
state:</div><div><br></div><div>--></div><div><div>Host's addresses =
are: 137.138.128.148</div><div>Host's 137.138.128.148 time is Tue Feb 11 =
09:41:01 2014</div>
<div>Local time is Tue Feb 11 09:41:02 2014 (time differential 1 =
secs)</div><div>Last yes vote for 137.138.128.148 was 4 secs ago (sync =
site);</div><div>Last vote started 4 secs ago (at Tue Feb 11 09:40:58 =
2014)</div><div>
Local db version is 1392106724.206</div><div>I am not sync =
site</div><div>Lowest host 137.138.128.148 was set 4 secs =
ago</div><div>Sync host 137.138.128.148 was set 4 secs ago</div><div>I =
am currently managing write trans 1392106724.-1892283756</div>
<div>Sync site's db version is 1392106724.206</div><div>0 locked pages, =
0 of them for write</div><div>There are write locks held</div><div>Last =
time a new db version was labelled was:</div><div> =
1337 secs ago (at Tue Feb 11 09:18:45 2014)</div>
</div><div><--</div><div><br></div><div>so there is no sync site any =
longer and the vos command gets a no quorum =
error.</div><div><br></div><div>As this also happens when we do not =
move volumes around (like at 3am), but other operations such</div>
<div>as the backup touch the volumes, I would suspect that VLDB =
operations in general can trigger this.</div><div><br></div><div>Is this =
a known issue? </div><div><br></div><div>I had understood that it =
should be OK to run 1.4 and 1.6 file servers in parallel and that the DB =
servers</div>
<div>could be updated after the file servers, but maybe that is not =
correct?</div><div><br></div><div>Thanks!</div><div> Arne</div><div><=
br></div><div><br><div>
<div =
style=3D"text-indent:0px;letter-spacing:normal;font-variant:normal;text-al=
ign:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;t=
ext-transform:none;font-size:medium;white-space:normal;font-family:Helveti=
ca;word-wrap:break-word;word-spacing:0px">
<div =
style=3D"text-indent:0px;letter-spacing:normal;font-variant:normal;text-al=
ign:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;t=
ext-transform:none;font-size:medium;white-space:normal;font-family:Helveti=
ca;word-wrap:break-word;word-spacing:0px">
--</div><div =
style=3D"text-indent:0px;letter-spacing:normal;font-variant:normal;text-al=
ign:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;t=
ext-transform:none;font-size:medium;white-space:normal;font-family:Helveti=
ca;word-wrap:break-word;word-spacing:0px">
Arne Wiebalck<br>CERN IT</div></div>
</div>
<br></div></div></blockquote></div><br><br clear=3D"all"><br>-- <br><div =
dir=3D"ltr">D</div>
</div>
</blockquote></div><br></body></html>=
--Apple-Mail=_5C05B5A2-9FD1-4C27-BE89-F1C81279AC1C--
--Apple-Mail=_7C5A9ACA-C33A-4EEE-8BE6-75CA563B40E3
Content-Disposition: attachment; filename="smime.p7s"
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIINuDCCBi4w
ggUWoAMCAQICCmEPqkwAAAAAAAMwDQYJKoZIhvcNAQEFBQAwQTESMBAGCgmSJomT8ixkARkWAmNo
MRQwEgYKCZImiZPyLGQBGRYEY2VybjEVMBMGA1UEAxMMQ0VSTiBSb290IENBMB4XDTA2MTAwMzA5
MzYxM1oXDTE2MTAwMzA5NDYxM1owWTESMBAGCgmSJomT8ixkARkWAmNoMRQwEgYKCZImiZPyLGQB
GRYEY2VybjEtMCsGA1UEAxMkQ0VSTiBUcnVzdGVkIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwdFGqthhWlgUOSZ6C6hReNEVGzbjf2IQgxo7
/rOfOQHZH3krQPQ37fqFroEr46PrruymZ/U+QAzmESZQ4Z+nCfBhm7cEi0TIhihHd4cEPaPxawGR
T9Ck7BBk9za8TUljF6c/JodnIcmIrpbazEbSAS1KEXwETHDsTrQ7lJ+6SzDP4/oOwrHrgJx+tKsm
gOsFSbBEK4OYx1UYQpYS9OJQk2Sc0q4a/SCSu+xbN8ppmgV3WFytN8NW20n3NpCCWYPzo9rXmPRA
7a/c6mf+RV5gPCnUqeW6KUvix5kz9+X8/4SQV/fU12OPdRvtkqcC+PpiePK7bjMLQJEYwvchJrSz
AwIDAQABo4IDDjCCAwowDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUmMyS0EYwNoyw7ZgNclGp
R0zdviEwCwYDVR0PBAQDAgGGMBAGCSsGAQQBgjcVAQQDAgECMCMGCSsGAQQBgjcVAgQWBBT/Rljl
vgfrVK8GmAaYe+TbiXbJ7DBRBgNVHSAESjBIMEYGCisGAQQBYAoCAQEwODA2BggrBgEFBQcCARYq
aHR0cDovL2NhLmNlcm4uY2gvY2EvY3JsL3BvbGljeS9jcC1jcHMucGRmMBkGCSsGAQQBgjcUAgQM
HgoAUwB1AGIAQwBBMB8GA1UdIwQYMBaAFJgK9+w+7FnWHa2ZvLUBPt7spudQMIH8BgNVHR8EgfQw
gfEwge6ggeuggeiGLWh0dHA6Ly9jYS5jZXJuLmNoL2NhL2NybC9DRVJOJTIwUm9vdCUyMENBLmNy
bIaBtmxkYXA6Ly8vQ049Q0VSTiUyMFJvb3QlMjBDQSxDTj1jZXJucm9vdGNhLENOPUNEUCxDTj1Q
dWJsaWMlMjBLZXklMjBTZXJ2aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWNl
cm4sREM9Y2g/Y2VydGlmaWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNlP29iamVjdENsYXNzPWNSTERp
c3RyaWJ1dGlvblBvaW50MIIBBAYIKwYBBQUHAQEEgfcwgfQwRAYIKwYBBQUHMAKGOGh0dHA6Ly9j
YS5jZXJuLmNoL2NhL2NybC9jZXJucm9vdGNhX0NFUk4lMjBSb290JTIwQ0EuY3J0MIGrBggrBgEF
BQcwAoaBnmxkYXA6Ly8vQ049Q0VSTiUyMFJvb3QlMjBDQSxDTj1BSUEsQ049UHVibGljJTIwS2V5
JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1jZXJuLERDPWNoP2NB
Q2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1jZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MA0GCSqG
SIb3DQEBBQUAA4IBAQAfEzvOeYohKndmJqnVdiCqZ38tSBxOOPsKUHW4UY1jBfYMXbnZ9keFQFlK
/g5X4aZPNBEHXw0eKpQVsMhEPWQrvx8T/f7GwtU+JNQhkgK9tnezmHxYzWgEC9MXZhfYzFSwMIF6
kSKllmUTnN35uF1EnT8+64daje+yEVcpmM34p8Fw125/WpKnRmwNp0YkUk6uMti6Y6vOTHttzIN5
P6elGoat8sadMqrVnaMNzG8hGUvSkYivYBs7msAPuwmXgLvIkXWPW+MDFs+x5Kzx75ZHv3c2WoKg
UxL5KZH9QqiR7t8P6YBfYW6SpzyGRi4QHN/iOLhXZ06R6aPljLEOn41JMIIHgjCCBmqgAwIBAgIK
G2rSugACAAJRYzANBgkqhkiG9w0BAQUFADBZMRIwEAYKCZImiZPyLGQBGRYCY2gxFDASBgoJkiaJ
k/IsZAEZFgRjZXJuMS0wKwYDVQQDEyRDRVJOIFRydXN0ZWQgQ2VydGlmaWNhdGlvbiBBdXRob3Jp
dHkwHhcNMTMxMDI3MTAyNjQyWhcNMTQxMDI3MTAyNjQyWjCBjjESMBAGCgmSJomT8ixkARkWAmNo
MRQwEgYKCZImiZPyLGQBGRYEY2VybjEWMBQGA1UECxMNT3JnYW5pYyBVbml0czEOMAwGA1UECxMF
VXNlcnMxETAPBgNVBAMTCHdpZWJhbGNrMQ8wDQYDVQQDEwY1NTg1MTAxFjAUBgNVBAMTDUFybmUg
V2llYmFsY2swggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCxYDn8A4EeRTjaXpd8IrE3
Z3r/Fch6OlueWx7+XOBmxjvwCOMYlLt7WQeowLORXz7aaVyofoZHqmZirNnOQgTpzs/6UQT6DrXr
bKFZUsoJ/07hsk4QQyxo2WxCMUpgAMMmx13L2erSb2sNwtYhS51Mz1hFFmjE3wiAqftJOEww9vHB
rjeR8tlH0ifD7oFgGIhfAsx9q79aW8YOAHIBUnyrK1W0pP9qp3+ZwuH6sUXyDvN9BjIiEIxeDcb+
9MtAiENXxWwNfpWGmOousEyRNwyjXCpaerDvYsgbeGo+WyckGWleGxVH6B7glokA71OuEzUxUTHz
pazGDDLxwaP36+bFAgMBAAGjggQUMIIEEDAdBgNVHQ4EFgQUMiRKoPIkk3aChfZ+QXzfNmTYkUYw
HwYDVR0jBBgwFoAUmMyS0EYwNoyw7ZgNclGpR0zdviEwggEyBgNVHR8EggEpMIIBJTCCASGgggEd
oIIBGYZHaHR0cDovL2NhLmNlcm4uY2gvY2EvY3JsL0NFUk4lMjBUcnVzdGVkJTIwQ2VydGlmaWNh
dGlvbiUyMEF1dGhvcml0eS5jcmyGgc1sZGFwOi8vL0NOPUNFUk4lMjBUcnVzdGVkJTIwQ2VydGlm
aWNhdGlvbiUyMEF1dGhvcml0eSxDTj1DRVJOUEtJLENOPUNEUCxDTj1QdWJsaWMlMjBLZXklMjBT
ZXJ2aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWNlcm4sREM9Y2g/Y2VydGlm
aWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNlP29iamVjdENsYXNzPWNSTERpc3RyaWJ1dGlvblBvaW50
MIIBLwYIKwYBBQUHAQEEggEhMIIBHTCBxQYIKwYBBQUHMAKGgbhsZGFwOi8vL0NOPUNFUk4lMjBU
cnVzdGVkJTIwQ2VydGlmaWNhdGlvbiUyMEF1dGhvcml0eSxDTj1BSUEsQ049UHVibGljJTIwS2V5
JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1jZXJuLERDPWNoP2NB
Q2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1jZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MFMGCCsG
AQUFBzAChkdodHRwOi8vY2EuY2Vybi5jaC9jYS9jcmwvQ0VSTiUyMFRydXN0ZWQlMjBDZXJ0aWZp
Y2F0aW9uJTIwQXV0aG9yaXR5LmNydDAOBgNVHQ8BAf8EBAMCBaAwPQYJKwYBBAGCNxUHBDAwLgYm
KwYBBAGCNxUIg73QCYLtjQ2G7Ysrgd71N4WA0GIehb+6A4TEzEwCAWQCAQowKQYDVR0lBCIwIAYI
KwYBBQUHAwIGCCsGAQUFBwMEBgorBgEEAYI3CgMEMCUGA1UdIAQeMBwwDAYKKwYBBAFgCgIBAjAM
BgoqhkiG90wFAgIBMDUGCSsGAQQBgjcVCgQoMCYwCgYIKwYBBQUHAwIwCgYIKwYBBQUHAwQwDAYK
KwYBBAGCNwoDBDBHBgNVHREEQDA+oCUGCisGAQQBgjcUAgOgFwwVYXJuZS53aWViYWxja0BjZXJu
LmNogRVBcm5lLldpZWJhbGNrQGNlcm4uY2gwRAYJKoZIhvcNAQkPBDcwNTAOBggqhkiG9w0DAgIC
AIAwDgYIKoZIhvcNAwQCAgCAMAcGBSsOAwIHMAoGCCqGSIb3DQMHMA0GCSqGSIb3DQEBBQUAA4IB
AQAp+9JprmqNFehFL9K03clXIOFOu6b74SXweYvfX6BEFYxGSxOuAR7Y3o4R/RXtqIBBHPVesYQv
O1n3QmByEN12zZMspsvyzVVZnXa0lxr8HGg9ALDLZAv5FGaDuivXe7K9ZNwBdDcXg5LygZNyoFHX
z4dyyAQr2N9k/3ll1nkJ+C4J38GZpUOiRmzfvS4I30//HFBMK6eKBh3urD+lJMKGDqhdrMLoMVCw
UFkT0Pj+swGhdR/dz/ANaXU433wws3nKGnWp4kea84QJtZCHYp+C87XYKYoZzs0BTxWEY29vhiw5
xCU2quiUWWNJZ7uWBjeqTOzyj0CfHtCrddsJjnOmMYIC4TCCAt0CAQEwZzBZMRIwEAYKCZImiZPy
LGQBGRYCY2gxFDASBgoJkiaJk/IsZAEZFgRjZXJuMS0wKwYDVQQDEyRDRVJOIFRydXN0ZWQgQ2Vy
dGlmaWNhdGlvbiBBdXRob3JpdHkCChtq0roAAgACUWMwCQYFKw4DAhoFAKCCAU8wGAYJKoZIhvcN
AQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTQwMjExMTcyNTE4WjAjBgkqhkiG9w0B
CQQxFgQU8vmCHHkl5BMIM8COrLJDnTo80CAwdgYJKwYBBAGCNxAEMWkwZzBZMRIwEAYKCZImiZPy
LGQBGRYCY2gxFDASBgoJkiaJk/IsZAEZFgRjZXJuMS0wKwYDVQQDEyRDRVJOIFRydXN0ZWQgQ2Vy
dGlmaWNhdGlvbiBBdXRob3JpdHkCChtq0roAAgACUWMweAYLKoZIhvcNAQkQAgsxaaBnMFkxEjAQ
BgoJkiaJk/IsZAEZFgJjaDEUMBIGCgmSJomT8ixkARkWBGNlcm4xLTArBgNVBAMTJENFUk4gVHJ1
c3RlZCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eQIKG2rSugACAAJRYzANBgkqhkiG9w0BAQEFAASC
AQBnibUCkLfLtRWLMoQvM21AkGQVXrNktoXe64cLKlrU5O0139SSHb63c6TUdLYXCaUURLGGe+HW
imhLeJ5J1LQKLiEmnPYNVhL0aUidlSv1VpsGyZ64Sq9pcxA1ysH8KR0x/Ws0PLyiLt1DS8YDo1xV
WId9wfdNMGnfRG6BYFE+6sqFLnESnRvhGD27zoMmQ5oSUcxFSq7cKnZXPe/3JmPSpP5NybOGn8rh
evHVqJcP9JDJquJL4AQmVCzQ1pzlvuY4kbGWk2+m/aurerZKCWwGh9N1I9NE1BdAqx9kjx6RVbtD
3cix19h8ICHL+A+OEOvFBAOX1wQE6irc32pj37FqAAAAAAAA
--Apple-Mail=_7C5A9ACA-C33A-4EEE-8BE6-75CA563B40E3--