[OpenAFS] DB quorum problems on 1.4/1.6 mixed cell
Arne Wiebalck
Arne.Wiebalck@cern.ch
Thu, 13 Feb 2014 15:29:14 +0000
--Apple-Mail=_3E770697-DA71-4FD5-A893-4969390FA44F
Content-Type: multipart/alternative;
boundary="Apple-Mail=_1753BCB7-4222-424E-993D-5E5DFF340198"
--Apple-Mail=_1753BCB7-4222-424E-993D-5E5DFF340198
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=windows-1252
Just to confirm: I restarted all VL servers in our cell, the transaction =
IDs were reset and
so far I wasn't able to reproduce the problem. So it seems that it was =
indeed the
negative transaction ID problem in 1.4 VL servers mentioned earlier in =
this thread.
Cheers,
Arne=20
On Feb 11, 2014, at 6:25 PM, Arne Wiebalck <Arne.Wiebalck@cern.ch> =
wrote:
> Thanks Andrew and Derrick!
>=20
> We've seen the "major synchronisation error" as well when trying to =
provoke that problem.
> This and the fact that we have the very same quorum issue about one =
year ago when
> restarting the VLDB servers made the problem go away for some time =
seem to indicate
> it's indeed the issue you mention. This was when we first added 1.6 =
servers to our cell, btw.
> Apparently, we're pretty lucky ;)
>=20
> I'll restart our VLDB servers =85
>=20
> Thanks!
> Arne
>=20
>=20
> On Feb 11, 2014, at 5:48 PM, D Brashear <shadow@gmail.com>
> wrote:
>=20
>> The 1.4/1.6 issue is surely a red herring. You hit the nail when you =
mentioned negative transaction IDs. There was a bugfix early in the 1.6 =
series which handled that; you probably want to just restart all your =
dbservers so you can start counting up to rollover again, until you get =
to the point of updating them.
>>=20
>>=20
>> On Tue, Feb 11, 2014 at 4:50 AM, Arne Wiebalck =
<Arne.Wiebalck@cern.ch> wrote:
>> Hi,
>>=20
>> We've recently added some 1.6.6 servers into our cell which is mainly =
on 1.4.15 (i.e. most of the file servers
>> and the DB servers). We now encounter quorum problems with our VLDB =
servers.
>>=20
>> The primary symptom is that releases fail with "u: no quorum =
elected".
>>=20
>> VLLog on the sync site shows at that moment:
>> -->
>> Tue Feb 11 09:51:47 2014 ubik:server 137.138.246.51 is back up: will =
be contacted through 137.138.246.51
>> Tue Feb 11 09:51:47 2014 ubik:server 137.138.246.50 is back up: will =
be contacted through 137.138.246.50
>> <--
>> where 137.138.246.50 and 137.138.246.51 are the non-sync sites.
>>=20
>> We can relatively easy trigger this problem by moving volumes between =
1.4 and 1.6 based servers (1.4/1.4
>> and 1.6/1.6 transfers seems OK): the move gets stuck and udebug shows=20=
>>=20
>> -->
>> Host's addresses are: 137.138.128.148
>> Host's 137.138.128.148 time is Tue Feb 11 09:38:02 2014
>> Local time is Tue Feb 11 09:38:02 2014 (time differential 0 secs)
>> Last yes vote for 137.138.128.148 was 5 secs ago (sync site);
>> Last vote started 5 secs ago (at Tue Feb 11 09:37:57 2014)
>> Local db version is 1392106724.154
>> I am sync site until 54 secs from now (at Tue Feb 11 09:38:56 2014) =
(3 servers)
>> Recovery state f
>> I am currently managing write trans 1392106724.-1892301558
>> Sync site's db version is 1392106724.154
>> 0 locked pages, 0 of them for write
>> There are write locks held
>> Last time a new db version was labelled was:
>> 1158 secs ago (at Tue Feb 11 09:18:44 2014)
>>=20
>> Server (137.138.246.51): (db 1392106724.153)
>> last vote rcvd 20 secs ago (at Tue Feb 11 09:37:42 2014),
>> last beacon sent 20 secs ago (at Tue Feb 11 09:37:42 2014), last =
vote was yes
>> dbcurrent=3D0, up=3D0 beaconSince=3D0
>>=20
>> Server (137.138.246.50): (db 1392106724.154)
>> last vote rcvd 6 secs ago (at Tue Feb 11 09:37:56 2014),
>> last beacon sent 5 secs ago (at Tue Feb 11 09:37:57 2014), last =
vote was yes
>> dbcurrent=3D1, up=3D1 beaconSince=3D1
>> <--
>>=20
>> Note that the sync site has gone to Recovery state f and that the =
time at which the last vote was received
>> on the other two servers has quite a time gap which gets larger with =
time. Is the negative trans ID ok?
>>=20
>> At some point the sync site loses its sync site state:
>>=20
>> -->
>> Host's addresses are: 137.138.128.148
>> Host's 137.138.128.148 time is Tue Feb 11 09:41:01 2014
>> Local time is Tue Feb 11 09:41:02 2014 (time differential 1 secs)
>> Last yes vote for 137.138.128.148 was 4 secs ago (sync site);
>> Last vote started 4 secs ago (at Tue Feb 11 09:40:58 2014)
>> Local db version is 1392106724.206
>> I am not sync site
>> Lowest host 137.138.128.148 was set 4 secs ago
>> Sync host 137.138.128.148 was set 4 secs ago
>> I am currently managing write trans 1392106724.-1892283756
>> Sync site's db version is 1392106724.206
>> 0 locked pages, 0 of them for write
>> There are write locks held
>> Last time a new db version was labelled was:
>> 1337 secs ago (at Tue Feb 11 09:18:45 2014)
>> <--
>>=20
>> so there is no sync site any longer and the vos command gets a no =
quorum error.
>>=20
>> As this also happens when we do not move volumes around (like at =
3am), but other operations such
>> as the backup touch the volumes, I would suspect that VLDB operations =
in general can trigger this.
>>=20
>> Is this a known issue?=20
>>=20
>> I had understood that it should be OK to run 1.4 and 1.6 file servers =
in parallel and that the DB servers
>> could be updated after the file servers, but maybe that is not =
correct?
>>=20
>> Thanks!
>> Arne
>>=20
>>=20
>> --
>> Arne Wiebalck
>> CERN IT
>>=20
>>=20
>>=20
>>=20
>> --=20
>> D
>=20
--Apple-Mail=_1753BCB7-4222-424E-993D-5E5DFF340198
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=windows-1252
<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dwindows-1252"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Just =
to confirm: I restarted all VL servers in our cell, the transaction IDs =
were reset and<div>so far I wasn't able to reproduce the problem. So it =
seems that it was indeed the</div><div>negative transaction ID =
problem in 1.4 VL servers mentioned earlier in this =
thread.</div><div><br></div><div>Cheers,</div><div> Arne </div><=
div><div><br></div><div><div apple-content-edited=3D"true">
</div>
<br><div><div>On Feb 11, 2014, at 6:25 PM, Arne Wiebalck <<a =
href=3D"mailto:Arne.Wiebalck@cern.ch">Arne.Wiebalck@cern.ch</a>> =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dwindows-1252"><div style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><div>Thanks Andrew and Derrick!</div><div><br></div><div>We've seen =
the "major synchronisation error" as well when trying to provoke that =
problem.</div><div>This and the fact that we have the very same quorum =
issue about one year ago when</div><div>restarting the VLDB servers =
made the problem go away for some time seem to =
indicate</div><div>it's indeed the issue you mention. This was when we =
first added 1.6 servers to our cell, btw.</div><div>Apparently, we're =
pretty lucky ;)</div><div><br></div><div>I'll restart our VLDB servers =
=85</div><div><br></div><div>Thanks!</div><div> Arne</div><div><br></=
div><br><div><div>On Feb 11, 2014, at 5:48 PM, D Brashear <<a =
href=3D"mailto:shadow@gmail.com">shadow@gmail.com</a>></div><div> =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><meta http-equiv=3D"Content-Type" content=3D"text/html; =
charset=3Diso-8859-1"><div dir=3D"ltr">The 1.4/1.6 issue is surely a red =
herring. You hit the nail when you mentioned negative transaction IDs. =
There was a bugfix early in the 1.6 series which handled that; you =
probably want to just restart all your dbservers so you can start =
counting up to rollover again, until you get to the point of updating =
them.<br>
</div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On =
Tue, Feb 11, 2014 at 4:50 AM, Arne Wiebalck <span dir=3D"ltr"><<a =
href=3D"mailto:Arne.Wiebalck@cern.ch" =
target=3D"_blank">Arne.Wiebalck@cern.ch</a>></span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div =
style=3D"word-wrap:break-word">Hi,<div><br></div><div>We've recently =
added some 1.6.6 servers into our cell which is mainly on 1.4.15 (i.e. =
most of the file servers</div>
<div>and the DB servers). We now encounter quorum problems with our VLDB =
servers.</div><div><br></div><div>The primary symptom is that releases =
fail with "u: no quorum elected".</div><div><br></div><div>VLLog on the =
sync site shows at that moment:</div>
<div>--></div><div><div>Tue Feb 11 09:51:47 2014 ubik:server =
137.138.246.51 is back up: will be contacted through =
137.138.246.51</div><div>Tue Feb 11 09:51:47 2014 ubik:server =
137.138.246.50 is back up: will be contacted through =
137.138.246.50</div>
</div><div><--</div><div>where 137.138.246.50 =
and 137.138.246.51 are the non-sync =
sites.</div><div><br></div><div>We can relatively easy trigger this =
problem by moving volumes between 1.4 and 1.6 based servers =
(1.4/1.4</div>
<div>and 1.6/1.6 transfers seems OK): the move gets stuck and udebug =
shows </div><div><br></div><div>--></div><div><div>Host's =
addresses are: 137.138.128.148</div><div>Host's 137.138.128.148 time is =
Tue Feb 11 09:38:02 2014</div>
<div>Local time is Tue Feb 11 09:38:02 2014 (time differential 0 =
secs)</div><div>Last yes vote for 137.138.128.148 was 5 secs ago (sync =
site);</div><div>Last vote started 5 secs ago (at Tue Feb 11 09:37:57 =
2014)</div><div>
Local db version is 1392106724.154</div><div>I am sync site until 54 =
secs from now (at Tue Feb 11 09:38:56 2014) (3 =
servers)</div><div>Recovery state f</div><div>I am currently managing =
write trans 1392106724.-1892301558</div>
<div>Sync site's db version is 1392106724.154</div><div>0 locked pages, =
0 of them for write</div><div>There are write locks held</div><div>Last =
time a new db version was labelled was:</div><div> =
1158 secs ago (at Tue Feb 11 09:18:44 2014)</div>
<div><br></div><div>Server (137.138.246.51): (db =
1392106724.153)</div><div> last vote rcvd 20 secs ago (at =
Tue Feb 11 09:37:42 2014),</div><div> last beacon sent 20 =
secs ago (at Tue Feb 11 09:37:42 2014), last vote was yes</div>
<div> dbcurrent=3D0, up=3D0 =
beaconSince=3D0</div><div><br></div><div>Server (137.138.246.50): (db =
1392106724.154)</div><div> last vote rcvd 6 secs ago (at =
Tue Feb 11 09:37:56 2014),</div><div> last beacon sent 5 =
secs ago (at Tue Feb 11 09:37:57 2014), last vote was yes</div>
<div> dbcurrent=3D1, up=3D1 =
beaconSince=3D1</div></div><div><--</div><div><br></div><div>Note =
that the sync site has gone to Recovery state f and that the time at =
which the last vote was received</div><div>on the other two servers has =
quite a time gap which gets larger with time. Is the negative trans ID =
ok?</div>
<div><br></div><div>At some point the sync site loses its sync site =
state:</div><div><br></div><div>--></div><div><div>Host's addresses =
are: 137.138.128.148</div><div>Host's 137.138.128.148 time is Tue Feb 11 =
09:41:01 2014</div>
<div>Local time is Tue Feb 11 09:41:02 2014 (time differential 1 =
secs)</div><div>Last yes vote for 137.138.128.148 was 4 secs ago (sync =
site);</div><div>Last vote started 4 secs ago (at Tue Feb 11 09:40:58 =
2014)</div><div>
Local db version is 1392106724.206</div><div>I am not sync =
site</div><div>Lowest host 137.138.128.148 was set 4 secs =
ago</div><div>Sync host 137.138.128.148 was set 4 secs ago</div><div>I =
am currently managing write trans 1392106724.-1892283756</div>
<div>Sync site's db version is 1392106724.206</div><div>0 locked pages, =
0 of them for write</div><div>There are write locks held</div><div>Last =
time a new db version was labelled was:</div><div> =
1337 secs ago (at Tue Feb 11 09:18:45 2014)</div>
</div><div><--</div><div><br></div><div>so there is no sync site any =
longer and the vos command gets a no quorum =
error.</div><div><br></div><div>As this also happens when we do not =
move volumes around (like at 3am), but other operations such</div>
<div>as the backup touch the volumes, I would suspect that VLDB =
operations in general can trigger this.</div><div><br></div><div>Is this =
a known issue? </div><div><br></div><div>I had understood that it =
should be OK to run 1.4 and 1.6 file servers in parallel and that the DB =
servers</div>
<div>could be updated after the file servers, but maybe that is not =
correct?</div><div><br></div><div>Thanks!</div><div> Arne</div><div><=
br></div><div><br><div>
<div =
style=3D"text-indent:0px;letter-spacing:normal;font-variant:normal;text-al=
ign:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;t=
ext-transform:none;font-size:medium;white-space:normal;font-family:Helveti=
ca;word-wrap:break-word;word-spacing:0px">
<div =
style=3D"text-indent:0px;letter-spacing:normal;font-variant:normal;text-al=
ign:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;t=
ext-transform:none;font-size:medium;white-space:normal;font-family:Helveti=
ca;word-wrap:break-word;word-spacing:0px">
--</div><div =
style=3D"text-indent:0px;letter-spacing:normal;font-variant:normal;text-al=
ign:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;t=
ext-transform:none;font-size:medium;white-space:normal;font-family:Helveti=
ca;word-wrap:break-word;word-spacing:0px">
Arne Wiebalck<br>CERN IT</div></div>
</div>
<br></div></div></blockquote></div><br><br clear=3D"all"><br>-- <br><div =
dir=3D"ltr">D</div>
</div>
=
</blockquote></div><br></div></blockquote></div><br></div></div></body></h=
tml>=
--Apple-Mail=_1753BCB7-4222-424E-993D-5E5DFF340198--
--Apple-Mail=_3E770697-DA71-4FD5-A893-4969390FA44F
Content-Disposition: attachment; filename="smime.p7s"
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIINuDCCBi4w
ggUWoAMCAQICCmEPqkwAAAAAAAMwDQYJKoZIhvcNAQEFBQAwQTESMBAGCgmSJomT8ixkARkWAmNo
MRQwEgYKCZImiZPyLGQBGRYEY2VybjEVMBMGA1UEAxMMQ0VSTiBSb290IENBMB4XDTA2MTAwMzA5
MzYxM1oXDTE2MTAwMzA5NDYxM1owWTESMBAGCgmSJomT8ixkARkWAmNoMRQwEgYKCZImiZPyLGQB
GRYEY2VybjEtMCsGA1UEAxMkQ0VSTiBUcnVzdGVkIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwdFGqthhWlgUOSZ6C6hReNEVGzbjf2IQgxo7
/rOfOQHZH3krQPQ37fqFroEr46PrruymZ/U+QAzmESZQ4Z+nCfBhm7cEi0TIhihHd4cEPaPxawGR
T9Ck7BBk9za8TUljF6c/JodnIcmIrpbazEbSAS1KEXwETHDsTrQ7lJ+6SzDP4/oOwrHrgJx+tKsm
gOsFSbBEK4OYx1UYQpYS9OJQk2Sc0q4a/SCSu+xbN8ppmgV3WFytN8NW20n3NpCCWYPzo9rXmPRA
7a/c6mf+RV5gPCnUqeW6KUvix5kz9+X8/4SQV/fU12OPdRvtkqcC+PpiePK7bjMLQJEYwvchJrSz
AwIDAQABo4IDDjCCAwowDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUmMyS0EYwNoyw7ZgNclGp
R0zdviEwCwYDVR0PBAQDAgGGMBAGCSsGAQQBgjcVAQQDAgECMCMGCSsGAQQBgjcVAgQWBBT/Rljl
vgfrVK8GmAaYe+TbiXbJ7DBRBgNVHSAESjBIMEYGCisGAQQBYAoCAQEwODA2BggrBgEFBQcCARYq
aHR0cDovL2NhLmNlcm4uY2gvY2EvY3JsL3BvbGljeS9jcC1jcHMucGRmMBkGCSsGAQQBgjcUAgQM
HgoAUwB1AGIAQwBBMB8GA1UdIwQYMBaAFJgK9+w+7FnWHa2ZvLUBPt7spudQMIH8BgNVHR8EgfQw
gfEwge6ggeuggeiGLWh0dHA6Ly9jYS5jZXJuLmNoL2NhL2NybC9DRVJOJTIwUm9vdCUyMENBLmNy
bIaBtmxkYXA6Ly8vQ049Q0VSTiUyMFJvb3QlMjBDQSxDTj1jZXJucm9vdGNhLENOPUNEUCxDTj1Q
dWJsaWMlMjBLZXklMjBTZXJ2aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWNl
cm4sREM9Y2g/Y2VydGlmaWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNlP29iamVjdENsYXNzPWNSTERp
c3RyaWJ1dGlvblBvaW50MIIBBAYIKwYBBQUHAQEEgfcwgfQwRAYIKwYBBQUHMAKGOGh0dHA6Ly9j
YS5jZXJuLmNoL2NhL2NybC9jZXJucm9vdGNhX0NFUk4lMjBSb290JTIwQ0EuY3J0MIGrBggrBgEF
BQcwAoaBnmxkYXA6Ly8vQ049Q0VSTiUyMFJvb3QlMjBDQSxDTj1BSUEsQ049UHVibGljJTIwS2V5
JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1jZXJuLERDPWNoP2NB
Q2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1jZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MA0GCSqG
SIb3DQEBBQUAA4IBAQAfEzvOeYohKndmJqnVdiCqZ38tSBxOOPsKUHW4UY1jBfYMXbnZ9keFQFlK
/g5X4aZPNBEHXw0eKpQVsMhEPWQrvx8T/f7GwtU+JNQhkgK9tnezmHxYzWgEC9MXZhfYzFSwMIF6
kSKllmUTnN35uF1EnT8+64daje+yEVcpmM34p8Fw125/WpKnRmwNp0YkUk6uMti6Y6vOTHttzIN5
P6elGoat8sadMqrVnaMNzG8hGUvSkYivYBs7msAPuwmXgLvIkXWPW+MDFs+x5Kzx75ZHv3c2WoKg
UxL5KZH9QqiR7t8P6YBfYW6SpzyGRi4QHN/iOLhXZ06R6aPljLEOn41JMIIHgjCCBmqgAwIBAgIK
HYLrgQACAAHOszANBgkqhkiG9w0BAQUFADBZMRIwEAYKCZImiZPyLGQBGRYCY2gxFDASBgoJkiaJ
k/IsZAEZFgRjZXJuMS0wKwYDVQQDEyRDRVJOIFRydXN0ZWQgQ2VydGlmaWNhdGlvbiBBdXRob3Jp
dHkwHhcNMTMwMzI2MDkxNjM1WhcNMTQwMzI2MDkxNjM1WjCBjjESMBAGCgmSJomT8ixkARkWAmNo
MRQwEgYKCZImiZPyLGQBGRYEY2VybjEWMBQGA1UECxMNT3JnYW5pYyBVbml0czEOMAwGA1UECxMF
VXNlcnMxETAPBgNVBAMTCHdpZWJhbGNrMQ8wDQYDVQQDEwY1NTg1MTAxFjAUBgNVBAMTDUFybmUg
V2llYmFsY2swggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDKWiuJsxwe2ZAkHn+FKzTT
VBNe7lhl2PmTjPmZsedrVI0gZkUpjlb6x21avo3S+kacxPpVdNVmidE1k7Dmn+MLLwli8Ha+ePqN
e2kD33K01bjf8G1l5KquZwMsJKB2gSwSs1JK5uq3qcB5b1mrzPVOqOqNPUMg1NUkEjYql4ooG5Cj
GfVwlAhRarBssM07JtnToszp/fiCqttCR4pAAjURvQKV3T+Epiexwgzkv57mLAXe3vNf8giDigJP
zvNc+3pGqA8faVOGSqOo+9m6NheYPH4pcYcnhgA177HRfHxJ9fqzEahLNzXjoa9vtW8cpbSxyzY1
wbOCvQW1hz/uCP/5AgMBAAGjggQUMIIEEDAdBgNVHQ4EFgQUFIvKPnfKzu1UVQjsxgpwQPtpkwAw
HwYDVR0jBBgwFoAUmMyS0EYwNoyw7ZgNclGpR0zdviEwggEyBgNVHR8EggEpMIIBJTCCASGgggEd
oIIBGYZHaHR0cDovL2NhLmNlcm4uY2gvY2EvY3JsL0NFUk4lMjBUcnVzdGVkJTIwQ2VydGlmaWNh
dGlvbiUyMEF1dGhvcml0eS5jcmyGgc1sZGFwOi8vL0NOPUNFUk4lMjBUcnVzdGVkJTIwQ2VydGlm
aWNhdGlvbiUyMEF1dGhvcml0eSxDTj1DRVJOUEtJLENOPUNEUCxDTj1QdWJsaWMlMjBLZXklMjBT
ZXJ2aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWNlcm4sREM9Y2g/Y2VydGlm
aWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNlP29iamVjdENsYXNzPWNSTERpc3RyaWJ1dGlvblBvaW50
MIIBLwYIKwYBBQUHAQEEggEhMIIBHTCBxQYIKwYBBQUHMAKGgbhsZGFwOi8vL0NOPUNFUk4lMjBU
cnVzdGVkJTIwQ2VydGlmaWNhdGlvbiUyMEF1dGhvcml0eSxDTj1BSUEsQ049UHVibGljJTIwS2V5
JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1jZXJuLERDPWNoP2NB
Q2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1jZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MFMGCCsG
AQUFBzAChkdodHRwOi8vY2EuY2Vybi5jaC9jYS9jcmwvQ0VSTiUyMFRydXN0ZWQlMjBDZXJ0aWZp
Y2F0aW9uJTIwQXV0aG9yaXR5LmNydDAOBgNVHQ8BAf8EBAMCBaAwPQYJKwYBBAGCNxUHBDAwLgYm
KwYBBAGCNxUIg73QCYLtjQ2G7Ysrgd71N4WA0GIehb+6A4TEzEwCAWQCAQowKQYDVR0lBCIwIAYI
KwYBBQUHAwIGCCsGAQUFBwMEBgorBgEEAYI3CgMEMCUGA1UdIAQeMBwwDAYKKwYBBAFgCgIBAjAM
BgoqhkiG90wFAgIBMDUGCSsGAQQBgjcVCgQoMCYwCgYIKwYBBQUHAwIwCgYIKwYBBQUHAwQwDAYK
KwYBBAGCNwoDBDBHBgNVHREEQDA+oCUGCisGAQQBgjcUAgOgFwwVYXJuZS53aWViYWxja0BjZXJu
LmNogRVBcm5lLldpZWJhbGNrQGNlcm4uY2gwRAYJKoZIhvcNAQkPBDcwNTAOBggqhkiG9w0DAgIC
AIAwDgYIKoZIhvcNAwQCAgCAMAcGBSsOAwIHMAoGCCqGSIb3DQMHMA0GCSqGSIb3DQEBBQUAA4IB
AQBPiPoalqg44BSHOfc5XKMLl8dbU2ixH84cWFWmv8WrkM1oTVOopGUi9dO2qiN3PbzgkW3h4yB6
hEvTn7CVCfuzPXjEb9dQEJEmVheW8xA96GYWYWdXqdNHWtPMJB4tFh0skOMo35DwavZlrE30LLR+
BK8faePfnQ6PgPrdBESQaU4ZemLS5Q/eKfz2d4kSt5+70rj8ypD7pmMdUu/n1TFxuG3UNCFSMs6G
SUV0/PBjyV1O8jGtVbE9vNZE+NTuzWiYgI5Z6Pl6cHefI6miSn++PDjHzfaIPi0san0KMOXLqvA8
ZfjtAb2QMEvvKziTdg9Q4Y7wHCdi7aqKWmTbwyqUMYIC4TCCAt0CAQEwZzBZMRIwEAYKCZImiZPy
LGQBGRYCY2gxFDASBgoJkiaJk/IsZAEZFgRjZXJuMS0wKwYDVQQDEyRDRVJOIFRydXN0ZWQgQ2Vy
dGlmaWNhdGlvbiBBdXRob3JpdHkCCh2C64EAAgABzrMwCQYFKw4DAhoFAKCCAU8wGAYJKoZIhvcN
AQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTQwMjEzMTUyOTE3WjAjBgkqhkiG9w0B
CQQxFgQUsnHakSmJ6KbWl14IiUPXxojCitswdgYJKwYBBAGCNxAEMWkwZzBZMRIwEAYKCZImiZPy
LGQBGRYCY2gxFDASBgoJkiaJk/IsZAEZFgRjZXJuMS0wKwYDVQQDEyRDRVJOIFRydXN0ZWQgQ2Vy
dGlmaWNhdGlvbiBBdXRob3JpdHkCCh2C64EAAgABzrMweAYLKoZIhvcNAQkQAgsxaaBnMFkxEjAQ
BgoJkiaJk/IsZAEZFgJjaDEUMBIGCgmSJomT8ixkARkWBGNlcm4xLTArBgNVBAMTJENFUk4gVHJ1
c3RlZCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eQIKHYLrgQACAAHOszANBgkqhkiG9w0BAQEFAASC
AQAzwizCXY4Vs1ZXBBS39XPC6GoFcP0WYDr+OZkoX8K2vbpiuIEEsRKqS0P4s7fMc8RlNKlKNCI7
ZDTbzJ51nrfbQXn2YUXoeaHegbBNfFWHMAdH7owGtNWudM02/nqcL/FwMf+PZTU2S/e9WFQ83gl3
goXFe+aH7BYnLxyXRWjH6ycRgq4R+z5oReGWo9hY9SWGZ4FOugAV3J2lCstpZqyR8UTewIbVnJqi
tVLX4n0E7umm4Kg1Swk5zbL7+/sWbSW6ZgcE6JBzHwjiQ0Wm+KlSNjZkMuOy3GZv0qqFgu7NtRma
jsEv5getUEE9EYFnWsCQs7ruIM8Y2u2+NhLFXBn5AAAAAAAA
--Apple-Mail=_3E770697-DA71-4FD5-A893-4969390FA44F--