[OpenAFS] DB quorum problems on 1.4/1.6 mixed cell
Arne Wiebalck
Arne.Wiebalck@cern.ch
Tue, 11 Feb 2014 09:50:35 +0000
--Apple-Mail=_6DAC8063-230F-4E14-97BE-DEFCFC62DD7C
Content-Type: multipart/alternative;
boundary="Apple-Mail=_102357F7-6206-4791-97AE-97ADB897EBCD"
--Apple-Mail=_102357F7-6206-4791-97AE-97ADB897EBCD
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
Hi,
We've recently added some 1.6.6 servers into our cell which is mainly on =
1.4.15 (i.e. most of the file servers
and the DB servers). We now encounter quorum problems with our VLDB =
servers.
The primary symptom is that releases fail with "u: no quorum elected".
VLLog on the sync site shows at that moment:
-->
Tue Feb 11 09:51:47 2014 ubik:server 137.138.246.51 is back up: will be =
contacted through 137.138.246.51
Tue Feb 11 09:51:47 2014 ubik:server 137.138.246.50 is back up: will be =
contacted through 137.138.246.50
<--
where 137.138.246.50 and 137.138.246.51 are the non-sync sites.
We can relatively easy trigger this problem by moving volumes between =
1.4 and 1.6 based servers (1.4/1.4
and 1.6/1.6 transfers seems OK): the move gets stuck and udebug shows=20
-->
Host's addresses are: 137.138.128.148
Host's 137.138.128.148 time is Tue Feb 11 09:38:02 2014
Local time is Tue Feb 11 09:38:02 2014 (time differential 0 secs)
Last yes vote for 137.138.128.148 was 5 secs ago (sync site);
Last vote started 5 secs ago (at Tue Feb 11 09:37:57 2014)
Local db version is 1392106724.154
I am sync site until 54 secs from now (at Tue Feb 11 09:38:56 2014) (3 =
servers)
Recovery state f
I am currently managing write trans 1392106724.-1892301558
Sync site's db version is 1392106724.154
0 locked pages, 0 of them for write
There are write locks held
Last time a new db version was labelled was:
1158 secs ago (at Tue Feb 11 09:18:44 2014)
Server (137.138.246.51): (db 1392106724.153)
last vote rcvd 20 secs ago (at Tue Feb 11 09:37:42 2014),
last beacon sent 20 secs ago (at Tue Feb 11 09:37:42 2014), last =
vote was yes
dbcurrent=3D0, up=3D0 beaconSince=3D0
Server (137.138.246.50): (db 1392106724.154)
last vote rcvd 6 secs ago (at Tue Feb 11 09:37:56 2014),
last beacon sent 5 secs ago (at Tue Feb 11 09:37:57 2014), last vote =
was yes
dbcurrent=3D1, up=3D1 beaconSince=3D1
<--
Note that the sync site has gone to Recovery state f and that the time =
at which the last vote was received
on the other two servers has quite a time gap which gets larger with =
time. Is the negative trans ID ok?
At some point the sync site loses its sync site state:
-->
Host's addresses are: 137.138.128.148
Host's 137.138.128.148 time is Tue Feb 11 09:41:01 2014
Local time is Tue Feb 11 09:41:02 2014 (time differential 1 secs)
Last yes vote for 137.138.128.148 was 4 secs ago (sync site);
Last vote started 4 secs ago (at Tue Feb 11 09:40:58 2014)
Local db version is 1392106724.206
I am not sync site
Lowest host 137.138.128.148 was set 4 secs ago
Sync host 137.138.128.148 was set 4 secs ago
I am currently managing write trans 1392106724.-1892283756
Sync site's db version is 1392106724.206
0 locked pages, 0 of them for write
There are write locks held
Last time a new db version was labelled was:
1337 secs ago (at Tue Feb 11 09:18:45 2014)
<--
so there is no sync site any longer and the vos command gets a no quorum =
error.
As this also happens when we do not move volumes around (like at 3am), =
but other operations such
as the backup touch the volumes, I would suspect that VLDB operations in =
general can trigger this.
Is this a known issue?=20
I had understood that it should be OK to run 1.4 and 1.6 file servers in =
parallel and that the DB servers
could be updated after the file servers, but maybe that is not correct?
Thanks!
Arne
--
Arne Wiebalck
CERN IT
--Apple-Mail=_102357F7-6206-4791-97AE-97ADB897EBCD
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=us-ascii
<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
">Hi,<div><br></div><div>We've recently added some 1.6.6 servers into =
our cell which is mainly on 1.4.15 (i.e. most of the file =
servers</div><div>and the DB servers). We now encounter quorum problems =
with our VLDB servers.</div><div><br></div><div>The primary symptom is =
that releases fail with "u: no quorum =
elected".</div><div><br></div><div>VLLog on the sync site shows at that =
moment:</div><div>--></div><div><div>Tue Feb 11 09:51:47 2014 =
ubik:server 137.138.246.51 is back up: will be contacted through =
137.138.246.51</div><div>Tue Feb 11 09:51:47 2014 ubik:server =
137.138.246.50 is back up: will be contacted through =
137.138.246.50</div></div><div><--</div><div>where 137.138.2=
46.50 and 137.138.246.51 are the non-sync =
sites.</div><div><br></div><div>We can relatively easy trigger this =
problem by moving volumes between 1.4 and 1.6 based servers =
(1.4/1.4</div><div>and 1.6/1.6 transfers seems OK): the move gets stuck =
and udebug =
shows </div><div><br></div><div>--></div><div><div>Host's =
addresses are: 137.138.128.148</div><div>Host's 137.138.128.148 time is =
Tue Feb 11 09:38:02 2014</div><div>Local time is Tue Feb 11 09:38:02 =
2014 (time differential 0 secs)</div><div>Last yes vote for =
137.138.128.148 was 5 secs ago (sync site);</div><div>Last vote started =
5 secs ago (at Tue Feb 11 09:37:57 2014)</div><div>Local db version is =
1392106724.154</div><div>I am sync site until 54 secs from now (at Tue =
Feb 11 09:38:56 2014) (3 servers)</div><div>Recovery state f</div><div>I =
am currently managing write trans 1392106724.-1892301558</div><div>Sync =
site's db version is 1392106724.154</div><div>0 locked pages, 0 of them =
for write</div><div>There are write locks held</div><div>Last time a new =
db version was labelled was:</div><div> =
1158 secs ago (at Tue Feb 11 09:18:44 =
2014)</div><div><br></div><div>Server (137.138.246.51): (db =
1392106724.153)</div><div> last vote rcvd 20 secs ago (at =
Tue Feb 11 09:37:42 2014),</div><div> last beacon sent 20 =
secs ago (at Tue Feb 11 09:37:42 2014), last vote was =
yes</div><div> dbcurrent=3D0, up=3D0 =
beaconSince=3D0</div><div><br></div><div>Server (137.138.246.50): (db =
1392106724.154)</div><div> last vote rcvd 6 secs ago (at =
Tue Feb 11 09:37:56 2014),</div><div> last beacon sent 5 =
secs ago (at Tue Feb 11 09:37:57 2014), last vote was =
yes</div><div> dbcurrent=3D1, up=3D1 =
beaconSince=3D1</div></div><div><--</div><div><br></div><div>Note =
that the sync site has gone to Recovery state f and that the time at =
which the last vote was received</div><div>on the other two servers has =
quite a time gap which gets larger with time. Is the negative trans ID =
ok?</div><div><br></div><div>At some point the sync site loses its sync =
site state:</div><div><br></div><div>--></div><div><div>Host's =
addresses are: 137.138.128.148</div><div>Host's 137.138.128.148 time is =
Tue Feb 11 09:41:01 2014</div><div>Local time is Tue Feb 11 09:41:02 =
2014 (time differential 1 secs)</div><div>Last yes vote for =
137.138.128.148 was 4 secs ago (sync site);</div><div>Last vote started =
4 secs ago (at Tue Feb 11 09:40:58 2014)</div><div>Local db version is =
1392106724.206</div><div>I am not sync site</div><div>Lowest host =
137.138.128.148 was set 4 secs ago</div><div>Sync host 137.138.128.148 =
was set 4 secs ago</div><div>I am currently managing write trans =
1392106724.-1892283756</div><div>Sync site's db version is =
1392106724.206</div><div>0 locked pages, 0 of them for =
write</div><div>There are write locks held</div><div>Last time a new db =
version was labelled was:</div><div> =
1337 secs ago (at Tue Feb 11 09:18:45 =
2014)</div></div><div><--</div><div><br></div><div>so there is no =
sync site any longer and the vos command gets a no quorum =
error.</div><div><br></div><div>As this also happens when we do not =
move volumes around (like at 3am), but other operations =
such</div><div>as the backup touch the volumes, I would suspect =
that VLDB operations in general can trigger =
this.</div><div><br></div><div>Is this a known =
issue? </div><div><br></div><div>I had understood that it should be =
OK to run 1.4 and 1.6 file servers in parallel and that the DB =
servers</div><div>could be updated after the file servers, but maybe =
that is not =
correct?</div><div><br></div><div>Thanks!</div><div> Arne</div><div><=
br></div><div><br><div>
<div style=3D"color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
medium; font-style: normal; font-variant: normal; font-weight: normal; =
letter-spacing: normal; line-height: normal; orphans: 2; text-align: =
-webkit-auto; text-indent: 0px; text-transform: none; white-space: =
normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; =
-webkit-text-stroke-width: 0px; word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div =
style=3D"color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; =
font-style: normal; font-variant: normal; font-weight: normal; =
letter-spacing: normal; line-height: normal; orphans: 2; text-align: =
-webkit-auto; text-indent: 0px; text-transform: none; white-space: =
normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; =
-webkit-text-stroke-width: 0px; word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
">--</div><div style=3D"color: rgb(0, 0, 0); font-family: Helvetica; =
font-size: medium; font-style: normal; font-variant: normal; =
font-weight: normal; letter-spacing: normal; line-height: normal; =
orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: =
none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; =
word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: =
after-white-space; ">Arne Wiebalck<br>CERN IT</div></div>
</div>
<br></div></body></html>=
--Apple-Mail=_102357F7-6206-4791-97AE-97ADB897EBCD--
--Apple-Mail=_6DAC8063-230F-4E14-97BE-DEFCFC62DD7C
Content-Disposition: attachment; filename="smime.p7s"
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIINuDCCBi4w
ggUWoAMCAQICCmEPqkwAAAAAAAMwDQYJKoZIhvcNAQEFBQAwQTESMBAGCgmSJomT8ixkARkWAmNo
MRQwEgYKCZImiZPyLGQBGRYEY2VybjEVMBMGA1UEAxMMQ0VSTiBSb290IENBMB4XDTA2MTAwMzA5
MzYxM1oXDTE2MTAwMzA5NDYxM1owWTESMBAGCgmSJomT8ixkARkWAmNoMRQwEgYKCZImiZPyLGQB
GRYEY2VybjEtMCsGA1UEAxMkQ0VSTiBUcnVzdGVkIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwdFGqthhWlgUOSZ6C6hReNEVGzbjf2IQgxo7
/rOfOQHZH3krQPQ37fqFroEr46PrruymZ/U+QAzmESZQ4Z+nCfBhm7cEi0TIhihHd4cEPaPxawGR
T9Ck7BBk9za8TUljF6c/JodnIcmIrpbazEbSAS1KEXwETHDsTrQ7lJ+6SzDP4/oOwrHrgJx+tKsm
gOsFSbBEK4OYx1UYQpYS9OJQk2Sc0q4a/SCSu+xbN8ppmgV3WFytN8NW20n3NpCCWYPzo9rXmPRA
7a/c6mf+RV5gPCnUqeW6KUvix5kz9+X8/4SQV/fU12OPdRvtkqcC+PpiePK7bjMLQJEYwvchJrSz
AwIDAQABo4IDDjCCAwowDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUmMyS0EYwNoyw7ZgNclGp
R0zdviEwCwYDVR0PBAQDAgGGMBAGCSsGAQQBgjcVAQQDAgECMCMGCSsGAQQBgjcVAgQWBBT/Rljl
vgfrVK8GmAaYe+TbiXbJ7DBRBgNVHSAESjBIMEYGCisGAQQBYAoCAQEwODA2BggrBgEFBQcCARYq
aHR0cDovL2NhLmNlcm4uY2gvY2EvY3JsL3BvbGljeS9jcC1jcHMucGRmMBkGCSsGAQQBgjcUAgQM
HgoAUwB1AGIAQwBBMB8GA1UdIwQYMBaAFJgK9+w+7FnWHa2ZvLUBPt7spudQMIH8BgNVHR8EgfQw
gfEwge6ggeuggeiGLWh0dHA6Ly9jYS5jZXJuLmNoL2NhL2NybC9DRVJOJTIwUm9vdCUyMENBLmNy
bIaBtmxkYXA6Ly8vQ049Q0VSTiUyMFJvb3QlMjBDQSxDTj1jZXJucm9vdGNhLENOPUNEUCxDTj1Q
dWJsaWMlMjBLZXklMjBTZXJ2aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWNl
cm4sREM9Y2g/Y2VydGlmaWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNlP29iamVjdENsYXNzPWNSTERp
c3RyaWJ1dGlvblBvaW50MIIBBAYIKwYBBQUHAQEEgfcwgfQwRAYIKwYBBQUHMAKGOGh0dHA6Ly9j
YS5jZXJuLmNoL2NhL2NybC9jZXJucm9vdGNhX0NFUk4lMjBSb290JTIwQ0EuY3J0MIGrBggrBgEF
BQcwAoaBnmxkYXA6Ly8vQ049Q0VSTiUyMFJvb3QlMjBDQSxDTj1BSUEsQ049UHVibGljJTIwS2V5
JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1jZXJuLERDPWNoP2NB
Q2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1jZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MA0GCSqG
SIb3DQEBBQUAA4IBAQAfEzvOeYohKndmJqnVdiCqZ38tSBxOOPsKUHW4UY1jBfYMXbnZ9keFQFlK
/g5X4aZPNBEHXw0eKpQVsMhEPWQrvx8T/f7GwtU+JNQhkgK9tnezmHxYzWgEC9MXZhfYzFSwMIF6
kSKllmUTnN35uF1EnT8+64daje+yEVcpmM34p8Fw125/WpKnRmwNp0YkUk6uMti6Y6vOTHttzIN5
P6elGoat8sadMqrVnaMNzG8hGUvSkYivYBs7msAPuwmXgLvIkXWPW+MDFs+x5Kzx75ZHv3c2WoKg
UxL5KZH9QqiR7t8P6YBfYW6SpzyGRi4QHN/iOLhXZ06R6aPljLEOn41JMIIHgjCCBmqgAwIBAgIK
HYLrgQACAAHOszANBgkqhkiG9w0BAQUFADBZMRIwEAYKCZImiZPyLGQBGRYCY2gxFDASBgoJkiaJ
k/IsZAEZFgRjZXJuMS0wKwYDVQQDEyRDRVJOIFRydXN0ZWQgQ2VydGlmaWNhdGlvbiBBdXRob3Jp
dHkwHhcNMTMwMzI2MDkxNjM1WhcNMTQwMzI2MDkxNjM1WjCBjjESMBAGCgmSJomT8ixkARkWAmNo
MRQwEgYKCZImiZPyLGQBGRYEY2VybjEWMBQGA1UECxMNT3JnYW5pYyBVbml0czEOMAwGA1UECxMF
VXNlcnMxETAPBgNVBAMTCHdpZWJhbGNrMQ8wDQYDVQQDEwY1NTg1MTAxFjAUBgNVBAMTDUFybmUg
V2llYmFsY2swggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDKWiuJsxwe2ZAkHn+FKzTT
VBNe7lhl2PmTjPmZsedrVI0gZkUpjlb6x21avo3S+kacxPpVdNVmidE1k7Dmn+MLLwli8Ha+ePqN
e2kD33K01bjf8G1l5KquZwMsJKB2gSwSs1JK5uq3qcB5b1mrzPVOqOqNPUMg1NUkEjYql4ooG5Cj
GfVwlAhRarBssM07JtnToszp/fiCqttCR4pAAjURvQKV3T+Epiexwgzkv57mLAXe3vNf8giDigJP
zvNc+3pGqA8faVOGSqOo+9m6NheYPH4pcYcnhgA177HRfHxJ9fqzEahLNzXjoa9vtW8cpbSxyzY1
wbOCvQW1hz/uCP/5AgMBAAGjggQUMIIEEDAdBgNVHQ4EFgQUFIvKPnfKzu1UVQjsxgpwQPtpkwAw
HwYDVR0jBBgwFoAUmMyS0EYwNoyw7ZgNclGpR0zdviEwggEyBgNVHR8EggEpMIIBJTCCASGgggEd
oIIBGYZHaHR0cDovL2NhLmNlcm4uY2gvY2EvY3JsL0NFUk4lMjBUcnVzdGVkJTIwQ2VydGlmaWNh
dGlvbiUyMEF1dGhvcml0eS5jcmyGgc1sZGFwOi8vL0NOPUNFUk4lMjBUcnVzdGVkJTIwQ2VydGlm
aWNhdGlvbiUyMEF1dGhvcml0eSxDTj1DRVJOUEtJLENOPUNEUCxDTj1QdWJsaWMlMjBLZXklMjBT
ZXJ2aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWNlcm4sREM9Y2g/Y2VydGlm
aWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNlP29iamVjdENsYXNzPWNSTERpc3RyaWJ1dGlvblBvaW50
MIIBLwYIKwYBBQUHAQEEggEhMIIBHTCBxQYIKwYBBQUHMAKGgbhsZGFwOi8vL0NOPUNFUk4lMjBU
cnVzdGVkJTIwQ2VydGlmaWNhdGlvbiUyMEF1dGhvcml0eSxDTj1BSUEsQ049UHVibGljJTIwS2V5
JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1jZXJuLERDPWNoP2NB
Q2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1jZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MFMGCCsG
AQUFBzAChkdodHRwOi8vY2EuY2Vybi5jaC9jYS9jcmwvQ0VSTiUyMFRydXN0ZWQlMjBDZXJ0aWZp
Y2F0aW9uJTIwQXV0aG9yaXR5LmNydDAOBgNVHQ8BAf8EBAMCBaAwPQYJKwYBBAGCNxUHBDAwLgYm
KwYBBAGCNxUIg73QCYLtjQ2G7Ysrgd71N4WA0GIehb+6A4TEzEwCAWQCAQowKQYDVR0lBCIwIAYI
KwYBBQUHAwIGCCsGAQUFBwMEBgorBgEEAYI3CgMEMCUGA1UdIAQeMBwwDAYKKwYBBAFgCgIBAjAM
BgoqhkiG90wFAgIBMDUGCSsGAQQBgjcVCgQoMCYwCgYIKwYBBQUHAwIwCgYIKwYBBQUHAwQwDAYK
KwYBBAGCNwoDBDBHBgNVHREEQDA+oCUGCisGAQQBgjcUAgOgFwwVYXJuZS53aWViYWxja0BjZXJu
LmNogRVBcm5lLldpZWJhbGNrQGNlcm4uY2gwRAYJKoZIhvcNAQkPBDcwNTAOBggqhkiG9w0DAgIC
AIAwDgYIKoZIhvcNAwQCAgCAMAcGBSsOAwIHMAoGCCqGSIb3DQMHMA0GCSqGSIb3DQEBBQUAA4IB
AQBPiPoalqg44BSHOfc5XKMLl8dbU2ixH84cWFWmv8WrkM1oTVOopGUi9dO2qiN3PbzgkW3h4yB6
hEvTn7CVCfuzPXjEb9dQEJEmVheW8xA96GYWYWdXqdNHWtPMJB4tFh0skOMo35DwavZlrE30LLR+
BK8faePfnQ6PgPrdBESQaU4ZemLS5Q/eKfz2d4kSt5+70rj8ypD7pmMdUu/n1TFxuG3UNCFSMs6G
SUV0/PBjyV1O8jGtVbE9vNZE+NTuzWiYgI5Z6Pl6cHefI6miSn++PDjHzfaIPi0san0KMOXLqvA8
ZfjtAb2QMEvvKziTdg9Q4Y7wHCdi7aqKWmTbwyqUMYIC4TCCAt0CAQEwZzBZMRIwEAYKCZImiZPy
LGQBGRYCY2gxFDASBgoJkiaJk/IsZAEZFgRjZXJuMS0wKwYDVQQDEyRDRVJOIFRydXN0ZWQgQ2Vy
dGlmaWNhdGlvbiBBdXRob3JpdHkCCh2C64EAAgABzrMwCQYFKw4DAhoFAKCCAU8wGAYJKoZIhvcN
AQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTQwMjExMDk1MTAyWjAjBgkqhkiG9w0B
CQQxFgQUNx64EcXc8quVaZcmTTRDqlJSLaIwdgYJKwYBBAGCNxAEMWkwZzBZMRIwEAYKCZImiZPy
LGQBGRYCY2gxFDASBgoJkiaJk/IsZAEZFgRjZXJuMS0wKwYDVQQDEyRDRVJOIFRydXN0ZWQgQ2Vy
dGlmaWNhdGlvbiBBdXRob3JpdHkCCh2C64EAAgABzrMweAYLKoZIhvcNAQkQAgsxaaBnMFkxEjAQ
BgoJkiaJk/IsZAEZFgJjaDEUMBIGCgmSJomT8ixkARkWBGNlcm4xLTArBgNVBAMTJENFUk4gVHJ1
c3RlZCBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eQIKHYLrgQACAAHOszANBgkqhkiG9w0BAQEFAASC
AQCA4ysiK+LoJQrUzCQaXa0lctuZugmP074HrhC/uDYazKMBXwycR4t76EgNsQbGI8YvyLFrMUMR
VgN4XCI4MGCJME6ZxUUixzzMivkoUfXEMJEN3YpQVXJ09g+xbCi2kXsCD7qXHd4FTLAnzWJcpDQJ
7LDUGOFF4Bv8rGhwazBWzWfuvGUv7WsD/+eqN/P01QCDrgxTJ2FrMupZHuDwfCaQi7nw517KZqik
uiF8swQ2/vrIWwsY5/yrSzgCPspIsDFgGKypgecB07Db4KM8AQIGQTKqTibky/9s7slEOHi/n2LU
NMIh174sWF3w42SqQcJDoiDicw9MoOtlW9tfZ2CpAAAAAAAA
--Apple-Mail=_6DAC8063-230F-4E14-97BE-DEFCFC62DD7C--