[OpenAFS] Setting up new 1.8.2 cell: possible protection server issues
Joseph Timothy Foley
foley@ru.is
Sun, 10 Feb 2019 09:51:25 +0000
--_000_51779493e7694916b80e45f3b5584022ruis_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hi all
I've been getting help on the IRC channel setting up a new cell for our CS =
department, but I've hit a roadblock that may need a 1.8.2 debugging expert=
. (Many thanks to auristor, billings, and patbarron)
I have setup 3 Centos7 hosts with IPA: ipa1.cs.ru.is, ipa2, ipa3.
IPA2 is the lowest numbered (for historical reasons) and is the Kerberos pr=
imary.
The other two are replication sites.
I have setup the Openafs clients using the yum packages
I've tried to follow the quickstart and https://wiki.openafs.org/admin/Inst=
allingOpenAFSonRHEL/
To the best of my ability, but I think something is wrong with the Protecti=
on server.
I've checked with rxdebug and there is connectivity between the 3 machines
I've added both "admin" and "foley" to system:adminstrators and using "bos =
adduser" to all the machines. "bos listuser" verifies this.
Symptom:
"pts membership admin" as admin works intermittently
[foley@ipa2 .cs.ru.is]$ pts membership admin
Groups admin (id: 1) is a member of:
system:administrators
[foley@ipa2 .cs.ru.is]$ pts membership admin
pts: Permission denied ; unable to get membership of admin (id: 1)
But with "-localauth" it always works.
[foley@ipa2 .cs.ru.is]$ klist -e
Ticket cache: KEYRING:persistent:1298400006:krb_ccache_qrL87VL
Default principal: admin@CS.RU.IS
Valid starting Expires Service principal
02/10/2019 09:42:12 02/11/2019 09:42:06 afs/cs.ru.is@CS.RU.IS
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
02/10/2019 09:42:10 02/11/2019 09:42:06 krbtgt/CS.RU.IS@CS.RU.IS
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
[foley@ipa2 .cs.ru.is]$ bos listusers ipa2
SUsers are: admin foley
[foley@ipa2 .cs.ru.is]$ pts examine admin
pts: Permission denied ; unable to find entry for (id: 1)
[foley@ipa2 .cs.ru.is]$ pts examine admin
Name: admin, id: 1, owner: system:administrators, creator: system:administr=
ators,
membership: 1, flags: S----, group quota: unlimited.
Possibly relevant logs:
On ipa2: (the lowest ip address) after a restart in /var/openafs/logs/PTLo=
g
Sun Feb 10 09:33:18 2019 Using 130.208.243.201 as my primary address
Sun Feb 10 09:33:18 2019 Starting AFS ptserver 1.1 (/usr/libexec/openafs/pt=
server)
Sun Feb 10 09:33:21 2019 ubik: A Remote Server has addresses:
Sun Feb 10 09:33:21 2019 ... 130.208.243.202
Sun Feb 10 09:33:24 2019 ubik: A Remote Server has addresses:
Sun Feb 10 09:33:24 2019 ... 130.208.243.205
But no mention of an election. I only see an election in the BackupLog.
I've tried setting a new key, just in case I got confused.
[root@ipa2 logs]# asetkey list
rxkad_krb5 kvno 1 enctype 17; key is: 3c54d85bad8dd99f938307e1a4bff=
2d5
rxkad_krb5 kvno 1 enctype 18; key is: a55c654701f21cd871278f09727ee=
9c6e7809f05f8eeebdfea9777e94f610ce1
rxkad_krb5 kvno 2 enctype 17; key is: 81f4e3ce6b8179833ad21a8539489=
a68
rxkad_krb5 kvno 2 enctype 18; key is: b90bbfbb11aa16a2cb0079b66467f=
a517bdaa4af101ab6ffab400cc6471c827e
All done.
(I've checked these on all 3 to make sure they were the same)
Trying to delete the old key gives an error
[root@ipa2 logs]# asetkey delete 1
asetkey: Unknown code acfg 1 (70354689) while deleting key 1
Symptom 2:
I can't release a read-only volume with those tickets
[foley@ipa2 .cs.ru.is]$ vos addsite ipa2 a root.afs
Could not lock the VLDB entry for the volume 536870915
VLDB: no permission access for call
Error in vos addsite command.
VLDB: no permission access for call
But -localauth works fine
[root@ipa2 logs]# vos addsite ipa2 a root.afs -localauth
Added replication site ipa2 /vicepa for volume root.afs
Symptom 3:
Even with all these issues, admin and foley can both create folders in the =
RW volume of the cell!
System and Package information (all 3 hosts should be identical):
[foley@ipa2 user]$ uname -a
Linux ipa2.cs.ru.is 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UT=
C 2018 x86_64 x86_64 x86_64 GNU/Linux
Name : openafs-client
Arch : x86_64
Version : 1.8.2
Release : 1.el7
Size : 1.1 M
Repo : installed
>From repo : storage-sig
Name : openafs-server
Arch : x86_64
Version : 1.8.2
Release : 1.el7
Size : 9.1 M
Repo : installed
>From repo : storage-sig
Any help would be appreciated.
Kind regards,
Joe
--
Dr. Joseph T. Foley <foley@ru.is> Assistant Professor, Dept. of Science & E=
ngineering, Reykjavik University
Menntavegur 1, Nauth=F3lsv=EDk | 101 Reykjav=EDk | Iceland | Phone: +354-59=
9-6569 | Fax +354-599-6201 | www.ru.is
--_000_51779493e7694916b80e45f3b5584022ruis_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"#0563C1" vlink=3D"#954F72">
<div class=3D"WordSection1">
<p class=3D"MsoNormal">Hi all<o:p></o:p></p>
<p class=3D"MsoNormal">I’ve been getting help on the IRC channel sett=
ing up a new cell for our CS department, but I’ve hit a roadblock tha=
t may need a 1.8.2 debugging expert. (Many thanks to auristor, billings, an=
d patbarron)<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">I have setup 3 Centos7 hosts with IPA: ipa1.cs=
.ru.is, ipa2, ipa3.<o:p></o:p></p>
<p class=3D"MsoNormal">IPA2 is the lowest numbered (for historical reasons)=
and is the Kerberos primary.<o:p></o:p></p>
<p class=3D"MsoNormal">The other two are replication sites.<o:p></o:p></p>
<p class=3D"MsoNormal">I have setup the Openafs clients using the yum packa=
ges<o:p></o:p></p>
<p class=3D"MsoNormal">I’ve tried to follow the quickstart and <a hre=
f=3D"https://wiki.openafs.org/admin/InstallingOpenAFSonRHEL/">
https://wiki.openafs.org/admin/InstallingOpenAFSonRHEL/</a><o:p></o:p></p>
<p class=3D"MsoNormal">To the best of my ability, but I think something is =
wrong with the Protection server.<o:p></o:p></p>
<p class=3D"MsoNormal">I’ve checked with rxdebug and there is connect=
ivity between the 3 machines<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">I’ve added both “admin” and “=
;foley” to system:adminstrators and using “bos adduser” t=
o all the machines. “bos listuser” verifies this.<o:p></o=
:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Symptom:<o:p></o:p></p>
<p class=3D"MsoNormal">“pts membership admin” as admin works in=
termittently<o:p></o:p></p>
<p class=3D"MsoNormal">[foley@ipa2 .cs.ru.is]$ pts membership admin<o:p></o=
:p></p>
<p class=3D"MsoNormal">Groups admin (id: 1) is a member of:<o:p></o:p></p>
<p class=3D"MsoNormal"> system:administrators<o:p></o:p></p>
<p class=3D"MsoNormal">[foley@ipa2 .cs.ru.is]$ pts membership admin<o:p></o=
:p></p>
<p class=3D"MsoNormal">pts: Permission denied ; unable to get membership of=
admin (id: 1)<o:p></o:p></p>
<p class=3D"MsoNormal">But with “-localauth” it always works.<o=
:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">[foley@ipa2 .cs.ru.is]$ klist -e<o:p></o:p></p>
<p class=3D"MsoNormal">Ticket cache: KEYRING:persistent:1298400006:krb_ccac=
he_qrL87VL<o:p></o:p></p>
<p class=3D"MsoNormal">Default principal: admin@CS.RU.IS<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Valid starting E=
xpires &nb=
sp; Service principal<o:p></o:p></p>
<p class=3D"MsoNormal">02/10/2019 09:42:12 02/11/2019 09:42:06 =
afs/cs.ru.is@CS.RU.IS<o:p></o:p></p>
<p class=3D"MsoNormal"> Etype (sk=
ey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96<o:p></o:p></p>
<p class=3D"MsoNormal">02/10/2019 09:42:10 02/11/2019 09:42:06 =
krbtgt/CS.RU.IS@CS.RU.IS<o:p></o:p></p>
<p class=3D"MsoNormal"> Etype (sk=
ey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">[foley@ipa2 .cs.ru.is]$ bos listusers ipa2<o:p></o:p=
></p>
<p class=3D"MsoNormal">SUsers are: admin foley<o:p></o:p></p>
<p class=3D"MsoNormal">[foley@ipa2 .cs.ru.is]$ pts examine admin<o:p></o:p>=
</p>
<p class=3D"MsoNormal">pts: Permission denied ; unable to find entry for (i=
d: 1)<o:p></o:p></p>
<p class=3D"MsoNormal">[foley@ipa2 .cs.ru.is]$ pts examine admin<o:p></o:p>=
</p>
<p class=3D"MsoNormal">Name: admin, id: 1, owner: system:administrators, cr=
eator: system:administrators,<o:p></o:p></p>
<p class=3D"MsoNormal"> membership: 1, flags: S----, group quota: unl=
imited.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Possibly relevant logs:<o:p></o:p></p>
<p class=3D"MsoNormal">On ipa2: (the lowest ip address) after a resta=
rt in /var/openafs/logs/PTLog<o:p></o:p></p>
<p class=3D"MsoNormal">Sun Feb 10 09:33:18 2019 Using 130.208.243.201 as my=
primary address<o:p></o:p></p>
<p class=3D"MsoNormal">Sun Feb 10 09:33:18 2019 Starting AFS ptserver 1.1 (=
/usr/libexec/openafs/ptserver)<o:p></o:p></p>
<p class=3D"MsoNormal">Sun Feb 10 09:33:21 2019 ubik: A Remote Server has a=
ddresses:<o:p></o:p></p>
<p class=3D"MsoNormal">Sun Feb 10 09:33:21 2019 ... 130.208.243.202<o:p></o=
:p></p>
<p class=3D"MsoNormal">Sun Feb 10 09:33:24 2019 ubik: A Remote Server has a=
ddresses:<o:p></o:p></p>
<p class=3D"MsoNormal">Sun Feb 10 09:33:24 2019 ... 130.208.243.205<o:p></o=
:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">But no mention of an election. I only see an e=
lection in the BackupLog.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">I’ve tried setting a new key, just in case I g=
ot confused.<o:p></o:p></p>
<p class=3D"MsoNormal">[root@ipa2 logs]# asetkey list<o:p></o:p></p>
<p class=3D"MsoNormal">rxkad_krb5 kvno &=
nbsp; 1 enctype 17; key is: 3c54d85bad8dd99f938307e1a4bff2d5<o:p></o:=
p></p>
<p class=3D"MsoNormal">rxkad_krb5 kvno &=
nbsp; 1 enctype 18; key is: a55c654701f21cd871278f09727ee9c6e7809f05f=
8eeebdfea9777e94f610ce1<o:p></o:p></p>
<p class=3D"MsoNormal">rxkad_krb5 kvno &=
nbsp; 2 enctype 17; key is: 81f4e3ce6b8179833ad21a8539489a68<o:p></o:=
p></p>
<p class=3D"MsoNormal">rxkad_krb5 kvno &=
nbsp; 2 enctype 18; key is: b90bbfbb11aa16a2cb0079b66467fa517bdaa4af1=
01ab6ffab400cc6471c827e<o:p></o:p></p>
<p class=3D"MsoNormal">All done.<o:p></o:p></p>
<p class=3D"MsoNormal">(I’ve checked these on all 3 to make sure they=
were the same)<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Trying to delete the old key gives an error<o:p></o:=
p></p>
<p class=3D"MsoNormal">[root@ipa2 logs]# asetkey delete 1<o:p></o:p></p>
<p class=3D"MsoNormal">asetkey: Unknown code acfg 1 (70354689) while deleti=
ng key 1<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Symptom 2:<o:p></o:p></p>
<p class=3D"MsoNormal">I can’t release a read-only volume with those =
tickets<o:p></o:p></p>
<p class=3D"MsoNormal">[foley@ipa2 .cs.ru.is]$ vos addsite ipa2 a root.afs<=
o:p></o:p></p>
<p class=3D"MsoNormal">Could not lock the VLDB entry for the volume 5368709=
15<o:p></o:p></p>
<p class=3D"MsoNormal">VLDB: no permission access for call<o:p></o:p></p>
<p class=3D"MsoNormal">Error in vos addsite command.<o:p></o:p></p>
<p class=3D"MsoNormal">VLDB: no permission access for call<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">But –localauth works fine<o:p></o:p></p>
<p class=3D"MsoNormal">[root@ipa2 logs]# vos addsite ipa2 a root.afs -local=
auth<o:p></o:p></p>
<p class=3D"MsoNormal">Added replication site ipa2 /vicepa for volume root.=
afs<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Symptom 3:<o:p></o:p></p>
<p class=3D"MsoNormal">Even with all these issues, admin and foley can both=
create folders in the RW volume of the cell!
<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">System and Package information (all 3 hosts should b=
e identical):<o:p></o:p></p>
<p class=3D"MsoNormal">[foley@ipa2 user]$ uname -a<o:p></o:p></p>
<p class=3D"MsoNormal">Linux ipa2.cs.ru.is 3.10.0-957.1.3.el7.x86_64 #1 SMP=
Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Name : ope=
nafs-client<o:p></o:p></p>
<p class=3D"MsoNormal">Arch : x86=
_64<o:p></o:p></p>
<p class=3D"MsoNormal">Version : 1.8.2<o:p></o:p></=
p>
<p class=3D"MsoNormal">Release : 1.el7<o:p></o:p></=
p>
<p class=3D"MsoNormal">Size : 1.1=
M<o:p></o:p></p>
<p class=3D"MsoNormal">Repo : ins=
talled<o:p></o:p></p>
<p class=3D"MsoNormal">From repo : storage-sig<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Name : ope=
nafs-server<o:p></o:p></p>
<p class=3D"MsoNormal">Arch : x86=
_64<o:p></o:p></p>
<p class=3D"MsoNormal">Version : 1.8.2<o:p></o:p></=
p>
<p class=3D"MsoNormal">Release : 1.el7<o:p></o:p></=
p>
<p class=3D"MsoNormal">Size : 9.1=
M<o:p></o:p></p>
<p class=3D"MsoNormal">Repo : ins=
talled<o:p></o:p></p>
<p class=3D"MsoNormal">From repo : storage-sig<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">Any help would be appreciated.<o:p></o:p></p>
<p class=3D"MsoNormal">Kind regards,<o:p></o:p></p>
<p class=3D"MsoNormal">Joe<o:p></o:p></p>
<p class=3D"MsoNormal">--<o:p></o:p></p>
<p class=3D"MsoNormal">Dr. Joseph T. Foley <foley@ru.is> Assistant Pr=
ofessor, Dept. of Science & Engineering, Reykjavik University<o:p></o:p=
></p>
<p class=3D"MsoNormal">Menntavegur 1, Nauth=F3lsv=EDk | 101 Reykjav=EDk | I=
celand | Phone: +354-599-6569 | Fax +354-599-6201 | www.ru.is<o:p><=
/o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>
--_000_51779493e7694916b80e45f3b5584022ruis_--