[OpenAFS] Setting up new 1.8.2 cell: possible protection server
issues
Joseph Timothy Foley
foley@ru.is
Sun, 10 Feb 2019 13:56:09 +0000
First of all, thank you Jan for your insight. I had forgotten about the "u=
debug" command. I was puzzled because the backup logs mention when an elec=
tion takes place, but the others don't!
I have checked and they are finishing elections and agreeing on a sync site=
(ipa2) on both 7002 and 7003.
This appears to be a kerberos integration problem of some sort. How do I g=
o about figuring out why both foley and admin are not being identified corr=
ectly? =20
This is doubly puzzling because the Kerberos integration worked fine when =
I initially had only one server installed. The problem arose when I setup =
the two other DB servers. (It also deleted foley and admin from the protec=
tion database, and deleted the root.afs volume!)
I rekeyed as best I can, since asetkey gives this error:
[root@ipa2 foley]# asetkey delete 1
asetkey: Unknown code acfg 1 (70354689) while deleting key 1
I did not realize that posting the key IDs would be a serious security hole=
. I will not post them again.
There is nothing in the cell at the moment, so I'm happy to wipe and start =
over if need be.
Joe
--
Dr. Joseph T. Foley <foley@ru.is> Assistant Professor, Reykjavik Universit=
y +354-599-6569
-----Original Message-----
From: openafs-info-admin@openafs.org <openafs-info-admin@openafs.org> On Be=
half Of Jan Iven
Sent: Sunday, 10 February, 2019 11:18
To: openafs-info@openafs.org
Subject: Re: [OpenAFS] Setting up new 1.8.2 cell: possible protection serve=
r issues
(top of my head):
* AFS clients and servers (even on the same machine) use different location=
s for CellServDB. It looks like the client file has all 3 hosts (which woul=
d explain the intermittent results), but the servers might not know about e=
ach other, which would mean elections don't work.
* suggest to check ptserver status on the servers (should print version) fo=
r i in iap1 ipa2 ipa3; do rxdebug "${i}" 7002; done
* check election status for ptserver:
for i in iap1 ipa2 ipa3; do udebug "${i}" 7002; done
The current master will print all participants, and all servers should agre=
e who is "sync site". If tehre are message about unknown servers, see "serv=
er" CellServDB
You also might want to split your problems:
* ptserver elections - needs service key. Then just use "-localauth"
* same for vlserver (port 7003, same thing as above)
* kerberos integration (i.e identifying "admin" account from Kerberos ticke=
t -> AFS token)
* volume releasing - needs service key, vlserver election to work, admin a=
ccount to be identified.
PS: please re-key, and do not post the new "asetkey list" again..
Cheers
jan
On 10/02/2019 10:51, Joseph Timothy Foley wrote:
> Hi all
>=20
> I've been getting help on the IRC channel setting up a new cell for=20
> our CS department, but I've hit a roadblock that may need a 1.8.2=20
> debugging expert. (Many thanks to auristor, billings, and patbarron)
>=20
> I have setup 3 Centos7 hosts with IPA:=A0 ipa1.cs.ru.is, ipa2, ipa3.
>=20
> IPA2 is the lowest numbered (for historical reasons) and is the=20
> Kerberos primary.
>=20
> The other two are replication sites.
>=20
> I have setup the Openafs clients using the yum packages
>=20
> I've tried to follow the quickstart and=20
> https://wiki.openafs.org/admin/InstallingOpenAFSonRHEL/
>=20
> To the best of my ability, but I think something is wrong with the=20
> Protection server.
>=20
> I've checked with rxdebug and there is connectivity between the 3=20
> machines
>=20
> I've added both "admin" and "foley" to system:adminstrators and using=20
> "bos adduser" to all the machines.=A0 "bos listuser" verifies this.
>=20
> Symptom:
>=20
> "pts membership admin" as admin works intermittently
>=20
> [foley@ipa2 .cs.ru.is]$ pts membership admin
>=20
> Groups admin (id: 1) is a member of:
>=20
> =A0 system:administrators
>=20
> [foley@ipa2 .cs.ru.is]$ pts membership admin
>=20
> pts: Permission denied ; unable to get membership of admin (id: 1)
>=20
> But with "-localauth" it always works.
>=20
> [foley@ipa2 .cs.ru.is]$ klist -e
>=20
> Ticket cache: KEYRING:persistent:1298400006:krb_ccache_qrL87VL
>=20
> Default principal: admin@CS.RU.IS
>=20
> Valid starting=A0=A0=A0=A0=A0=A0 Expires=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=
=A0=A0 Service principal
>=20
> 02/10/2019 09:42:12=A0 02/11/2019 09:42:06=A0 afs/cs.ru.is@CS.RU.IS
>=20
> =A0=A0=A0=A0=A0=A0=A0 Etype (skey, tkt): aes256-cts-hmac-sha1-96,=20
> aes256-cts-hmac-sha1-96
>=20
> 02/10/2019 09:42:10=A0 02/11/2019 09:42:06=A0 krbtgt/CS.RU.IS@CS.RU.IS
>=20
> =A0=A0=A0=A0=A0=A0=A0 Etype (skey, tkt): aes256-cts-hmac-sha1-96,=20
> aes256-cts-hmac-sha1-96
>=20
> [foley@ipa2 .cs.ru.is]$ bos listusers ipa2
>=20
> SUsers are: admin foley
>=20
> [foley@ipa2 .cs.ru.is]$ pts examine admin
>=20
> pts: Permission denied ; unable to find entry for (id: 1)
>=20
> [foley@ipa2 .cs.ru.is]$ pts examine admin
>=20
> Name: admin, id: 1, owner: system:administrators, creator:=20
> system:administrators,
>=20
> =A0 membership: 1, flags: S----, group quota: unlimited.
>=20
> Possibly relevant logs:
>=20
> On ipa2:=A0 (the lowest ip address) after a restart in=20
> /var/openafs/logs/PTLog
>=20
> Sun Feb 10 09:33:18 2019 Using 130.208.243.201 as my primary address
>=20
> Sun Feb 10 09:33:18 2019 Starting AFS ptserver 1.1
> (/usr/libexec/openafs/ptserver)
>=20
> Sun Feb 10 09:33:21 2019 ubik: A Remote Server has addresses:
>=20
> Sun Feb 10 09:33:21 2019 ... 130.208.243.202
>=20
> Sun Feb 10 09:33:24 2019 ubik: A Remote Server has addresses:
>=20
> Sun Feb 10 09:33:24 2019 ... 130.208.243.205
>=20
> But no mention of an election.=A0 I only see an election in the BackupLog=
.
>=20
> I've tried setting a new key, just in case I got confused.
>=20
> [root@ipa2 logs]# =A0asetkey list
>=20
> rxkad_krb5=A0=A0=A0=A0=A0 kvno=A0=A0=A0 1 enctype 17; key is:=20
> 3c54d85bad8dd99f938307e1a4bff2d5
>=20
> rxkad_krb5=A0=A0=A0=A0=A0 kvno=A0=A0=A0 1 enctype 18; key is:=20
> a55c654701f21cd871278f09727ee9c6e7809f05f8eeebdfea9777e94f610ce1
>=20
> rxkad_krb5=A0=A0=A0=A0=A0 kvno=A0=A0=A0 2 enctype 17; key is:=20
> 81f4e3ce6b8179833ad21a8539489a68
>=20
> rxkad_krb5=A0=A0=A0=A0=A0 kvno=A0=A0=A0 2 enctype 18; key is:=20
> b90bbfbb11aa16a2cb0079b66467fa517bdaa4af101ab6ffab400cc6471c827e
>=20
> All done.
>=20
> (I've checked these on all 3 to make sure they were the same)
>=20
> Trying to delete the old key gives an error
>=20
> [root@ipa2 logs]# asetkey delete 1
>=20
> asetkey: Unknown code acfg 1 (70354689) while deleting key 1
>=20
> Symptom 2:
>=20
> I can't release a read-only volume with those tickets
>=20
> [foley@ipa2 .cs.ru.is]$ vos addsite ipa2 a root.afs
>=20
> Could not lock the VLDB entry for the volume 536870915
>=20
> VLDB: no permission access for call
>=20
> Error in vos addsite command.
>=20
> VLDB: no permission access for call
>=20
> But -localauth works fine
>=20
> [root@ipa2 logs]# vos addsite ipa2 a root.afs -localauth
>=20
> Added replication site ipa2 /vicepa for volume root.afs
>=20
> Symptom 3:
>=20
> Even with all these issues, admin and foley can both create folders in=20
> the RW volume of the cell!
>=20
> System and Package information (all 3 hosts should be identical):
>=20
> [foley@ipa2 user]$ uname -a
>=20
> Linux ipa2.cs.ru.is 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29=20
> 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>=20
> Name=A0=A0=A0=A0=A0=A0=A0 : openafs-client
>=20
> Arch=A0=A0=A0=A0=A0=A0=A0 : x86_64
>=20
> Version=A0=A0=A0=A0 : 1.8.2
>=20
> Release=A0=A0=A0=A0 : 1.el7
>=20
> Size=A0=A0=A0=A0=A0=A0=A0 : 1.1 M
>=20
> Repo=A0=A0=A0=A0=A0=A0=A0 : installed
>=20
> From repo=A0=A0 : storage-sig
>=20
> Name=A0=A0=A0=A0=A0=A0=A0 : openafs-server
>=20
> Arch=A0=A0=A0=A0=A0=A0=A0 : x86_64
>=20
> Version=A0=A0=A0=A0 : 1.8.2
>=20
> Release=A0=A0=A0=A0 : 1.el7
>=20
> Size=A0=A0=A0 =A0=A0=A0=A0: 9.1 M
>=20
> Repo=A0=A0=A0=A0=A0=A0=A0 : installed
>=20
> From repo=A0=A0 : storage-sig
>=20
> Any help would be appreciated.
>=20
> Kind regards,
>=20
> Joe
>=20
> --
>=20
> Dr. Joseph T. Foley <foley@ru.is> Assistant Professor, Dept. of=20
> Science & Engineering, Reykjavik University
>=20
> Menntavegur 1, Nauth=F3lsv=EDk | 101 Reykjav=EDk | Iceland | Phone:=20
> +354-599-6569 | Fax +354-599-6201 | www.ru.is
>=20
_______________________________________________
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info