[OpenAFS] Serious trouble, mounting /afs, ptserver, database
rebuilding
Jeffrey Altman
jaltman@secure-endpoints.com
Thu, 24 Jul 2008 02:40:39 -0400
This is a cryptographically signed message in MIME format.
--------------ms010900030109010507010704
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Obtaining an AFS token via Kerberos should be possible even if you do
not have a working AFS server environment. Use "aklog -d" to obtain
debugging information regarding the Kerberos service principal that
is being acquired. The error you received "no KDC available" most
likely indicates that you have no configuration information specified
on your machine. You should start off by ensuring that the krb5.conf
file for your Kerberos realm is installed on your machine.
However even if you can't get a Kerberos ticket and authenticate that
way to your AFS servers, if you have direct access to the AFS servers
themselves you can use the -localauth parameter to all of the relevant
commands provided that your local machine account has the ability to
read the AFS KeyFile.
The error "Can't rebuild the database because it is not empty"
is produced because the verification of the PRDB integrity failed:
if ((ntohl(cheader.version) == PRDBVERSION)
&& ntohl(cheader.headerSize) == sizeof(cheader)
&& ntohl(cheader.eofPtr) != (afs_uint32) NULL
&& FindByID(tt, ANONYMOUSID) != 0) {
/* database exists, so we don't have to build it */
code = ubik_EndTrans(tt);
if (code)
return code;
return PRSUCCESS;
}
/* else we need to build a database */
But the PRDB is not empty and we do not want to destroy the corrupted
database if that is the only copy of the database that exists.
In order for the ptserver to automatically rebuild the PRDB two
things must be true.
1. The PRDB must be empty
2. The ptserver must be running in -noauth mode
Otherwise, if you build OpenAFS from source, in the src/ptserver
directory is a command called 'db_verify'. In addition to being able to
use db_verify to validate the contents of the PRDB, one of the
parameters to 'db_verify' is -rebuild. 'db_verify' can only be executed
while the ptserver is shutdown and should only be used after backups
of the file have been made.
NOTE: db_verify is intended to be used be individuals comfortable with
reading the OpenAFS sources and whom have knowledge of OpenAFS internals.
Jeffrey Altman
kanou wrote:
> so, i thank you all for your help. i m sorry for not beeing so
> experienced with afs/kerberos. it s pretty new to me.
> by now, my first server, the one that was broken, is running pretty good
> and everybody gets their files, but still the database needs rebuilding
> and i dont know how.
>
> the second one, that worked well until the other began running again, is
> in a bad state.
> nobody can connect to their files and i m not able to get a ticket from
> kerberos.
>
> bos status -server myserver2
> bos: no such entry (getting tickets)
> bos: running unauthenticated
> Instance fs, currently running normally.
> Auxiliary status is: file server running.
> Instance ptserver, currently running normally.
> Instance vlserver, currently running normally.
>
> and logs:
> ==> /var/log/openafs/PtLog <==
> ptserver: Unknown code pt 11 (267275) Can't rebuild database because it
> is not empty
>
> please tell me how to rebuild the database.
> kanou
>
> Am 23.07.2008 um 20:06 schrieb kanou:
>
>> Well i did:
>> udebug myserver 7002
>> Host's addresses are: ip-myserver
>> Host's ip-myserver time is Wed Jul 23 19:47:26 2008
>> Local time is Wed Jul 23 19:47:29 2008 (time differential 3 secs)
>> Last yes vote for ip-myserver2 was 24 secs ago (sync site);
>> Last vote started 24 secs ago (at Wed Jul 23 19:47:05 2008)
>> Local db version is 1214806124.7
>> I am not sync site
>> Lowest host ip-myserver2 was set 24 secs ago
>> Sync host ip-myserver2 was set 24 secs ago
>> Sync site's db version is 1214806124.7
>> 0 locked pages, 0 of them for write
>>
>> udebug myserver 7003
>> Host's addresses are: ip-myserver
>> Host's ip-myserver time is Wed Jul 23 19:49:34 2008
>> Local time is Wed Jul 23 19:49:35 2008 (time differential 1 secs)
>> Last yes vote for ip-myserver was 44 secs ago (not sync site);
>> Last vote started 44 secs ago (at Wed Jul 23 19:48:51 2008)
>> Local db version is 1216833173.3
>> I am not sync site
>> Lowest host ip-myserver2 was set 5 secs ago
>> Sync host 0.0.0.0 was set 154 secs ago
>> Sync site's db version is 1216833173.3
>> 0 locked pages, 0 of them for write
>>
>> and on myserver2:
>> udebug myserver2 7002
>> Host's addresses are: ip-myserver2
>> Host's ip-myserver2 time is Wed Jul 23 19:56:38 2008
>> Local time is Wed Jul 23 19:56:39 2008 (time differential 1 secs)
>> Last yes vote for ip-myserver2 was 6 secs ago (sync site);
>> Last vote started 6 secs ago (at Wed Jul 23 19:56:33 2008)
>> Local db version is 1214806124.7
>> I am sync site until 51 secs from now (at Wed Jul 23 19:57:30 2008) (2
>> servers)
>> Recovery state 1f
>> Sync site's db version is 1214806124.7
>> 0 locked pages, 0 of them for write
>>
>> Server (ip-myserver): (db 1214806124.7)
>> last vote rcvd 9 secs ago (at Wed Jul 23 19:56:30 2008),
>> last beacon sent 6 secs ago (at Wed Jul 23 19:56:33 2008), last
>> vote was yes
>> dbcurrent=1, up=1 beaconSince=1
>>
>> udebug myserver2 7003
>> Host's addresses are: ip-myserver2
>> Host's ip-myserver2 time is Wed Jul 23 19:57:50 2008
>> Local time is Wed Jul 23 19:57:50 2008 (time differential 0 secs)
>> Last yes vote for ip-myserver2 was 3 secs ago (sync site);
>> Last vote started 3 secs ago (at Wed Jul 23 19:57:47 2008)
>> Local db version is 1216835658.20
>> I am sync site until 56 secs from now (at Wed Jul 23 19:58:46 2008) (2
>> servers)
>> Recovery state 1f
>> Sync site's db version is 1216835658.20
>> 0 locked pages, 0 of them for write
>> Last time a new db version was labelled was:
>> 212 secs ago (at Wed Jul 23 19:54:18 2008)
>>
>> Server (ip-myserver): (db 1216835658.20)
>> last vote rcvd 4 secs ago (at Wed Jul 23 19:57:46 2008),
>> last beacon sent 3 secs ago (at Wed Jul 23 19:57:47 2008), last
>> vote was yes
>> dbcurrent=1, up=1 beaconSince=1
>>
>> The system on myserver (the first one) is now running again but if i
>> try to create a user i still get:
>> pts: database needs rebuilding ; unable to create user TESTUSER with
>> id 3563
>> Volume 536872154 created on partition /vicepa of myserver
>> Released volume cell.user successfully
>> fs: Invalid argument, possible reasons include:
>> -File not in AFS
>> -Too many users on access control list
>> -Tried to add non-existent user to access control list
>> pts: User or group doesn't exist ; unable to add user TESTUSER to
>> group TESTGROUP
>> pts: User or group doesn't exist ; unable to add user TESTUSER to
>> group TESTAFSUSER
>> Added replication site myserver /vicepa for volume user. TESTUSER
>>
>> Could please someone tell me how to rebuild the protection database?
>> I cant bos on myserver2 because i dont get a ticket from kerberos.
>> or does /etc/init.d/openafs-client stop and
>> /etc/init.d/openafs-fileserver stop the job to stop the whole system?
>>
>> thanks all for your help
>> kanou
>>
>>
>> Am 23.07.2008 um 19:30 schrieb Hartmut Reuter:
>>
>>> kanou wrote:
>>>> My logs on the second machine tell me:
>>>> ==> /var/log/openafs/FileLog.old <==
>>>> Wed Jul 23 19:03:37 2008 File server starting
>>>> Wed Jul 23 19:03:37 2008 afs_krb_get_lrealm failed, myserver2.
>>>> Wed Jul 23 19:03:37 2008 VL_RegisterAddrs rpc failed; will retry
>>>> periodically (code=5376, err=4)
>>>
>>>
>>> code 5376 means no quorum elected. Are you sure your database servers
>>> are all running?
>>>
>>> Try "udebug <server> 7002" for the ptserver
>>> and "udebug <server> 7003" for the vldb
>>>
>>>> Wed Jul 23 19:03:37 2008 Couldn't get CPS for AnyUser, will try
>>>> again in 30 seconds; code=267275.
>>>> ==> /var/log/openafs/SalvageLog <==
>>>> 07/23/2008 19:08:27 SALVAGING OF PARTITION /vicepa COMPLETED
>>>> and aklog gives me:
>>>> aklog: Couldn't get hrf.uni-koeln.de AFS tickets:
>>>> aklog: Cannot contact any KDC for requested realm while getting AFS
>>>> tickets
>>>> damn! i did not do anything on that second one!
>>>>> Just to make sure you're working on the correct file:
>>>>> As I understand you first deleted the file /var/lib/openafs/db/
>>>>> prdb.DB0.
>>>>> This file was then probably recreated when you restarted the ptserver.
>>>>> Run this command on the backupfile you made first (or better on a
>>>>> copy of the backup file).
>>>>>
>>>>> T/Christof
>>>>> ________________________________________
>>>>> From: openafs-info-admin@openafs.org [openafs-info-
>>>>> admin@openafs.org] On Behalf Of kanou [kanou@gmx.ch]
>>>>> Sent: Wednesday, July 23, 2008 6:46 PM
>>>>> To: openafs-info@openafs.org
>>>>> Subject: Re: [OpenAFS] Serious trouble, mounting /afs, ptserver,
>>>>> database rebuilding
>>>>>
>>>>> Thanks for your answer.
>>>>> Well I found the file prdb_check. It doesnt print any errors. Only
>>>>> thing I can find is with
>>>>> ./prdb_check -database /var/lib/openafs/db/prdb.DB0 -uheader -verbose
>>>>> this line:
>>>>> Ubik header size is 0 (should be 64)
>>>>>
>>>>> So there are no errors! I can start the server and everything runs
>>>>> fine but the machine wont mount /afs!
>>>>> kanou
>>>>>
>>>>> Am 23.07.2008 um 17:26 schrieb Steven Jenkins:
>>>>>
>>>>>> On Wed, Jul 23, 2008 at 10:51 AM, kanou <kanou@gmx.ch> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>> well, there is a file called db_verify.c in the folder
>>>>>>> /usr/src/modules/openafs/ptserver but I don' know how to build it.
>>>>>>
>>>>>>
>>>>>> If I recall correctly, db_verify gets renamed to 'prdb_check' during
>>>>>> the install, so you should check for the existence of that file.
>>>>>>
>>>>>> If you can't find it, you'll need to build it from source code: the
>>>>>> directions on the AFSLore wiki are a good place to start:
>>>>>>
>>>>>> http://www.dementia.org/twiki/bin/view/AFSLore/HowToBuildOpenAFSFromSource
>>>>>>
>>>>>>
>>>>>> If you have problems building openafs-stable-1_4_x, you could get
>>>>>> openafs-stable-1_4_7 instead, as that is the latest official release.
>>>>>>
>>>>>> Once you have built the tree, src/ptserver/db_verify should get
>>>>>> built,
>>>>>> so you can simply copy it out of the source tree for your use. If it
>>>>>> doesn't get built automatically for you, you can cd into src/ptserver
>>>>>> and do a 'make db_verify' manuall.
>>>>>>
>>>>>> Also, feel free to ask for help here or on the irc channel.
>>>>>>
>>>>>> Steven Jenkins
>>>>>> End Point Corporation
>>>>>> http://www.endpoint.com/
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenAFS-info mailing list
>>>>> OpenAFS-info@openafs.org
>>>>> https://lists.openafs.org/mailman/listinfo/openafs-info
>>>> _______________________________________________
>>>> OpenAFS-info mailing list
>>>> OpenAFS-info@openafs.org
>>>> https://lists.openafs.org/mailman/listinfo/openafs-info
>>>
>>>
>>> --
>>> -----------------------------------------------------------------
>>> Hartmut Reuter e-mail reuter@rzg.mpg.de
>>> phone +49-89-3299-1328
>>> fax +49-89-3299-1301
>>> RZG (Rechenzentrum Garching) web http://www.rzg.mpg.de/~hwr
>>> Computing Center of the Max-Planck-Gesellschaft (MPG) and the
>>> Institut fuer Plasmaphysik (IPP)
>>> -----------------------------------------------------------------
>>
>> _______________________________________________
>> OpenAFS-info mailing list
>> OpenAFS-info@openafs.org
>> https://lists.openafs.org/mailman/listinfo/openafs-info
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
--------------ms010900030109010507010704
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJeTCC
AxcwggKAoAMCAQICEDsE+kRcmomW1hYG6BoqhGEwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UE
BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT
I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA4MDUzMDE5MTUyOVoX
DTA5MDUzMDE5MTUyOVowczEPMA0GA1UEBBMGQWx0bWFuMRUwEwYDVQQqEwxKZWZmcmV5IEVy
aWMxHDAaBgNVBAMTE0plZmZyZXkgRXJpYyBBbHRtYW4xKzApBgkqhkiG9w0BCQEWHGphbHRt
YW5Ac2VjdXJlLWVuZHBvaW50cy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQCtf5bVJdYFtHIrV2XALpA5oaMu7FPYU7RP7vJhd8Cu9Kd9ud2crX2pHK4avuPaYb4Vg9qI
zPrPadePhJ3OWwNt1ZlUlpc5URnOfpg/I9iymZBUSnCFVLuIvoncacqyUlzqdYEF8XGEoEL6
6bj8uoCSX0D7ZjZiAS8993NvgiPYpf10acMyWQ4max+P7Wg9T03Nw2F6EsmP6gWxBRsekTXe
N6QjJdvaK0846lDqeBFoCEzIUMQXj2kiXVPCPEdxPc/L1sDMYf0GLaDIg8qyThpGd0X6DwfK
3RWcMy8DV7Q5Z+jSEdPn5X0l4anOTrjr3IwE57MC3bVs0EEpUODTzftnAgMBAAGjOTA3MCcG
A1UdEQQgMB6BHGphbHRtYW5Ac2VjdXJlLWVuZHBvaW50cy5jb20wDAYDVR0TAQH/BAIwADAN
BgkqhkiG9w0BAQUFAAOBgQA9kndmeLrdQOUbhNGGms/FnfDyraH4OjA4PIIMOCbGWK0YXczs
/Fqn4XkT70SG4s8v4Zg6TaAcJrZBVcZQXyzrhlF2Zev/g69zZMHQe+2r4i/3FBVKAtFCoea1
vgwJ5TfZYlKvt4D0Z4zexu9Y0VwCIR4plWjVD76zC2CGB/2fhjCCAxcwggKAoAMCAQICEDsE
+kRcmomW1hYG6BoqhGEwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoT
HFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25h
bCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA4MDUzMDE5MTUyOVoXDTA5MDUzMDE5MTUyOVow
czEPMA0GA1UEBBMGQWx0bWFuMRUwEwYDVQQqEwxKZWZmcmV5IEVyaWMxHDAaBgNVBAMTE0pl
ZmZyZXkgRXJpYyBBbHRtYW4xKzApBgkqhkiG9w0BCQEWHGphbHRtYW5Ac2VjdXJlLWVuZHBv
aW50cy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCtf5bVJdYFtHIrV2XA
LpA5oaMu7FPYU7RP7vJhd8Cu9Kd9ud2crX2pHK4avuPaYb4Vg9qIzPrPadePhJ3OWwNt1ZlU
lpc5URnOfpg/I9iymZBUSnCFVLuIvoncacqyUlzqdYEF8XGEoEL66bj8uoCSX0D7ZjZiAS89
93NvgiPYpf10acMyWQ4max+P7Wg9T03Nw2F6EsmP6gWxBRsekTXeN6QjJdvaK0846lDqeBFo
CEzIUMQXj2kiXVPCPEdxPc/L1sDMYf0GLaDIg8qyThpGd0X6DwfK3RWcMy8DV7Q5Z+jSEdPn
5X0l4anOTrjr3IwE57MC3bVs0EEpUODTzftnAgMBAAGjOTA3MCcGA1UdEQQgMB6BHGphbHRt
YW5Ac2VjdXJlLWVuZHBvaW50cy5jb20wDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQUFAAOB
gQA9kndmeLrdQOUbhNGGms/FnfDyraH4OjA4PIIMOCbGWK0YXczs/Fqn4XkT70SG4s8v4Zg6
TaAcJrZBVcZQXyzrhlF2Zev/g69zZMHQe+2r4i/3FBVKAtFCoea1vgwJ5TfZYlKvt4D0Z4ze
xu9Y0VwCIR4plWjVD76zC2CGB/2fhjCCAz8wggKooAMCAQICAQ0wDQYJKoZIhvcNAQEFBQAw
gdExCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUg
VG93bjEaMBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRp
b24gU2VydmljZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFp
bCBDQTErMCkGCSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw0w
MzA3MTcwMDAwMDBaFw0xMzA3MTYyMzU5NTlaMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxU
aGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwg
RnJlZW1haWwgSXNzdWluZyBDQTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAxKY8VXNV
+065yplaHmjAdQRwnd/p/6Me7L3N9VvyGna9fww6YfK/Uc4B1OVQCjDXAmNaLIkVcI7dyfAr
hVqqP3FWy688Cwfn8R+RNiQqE88r1fOCdz0Dviv+uxg+B79AgAJk16emu59l0cUqVIUPSAR/
p7bRPGEEQB5kGXJgt/sCAwEAAaOBlDCBkTASBgNVHRMBAf8ECDAGAQH/AgEAMEMGA1UdHwQ8
MDowOKA2oDSGMmh0dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3dGVQZXJzb25hbEZyZWVtYWls
Q0EuY3JsMAsGA1UdDwQEAwIBBjApBgNVHREEIjAgpB4wHDEaMBgGA1UEAxMRUHJpdmF0ZUxh
YmVsMi0xMzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPqCy7MDaNmrGcPf6+svsIXoUOWlJ1/
TCG4+DYfqi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUaC4J0niVQlGLH2ydxVyWN3amc
OY6MIE9lX5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk13iSx0x1G/11fZU8xggNkMIID
YAIBATB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5
KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQ
OwT6RFyaiZbWFgboGiqEYTAJBgUrDgMCGgUAoIIBwzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcN
AQcBMBwGCSqGSIb3DQEJBTEPFw0wODA3MjQwNjQwMzlaMCMGCSqGSIb3DQEJBDEWBBSi/6sT
5c/rljxjG5Mp1FG2QhbR4jBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqGSIb3DQMHMA4GCCqGSIb3
DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCBhQYJKwYB
BAGCNxAEMXgwdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcg
KFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3Vpbmcg
Q0ECEDsE+kRcmomW1hYG6BoqhGEwgYcGCyqGSIb3DQEJEAILMXigdjBiMQswCQYDVQQGEwJa
QTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhh
d3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEDsE+kRcmomW1hYG6BoqhGEwDQYJ
KoZIhvcNAQEBBQAEggEAg2pu+Xu0RkdwyweF1t/8ZGDYwpptGPhv90JgC8cuPOSFZregSEkz
xwB1PF2Raj40KARvcmXhU8CXHGIJQc6zc4EODp87Sh4r+LJvLmUpt6D+NqYssaW1ZfItDQh/
3+6JTe3kIJZnEob6tG6dF5KH7dstt3GEQpxIt3dZRU0zAKg8CW9y4rGTKN+wrOWwdWpQ6myI
DrY1pR2gBPPzmebg6EGW2WjBVjhdrAcNyW46z5hq55nz8KH2g0zJjpRPYFbPKfX/bwCRqN+i
TtjT3CNQQrB7JwB3N9gzYpdGiZ0pvqHq9ZrE29LS8xeVwYzImkHmvTP6xdnV1CpalW1q5pke
owAAAAAAAA==
--------------ms010900030109010507010704--