[OpenAFS] Root.cell always busy on servers

ted creedon tcreedon@easystreet.com
Wed, 6 Apr 2005 09:19:06 -0700 

Just a note:
1. both 1.2.11 afs servers had a bug in SuSe 9.0 xntpd, hiawatha would go 40
min out of sync wthout notice.
2. this time shift (40 min) may have happened during a vos release 
3. vos dumps appear to work
4. copying all of /afs/.bigcell to a large windows disk seems to work
5. I can't find any way to access /afs/bigcell on either server
6. looks like the database files are corrupted. See below.

Looks like the only way to get a RO volume back up is to completely rebuild
a new (1.3.80 from cvs)server with the vos dump files?

Tedc


hiawatha:/usr/afs/db # prdb_check prdb.DB0 -u
Ubik Header
   Magic           = 0x354545
   Size            = 0
   Version.epoch   = 1075172283
   Version.counter = 4
Ubik header size is 0 (should be 64)
hiawatha:/usr/afs/db # prdb_check prdb.DB0 -h
Usage: prdb_check -database <ptdb_file> [-uheader] [-pheader] [-entries]
[-verbose] [-help]
Where: -uheader  Display UBIK header
       -pheader  Display KADB header
       -entries  Display entries
       -verbose  verbose
hiawatha:/usr/afs/db # prdb_check prdb.DB0 -p
Version           = 0
Header Size       = 65600
Free Ptr          = 0x0
EOF  Ptr          = 0x10580
Max Group     ID  = -205
Max User      ID  = 501
Max Foreign   ID  = 0
Orphaned groups   = 0
User      Count   = 2
Group     Count   = 5
Name Hash         = 8191 buckets
ID   Hash         = 8191 buckets
hiawatha:/usr/afs/db # l
total 268
drwx------    2 root     root          280 Jan 26  2004 ./
drwxr-xr-x   10 root     root          224 Apr  2 09:13 ../
-rw-------    1 root     root         1088 Apr  2 09:13 bdb.DB0
-rw-------    1 root     root           64 Apr  5 09:23 bdb.DBSYS1
-rw-------    1 root     root        34880 Jan 27  2004 kaserver.DB0
-rw-------    1 root     root           64 Apr  5 09:23 kaserver.DBSYS1
-rw-------    1 root     root        67648 Jan 27  2004 prdb.DB0
-rw-------    1 root     root           64 Apr  5 09:23 prdb.DBSYS1
-rw-------    1 root     root       143424 Apr  5 09:28 vldb.DB0
-rw-------    1 root     root           64 Apr  5 09:28 vldb.DBSYS1
hiawatha:/usr/afs/db # prdb_check prdb.DBSYS1 -p
db_verify: server or network not responding couldn't read header
hiawatha:/usr/afs/db # prdb_check prdb.DBSYS1 -u
Ubik Header
   Magic           = 0x0
   Size            = 0
   Version.epoch   = 0
   Version.counter = 0
Ubik header size is 0 (should be 64)
Ubik header magic is 0x0 (should be 0x354545)
db_verify: server or network not responding couldn't read header
hiawatha:/usr/afs/db # kadb_check kaserver.DBSYS1 -u
Ubik Header
   Magic           = 0x0
   Size            = 0
   Version.epoch   = 0
   Version.counter = 0
Ubik header size is 0 (should be 64)
Ubik header magic is 0x0 (should be 0x354545)
kadb_check: reading db got 0 bytes
hiawatha:/usr/afs/db # kadb_check kaserver.DB0 -u
Ubik Header
   Magic           = 0x354545
   Size            = 0
   Version.epoch   = 1075172135
   Version.counter = 12
Ubik header size is 0 (should be 64)
Entry 0 is unrecognizable
Entry 1 is unrecognizable
Entry 2 is unrecognizable
Entry 3 is unrecognizable
Entry 4 is unrecognizable
Entry 5 is unrecognizable
Entry 6 is unrecognizable
Entry 7 is unrecognizable
Entry 8 is unrecognizable
kadb_check: reading db got 0 bytes
hiawatha:/usr/afs/db # vldb_check kaserver.DBSYS1 -u
Ubik Header
   Magic           = 0x0
   Size            = 0
   Version.epoch   = 0
   Version.counter = 0
Ubik header size is 0 (should be 64)
Ubik header magic is 0x0 (should be 0x354545)
error: read of 132120 bytes failed: 0 0
Header reports its size as 0 (should be 132120)
does not support vldb with version less than 3
hiawatha:/usr/afs/db # vldb_check kaserver.DB0 -u
Ubik Header
   Magic           = 0x354545
   Size            = 0
   Version.epoch   = 1075172135
   Version.counter = 12
Ubik header size is 0 (should be 64)
error: read of 132120 bytes failed: 34816 0
Header reports its size as 32832 (should be 132120)
VLDB entry 'Server' contains an unknown RW/RO index serverFlag
   index 0 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 1 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 2 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 3 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 4 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 5 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 6 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 7 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 8 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 9 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 10 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 11 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 12 : serverNumber 0 : serverPartition 0 : serverFlag 0
WARNING: VLDB entry '' has no RW volume
VLDB entry '' contains an unknown RW/RO index serverFlag
   index 0 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 1 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 2 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 3 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 4 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 5 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 6 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 7 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 8 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 9 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 10 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 11 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 12 : serverNumber 0 : serverPartition 0 : serverFlag 0
WARNING: VLDB entry '' has no RW volume
VLDB entry '' contains an unknown RW/RO index serverFlag
   index 0 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 1 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 2 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 3 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 4 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 5 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 6 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 7 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 8 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 9 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 10 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 11 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 12 : serverNumber 0 : serverPartition 0 : serverFlag 0
WARNING: VLDB entry '' has no RW volume
VLDB entry '' contains an unknown RW/RO index serverFlag
   index 0 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 1 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 2 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 3 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 4 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 5 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 6 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 7 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 8 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 9 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 10 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 11 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 12 : serverNumber 0 : serverPartition 0 : serverFlag 0
WARNING: VLDB entry '' has no RW volume
VLDB entry '' contains an unknown RW/RO index serverFlag
   index 0 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 1 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 2 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 3 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 4 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 5 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 6 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 7 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 8 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 9 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 10 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 11 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 12 : serverNumber 0 : serverPartition 0 : serverFlag 0
WARNING: VLDB entry '' has no RW volume
VLDB entry '' contains an unknown RW/RO index serverFlag
   index 0 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 1 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 2 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 3 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 4 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 5 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 6 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 7 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 8 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 9 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 10 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 11 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 12 : serverNumber 0 : serverPartition 0 : serverFlag 0
WARNING: VLDB entry '' has no RW volume
VLDB entry '' contains an unknown RW/RO index serverFlag
   index 0 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 1 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 2 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 3 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 4 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 5 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 6 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 7 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 8 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 9 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 10 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 11 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 12 : serverNumber 0 : serverPartition 0 : serverFlag 0
WARNING: VLDB entry '' has no RW volume
VLDB entry '' contains an unknown RW/RO index serverFlag
   index 0 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 1 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 2 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 3 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 4 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 5 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 6 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 7 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 8 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 9 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 10 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 11 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 12 : serverNumber 0 : serverPartition 0 : serverFlag 0
WARNING: VLDB entry '' has no RW volume
VLDB entry '' contains an unknown RW/RO index serverFlag
   index 0 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 1 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 2 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 3 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 4 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 5 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 6 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 7 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 8 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 9 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 10 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 11 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 12 : serverNumber 0 : serverPartition 0 : serverFlag 0
WARNING: VLDB entry '' has no RW volume
VLDB entry '' contains an unknown RW/RO index serverFlag
   index 0 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 1 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 2 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 3 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 4 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 5 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 6 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 7 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 8 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 9 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 10 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 11 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 12 : serverNumber 0 : serverPartition 0 : serverFlag 0
WARNING: VLDB entry '' has no RW volume
VLDB entry '' contains an unknown RW/RO index serverFlag
   index 0 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 1 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 2 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 3 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 4 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 5 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 6 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 7 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 8 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 9 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 10 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 11 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 12 : serverNumber 0 : serverPartition 0 : serverFlag 0
WARNING: VLDB entry '' has no RW volume
VLDB entry '' contains an unknown RW/RO index serverFlag
   index 0 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 1 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 2 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 3 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 4 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 5 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 6 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 7 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 8 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 9 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 10 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 11 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 12 : serverNumber 0 : serverPartition 0 : serverFlag 0
WARNING: VLDB entry '' has no RW volume
VLDB entry '' contains an unknown RW/RO index serverFlag
   index 0 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 1 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 2 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 3 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 4 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 5 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 6 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 7 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 8 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 9 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 10 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 11 : serverNumber 0 : serverPartition 0 : serverFlag 0
   index 12 : serverNumber 0 : serverPartition 0 : serverFlag 0
Header's maximum volume id is 9 and largest id found in VLDB is 4294967295
error: read of 8192 bytes failed: 0 0
Multihomed Block 0: Bad entry at 905117631: Not a valid multihomed block
MH block does not point to self 905117631 in header, 0 in block
INTERNAL ERROR: addresses 33720 and 33632 use same record slot 227
Name Hash 832: Bad entry '': Incorrect name hash chain (should be in 0)
error: read of 148 bytes failed: 0 0
Segmentation fault
hiawatha:/usr/afs/db # l
total 268
drwx------    2 root     root          280 Jan 26  2004 ./
drwxr-xr-x   10 root     root          224 Apr  2 09:13 ../
-rw-------    1 root     root         1088 Apr  2 09:13 bdb.DB0
-rw-------    1 root     root           64 Apr  5 09:23 bdb.DBSYS1
-rw-------    1 root     root        34880 Jan 27  2004 kaserver.DB0
-rw-------    1 root     root           64 Apr  5 09:23 kaserver.DBSYS1
-rw-------    1 root     root        67648 Jan 27  2004 prdb.DB0
-rw-------    1 root     root           64 Apr  5 09:23 prdb.DBSYS1
-rw-------    1 root     root       143424 Apr  5 09:28 vldb.DB0
-rw-------    1 root     root           64 Apr  5 09:28 vldb.DBSYS1
hiawatha:/usr/afs/db # bdb_check kaserver.DB0 -u
bash: bdb_check: command not found
hiawatha:/usr/afs/db #
hiawatha:/usr/afs/db #