[OpenAFS] Error in volume - anything that can be salvaged?

Harald Barth haba@kth.se
Tue, 11 Aug 2015 10:27:38 +0200 (CEST)


Something is really fishy with this volume....

This volume was salvaged recently after some power failures and a
scripted vos move. So during the vos move it was good enough
to be moved. After that it was salvaged (forceDAFS) now it's
broken beyond vos dump. See below.


07/18/2015 11:43:44 dispatching child to salvage volume 536889218...
07/18/2015 11:43:44 1 nVolumesInInodeFile 32 
07/18/2015 11:43:44 SALVAGING VOLUME 536889218.
07/18/2015 11:43:44 ftp.free.doc (536889218) updated 02/28/2011 15:00
07/18/2015 11:43:44 totalInodes 3596
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25540.13495
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25542.13496
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25544.13497
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25546.13498
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25548.13499
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25550.13500
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25552.13501
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25554.13502
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25556.13503
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25558.13504
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25560.13505
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25562.13506
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25564.13507
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25566.13508
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25568.13509
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25570.13510
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25572.13511
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25574.13512
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25576.13513
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25578.13514
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25580.13515
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25582.13516
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25584.13517
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25586.13518
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25588.13519
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25590.13520
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25592.13521
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25594.13522
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25596.13523
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25598.13524
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25600.13525
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25602.13526
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25604.13527
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25606.13528
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25608.13529
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25610.13530
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25612.13531
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25614.13532
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25616.13533
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25618.13534
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25620.13535
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25622.13536
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25624.13537
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25626.13538
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25628.13539
07/18/2015 11:43:44 Attaching orphaned file to volume's root dir as __ORPHANFILE__.25630.13540
07/18/2015 11:43:44 Vnode 25540: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25542: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25544: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25546: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25548: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25550: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25552: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25554: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25556: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25558: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25560: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25562: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25564: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25566: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25568: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25570: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25572: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25574: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25576: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25578: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25580: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25582: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25584: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25586: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25588: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25590: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25592: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25594: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25596: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25598: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25600: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25602: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25604: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25606: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25608: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25610: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25612: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25614: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25616: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25618: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25620: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25622: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25624: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25626: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25628: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Vnode 25630: link count incorrect (was 0, now 1)
07/18/2015 11:43:44 Salvaged ftp.free.doc (536889218): 3592 files, 761836 blocks


I think since that, the volume and it's backup volume can not be dumped:

# vos dump -id ftp.free.doc.backup -time 0 -localauth > /dev/null
Error in rx_EndCall
VOLSER:  Problems encountered in reading the dump file !
Error in vos dump command.
VOLSER:  Problems encountered in reading the dump file !
# vos dump -id ftp.free.doc -time 0 -localauth > /dev/null
Error in rx_EndCall
VOLSER:  Problems encountered in reading the dump file !
Error in vos dump command.
VOLSER:  Problems encountered in reading the dump file !

With the log:

Tue Aug 11 09:28:10 2015 DumpVnode: volume 536889218 vnode 1 has inconsistent length (index 2048 disk 6144); aborting dump
Tue Aug 11 09:30:23 2015 DumpVnode: volume 536889218 vnode 1 has inconsistent length (index 2048 disk 6144); aborting dump
Tue Aug 11 09:30:41 2015 DumpVnode: volume 536889218 vnode 1 has inconsistent length (index 2048 disk 6144); aborting dump
Tue Aug 11 09:56:28 2015 DumpVnode: volume 536889220 vnode 1 has inconsistent length (index 2048 disk 6144); aborting dump

Now I have a backup from before it broke and the volume had not been
changed for a long time, but would this be possible to fix?

And I don't like that salvage leaves a volume in a state that can not be dumped.

The version in question is debian package 1.6.9-2+deb8u2.

Harald.