[OpenAFS] Re: CopyOnWrite failure, leading to volume salvage

[Andrew Deason]

On Thu, 27 Sep 2012 10:45:45 +0100
Dameon Wagner <dameon.wagner@it.ox.ac.uk> wrote:

> #---8<-----------------------------------------------------------------
> CopyOnWrite failed: Partition /vicepa that contains volume 536874907 may be out of free inodes(errno = 2)
> Alloc_NewVnode: partition /vicepa idec 9492608003410095 failed
> Volume : 536874907 vnode = 592047 Failed to create inode: errno = 2
> #---8<-----------------------------------------------------------------
> 
> Troubleshooting showed that there was no shortage of inodes, so that
> wasn't the underlying issue (I read it as only a likely suggestion, or
> one possible cause).

It's not going to be that with that errno, I don't think. If there was
an actual error in creating the relevant file due to space issues you
would probably actually see ENOSPC. ENOENT is probably something just
wrong with the volume metadata.

If the linktable file somehow got suddenly deleted (or maybe
corrupted?), I think you'd get this. That would also cause the idec to
fail, and it would be something the salvager will correct (possibly
silently). I'm guessing, though; hard to say without being able to see
what was in /vicepX at the time.

> In the end, `bos salvage` with "-volume 536874907" fixed everything,
> with no known loss or corruption of data.  For the record, SalvageLog
> contained many lines (just over a 1000) like the following first
> three:
> 
> #---8<-----------------------------------------------------------------
> Vnode 16928: version < inode version; fixed (old status)
> Vnode 23352: version < inode version; fixed (old status)
> Vnode 53568: version < inode version; fixed (old status)
> ... ending with
> totalInodes 1690748
> Salvaged vhost.a071 (536874907): 939204 files, 361363522 blocks
> #---8<-----------------------------------------------------------------

Is there anything in there besides the 'version < inode version'
messages? (Those ones are (almost always) harmless.)

Can you 'vos examine' the relevant volume? If you don't want to make
public the information like volume/server names, you can obfuscate
stuff; I'd just want to see what the server/partition layout looks like
for that volume's sites.

-- 
Andrew Deason
adeason@sinenomine.net