[OpenAFS] Re: Crazy DAFS problem (with log)

[Ryan C. Underwood]

On Sun, Mar 20, 2011 at 09:57:22PM -0500, Andrew Deason wrote:
> 
> Oh, hmm, this starts too late. I guess you don't have the BosLog before
> this?

Nope, wish it rotated more old logs out but haven't looked into this.

> > 03/20/2011 16:36:17 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
> > 03/20/2011 16:40:14 dispatching child to salvage volume 536871273...
> > 03/20/2011 16:40:14 2 nVolumesInInodeFile 56 
> > 03/20/2011 16:40:14 CHECKING CLONED VOLUME 536871274.
> > 03/20/2011 16:40:14 www_logs.readonly (536871274) updated 03/20/2011 16:39
> > 03/20/2011 16:40:14 Vnode 22: length incorrect; (is 1794950 should be 0)
> 
> Do you know if this volume was recreated between these times? I'm not
> sure if the salvager is just lying, but this would indicate it deleted
> the RO, and then it came back again and again has invalid metadata.

If by recreated you mean a remsite/addsite, the answer is no.

> > [.. ad infinitum ]
> 
> Actually, it may help to continue for an interation or two more. I don't
> think these salvages have repeated themselves yet.

Will attach at the end.

> > It's a RO clone, should I just delete it and add it back?
> 
> Yes, if it'll let you. The salvage is supposed to delete it for you, but
> if it's not it's not.

I deleted all the clones in question and manually salvaged the RW
volumes then recreated and re-released the clones.  Let's see if the
gremlin comes back.

> Based on the salsrv logs, I'd be looking at vnodes 264, 266, and 268 in
> 536870915/6, and 22, 202, and 204 in 536871273/4. That is, see if the
> 'UFS-Filename' file on disk actually has the length in the 'volinfo'
> output.

If/when it comes back I'll dig into this stuff.

> >>>Shortly after the weekly scheduled fileserver restart
> 
> You shouldn't need to have these on. But they shouldn't be causing
> salvages or whatever, either.

OK, this used to be standard practice, but I set up back in the 1.2 days
so a lot could have changed since then.  I'll remove it.

Long SalsrvLog excerpt:

@(#) OpenAFS 1.6.0~pre3-0-debian built  2011-03-18 
03/20/2011 04:00:29 Starting OpenAFS Online Salvage Server 2.4 (/usr/lib/openafs/salvageserver)
03/20/2011 04:02:02 dispatching child to salvage volume 536870915...
03/20/2011 04:02:02 2 nVolumesInInodeFile 56 
03/20/2011 04:02:02 waiting for fileserver to finish scanning partition /vicepa...
03/20/2011 04:02:03 CHECKING CLONED VOLUME 536870916.
03/20/2011 04:02:03 root.cell.readonly (536870916) updated 03/20/2011 00:12
03/20/2011 04:02:03 Vnode 266: length incorrect; (is 2587609 should be 0)
03/20/2011 04:02:03 SALVAGING VOLUME 536870915.
03/20/2011 04:02:03 root.cell (536870915) updated 03/20/2011 00:12
03/20/2011 04:02:03 Vnode 266: length incorrect; changed from 2587609 to 0
03/20/2011 04:02:03 totalInodes 190
03/20/2011 04:02:03 Salvaged root.cell (536870915): 183 files, 62821 blocks
03/20/2011 04:02:05 dispatching child to salvage volume 536871273...
03/20/2011 04:02:05 2 nVolumesInInodeFile 56 
03/20/2011 04:02:05 CHECKING CLONED VOLUME 536871274.
03/20/2011 04:02:05 www_logs.readonly (536871274) updated 03/19/2011 18:35
03/20/2011 04:02:05 Vnode 22: length incorrect; (is 1683156 should be 0)
03/20/2011 04:02:05 SALVAGING VOLUME 536871273.
03/20/2011 04:02:05 www_logs (536871273) updated 03/19/2011 18:35
03/20/2011 04:02:05 Vnode 22: length incorrect; changed from 1683156 to 0
03/20/2011 04:02:05 Vnode 24: length incorrect; changed from 184572 to 0
03/20/2011 04:02:05 Vnode 202: length incorrect; changed from 232747 to 0
03/20/2011 04:02:05 totalInodes 245
03/20/2011 04:02:05 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 04:04:01 dispatching child to salvage volume 536870916...
03/20/2011 04:04:01 dispatching child to salvage volume 536870915...
03/20/2011 04:04:01 namei_ListAFSSubDirs: warning: VG 536870916 does not have a link table; salvager will recreate it.
03/20/2011 04:04:01 fileserver requested salvage of clone 536870916; scheduling salvage of volume group 536870915...
03/20/2011 04:04:01 1 nVolumesInInodeFile 28 
03/20/2011 04:04:01 SALVAGING VOLUME 536870915.
03/20/2011 04:04:01 root.cell (536870915) updated 03/20/2011 04:03
03/20/2011 04:04:01 totalInodes 187
03/20/2011 04:04:01 Salvaged root.cell (536870915): 183 files, 62824 blocks
03/20/2011 04:04:01 The volume header file /vicepa/V0536870916.vol is not associated with any actual data (deleted)
03/20/2011 04:04:13 dispatching child to salvage volume 536871274...
03/20/2011 04:04:13 dispatching child to salvage volume 536871273...
03/20/2011 04:04:13 namei_ListAFSSubDirs: warning: VG 536871274 does not have a link table; salvager will recreate it.
03/20/2011 04:04:13 fileserver requested salvage of clone 536871274; scheduling salvage of volume group 536871273...
03/20/2011 04:04:13 1 nVolumesInInodeFile 28 
03/20/2011 04:04:13 SALVAGING VOLUME 536871273.
03/20/2011 04:04:13 www_logs (536871273) updated 03/19/2011 18:35
03/20/2011 04:04:13 totalInodes 242
03/20/2011 04:04:13 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 04:04:13 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 04:14:14 dispatching child to salvage volume 536871273...
03/20/2011 04:14:14 2 nVolumesInInodeFile 56 
03/20/2011 04:14:14 CHECKING CLONED VOLUME 536871274.
03/20/2011 04:14:14 www_logs.readonly (536871274) updated 03/20/2011 04:14
03/20/2011 04:14:14 Vnode 22: length incorrect; (is 1695692 should be 0)
03/20/2011 04:14:14 SALVAGING VOLUME 536871273.
03/20/2011 04:14:14 www_logs (536871273) updated 03/20/2011 04:14
03/20/2011 04:14:14 Vnode 22: length incorrect; changed from 1695692 to 0
03/20/2011 04:14:14 Vnode 202: length incorrect; changed from 288593 to 0
03/20/2011 04:14:14 totalInodes 246
03/20/2011 04:14:14 Salvaged www_logs (536871273): 238 files, 242062 blocks
03/20/2011 04:16:02 dispatching child to salvage volume 536871273...
03/20/2011 04:16:02 1 nVolumesInInodeFile 28 
03/20/2011 04:16:02 SALVAGING VOLUME 536871273.
03/20/2011 04:16:02 www_logs (536871273) updated 03/20/2011 04:15
03/20/2011 04:16:02 totalInodes 242
03/20/2011 04:16:02 Salvaged www_logs (536871273): 238 files, 244003 blocks
03/20/2011 04:16:02 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 04:20:04 dispatching child to salvage volume 536871273...
03/20/2011 04:20:04 2 nVolumesInInodeFile 56 
03/20/2011 04:20:04 CHECKING CLONED VOLUME 536871274.
03/20/2011 04:20:04 www_logs.readonly (536871274) updated 03/20/2011 04:20
03/20/2011 04:20:04 Vnode 22: length incorrect; (is 1698459 should be 0)
03/20/2011 04:20:04 SALVAGING VOLUME 536871273.
03/20/2011 04:20:04 www_logs (536871273) updated 03/20/2011 04:20
03/20/2011 04:20:04 Vnode 22: length incorrect; changed from 1698459 to 0
03/20/2011 04:20:04 Vnode 24: length incorrect; changed from 253718 to 0
03/20/2011 04:20:04 Vnode 202: length incorrect; changed from 305092 to 0
03/20/2011 04:20:04 totalInodes 245
03/20/2011 04:20:04 Salvaged www_logs (536871273): 238 files, 241896 blocks
03/20/2011 04:20:07 dispatching child to salvage volume 536871273...
03/20/2011 04:20:07 1 nVolumesInInodeFile 28 
03/20/2011 04:20:07 SALVAGING VOLUME 536871273.
03/20/2011 04:20:07 www_logs (536871273) updated 03/20/2011 04:20
03/20/2011 04:20:07 totalInodes 242
03/20/2011 04:20:07 Salvaged www_logs (536871273): 238 files, 241896 blocks
03/20/2011 04:20:07 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 04:26:03 dispatching child to salvage volume 536871273...
03/20/2011 04:26:03 2 nVolumesInInodeFile 56 
03/20/2011 04:26:03 CHECKING CLONED VOLUME 536871274.
03/20/2011 04:26:03 www_logs.readonly (536871274) updated 03/20/2011 04:25
03/20/2011 04:26:03 Vnode 22: length incorrect; (is 1710976 should be 0)
03/20/2011 04:26:03 SALVAGING VOLUME 536871273.
03/20/2011 04:26:03 www_logs (536871273) updated 03/20/2011 04:25
03/20/2011 04:26:03 Vnode 22: length incorrect; changed from 1710976 to 0
03/20/2011 04:26:03 Vnode 24: length incorrect; changed from 254830 to 0
03/20/2011 04:26:03 Vnode 202: length incorrect; changed from 325308 to 0
03/20/2011 04:26:03 totalInodes 245
03/20/2011 04:26:03 Salvaged www_logs (536871273): 238 files, 241896 blocks
03/20/2011 04:26:08 dispatching child to salvage volume 536871273...
03/20/2011 04:26:08 1 nVolumesInInodeFile 28 
03/20/2011 04:26:08 SALVAGING VOLUME 536871273.
03/20/2011 04:26:08 www_logs (536871273) updated 03/20/2011 04:26
03/20/2011 04:26:08 totalInodes 242
03/20/2011 04:26:08 Salvaged www_logs (536871273): 238 files, 243884 blocks
03/20/2011 04:26:08 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 04:32:54 dispatching child to salvage volume 536871273...
03/20/2011 04:32:54 2 nVolumesInInodeFile 56 
03/20/2011 04:32:54 CHECKING CLONED VOLUME 536871274.
03/20/2011 04:32:54 www_logs.readonly (536871274) updated 03/20/2011 04:31
03/20/2011 04:32:54 Vnode 22: length incorrect; (is 1717139 should be 0)
03/20/2011 04:32:54 SALVAGING VOLUME 536871273.
03/20/2011 04:32:54 www_logs (536871273) updated 03/20/2011 04:31
03/20/2011 04:32:54 Vnode 22: length incorrect; changed from 1717139 to 0
03/20/2011 04:32:54 Vnode 24: length incorrect; changed from 255559 to 0
03/20/2011 04:32:54 Vnode 202: length incorrect; changed from 327428 to 0
03/20/2011 04:32:54 Vnode 204: length incorrect; changed from 83927 to 0
03/20/2011 04:32:54 totalInodes 245
03/20/2011 04:32:54 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 04:34:10 dispatching child to salvage volume 536871273...
03/20/2011 04:34:10 1 nVolumesInInodeFile 28 
03/20/2011 04:34:10 SALVAGING VOLUME 536871273.
03/20/2011 04:34:10 www_logs (536871273) updated 03/20/2011 04:34
03/20/2011 04:34:10 totalInodes 242
03/20/2011 04:34:10 Salvaged www_logs (536871273): 238 files, 243742 blocks
03/20/2011 04:34:10 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 04:35:51 dispatching child to salvage volume 536870915...
03/20/2011 04:35:51 2 nVolumesInInodeFile 56 
03/20/2011 04:35:51 CHECKING CLONED VOLUME 536870916.
03/20/2011 04:35:51 root.cell.readonly (536870916) updated 03/20/2011 04:33
03/20/2011 04:35:51 Vnode 264: length incorrect; (is 15833170 should be 0)
03/20/2011 04:35:51 SALVAGING VOLUME 536870915.
03/20/2011 04:35:51 root.cell (536870915) updated 03/20/2011 04:33
03/20/2011 04:35:51 Vnode 264: length incorrect; changed from 15833170 to 0
03/20/2011 04:35:51 totalInodes 190
03/20/2011 04:35:51 Salvaged root.cell (536870915): 183 files, 47363 blocks
03/20/2011 04:36:01 dispatching child to salvage volume 536870915...
03/20/2011 04:36:01 1 nVolumesInInodeFile 28 
03/20/2011 04:36:01 SALVAGING VOLUME 536870915.
03/20/2011 04:36:01 root.cell (536870915) updated 03/20/2011 04:33
03/20/2011 04:36:01 totalInodes 187
03/20/2011 04:36:01 Salvaged root.cell (536870915): 183 files, 47363 blocks
03/20/2011 04:36:01 The volume header file /vicepa/V0536870916.vol is not associated with any actual data (deleted)
03/20/2011 04:38:13 dispatching child to salvage volume 536871273...
03/20/2011 04:38:13 2 nVolumesInInodeFile 56 
03/20/2011 04:38:13 CHECKING CLONED VOLUME 536871274.
03/20/2011 04:38:13 www_logs.readonly (536871274) updated 03/20/2011 04:36
03/20/2011 04:38:13 Vnode 22: length incorrect; (is 1720552 should be 0)
03/20/2011 04:38:13 SALVAGING VOLUME 536871273.
03/20/2011 04:38:13 www_logs (536871273) updated 03/20/2011 04:36
03/20/2011 04:38:13 Vnode 22: length incorrect; changed from 1720552 to 0
03/20/2011 04:38:13 totalInodes 245
03/20/2011 04:38:13 Salvaged www_logs (536871273): 238 files, 242064 blocks
03/20/2011 04:38:18 dispatching child to salvage volume 536871273...
03/20/2011 04:38:18 1 nVolumesInInodeFile 28 
03/20/2011 04:38:18 SALVAGING VOLUME 536871273.
03/20/2011 04:38:18 www_logs (536871273) updated 03/20/2011 04:36
03/20/2011 04:38:18 totalInodes 242
03/20/2011 04:38:18 Salvaged www_logs (536871273): 238 files, 242064 blocks
03/20/2011 04:38:18 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 04:44:26 dispatching child to salvage volume 536871273...
03/20/2011 04:44:26 2 nVolumesInInodeFile 56 
03/20/2011 04:44:26 CHECKING CLONED VOLUME 536871274.
03/20/2011 04:44:26 www_logs.readonly (536871274) updated 03/20/2011 04:43
03/20/2011 04:44:26 Vnode 22: length incorrect; (is 1726514 should be 0)
03/20/2011 04:44:26 SALVAGING VOLUME 536871273.
03/20/2011 04:44:26 www_logs (536871273) updated 03/20/2011 04:43
03/20/2011 04:44:26 Vnode 22: length incorrect; changed from 1726514 to 0
03/20/2011 04:44:26 Vnode 24: length incorrect; changed from 256357 to 0
03/20/2011 04:44:26 Vnode 202: length incorrect; changed from 328016 to 0
03/20/2011 04:44:26 Vnode 204: length incorrect; changed from 84049 to 0
03/20/2011 04:44:26 totalInodes 245
03/20/2011 04:44:26 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 04:46:02 dispatching child to salvage volume 536871273...
03/20/2011 04:46:02 1 nVolumesInInodeFile 28 
03/20/2011 04:46:02 SALVAGING VOLUME 536871273.
03/20/2011 04:46:02 www_logs (536871273) updated 03/20/2011 04:45
03/20/2011 04:46:02 totalInodes 242
03/20/2011 04:46:02 Salvaged www_logs (536871273): 238 files, 243501 blocks
03/20/2011 04:46:02 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 04:50:09 dispatching child to salvage volume 536871273...
03/20/2011 04:50:09 2 nVolumesInInodeFile 56 
03/20/2011 04:50:09 CHECKING CLONED VOLUME 536871274.
03/20/2011 04:50:09 www_logs.readonly (536871274) updated 03/20/2011 04:49
03/20/2011 04:50:09 Vnode 22: length incorrect; (is 1730700 should be 0)
03/20/2011 04:50:09 SALVAGING VOLUME 536871273.
03/20/2011 04:50:09 www_logs (536871273) updated 03/20/2011 04:49
03/20/2011 04:50:09 Vnode 22: length incorrect; changed from 1730700 to 0
03/20/2011 04:50:09 Vnode 24: length incorrect; changed from 256955 to 0
03/20/2011 04:50:09 totalInodes 245
03/20/2011 04:50:09 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 04:50:13 dispatching child to salvage volume 536871273...
03/20/2011 04:50:13 1 nVolumesInInodeFile 28 
03/20/2011 04:50:13 SALVAGING VOLUME 536871273.
03/20/2011 04:50:13 www_logs (536871273) updated 03/20/2011 04:49
03/20/2011 04:50:13 totalInodes 242
03/20/2011 04:50:13 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 04:50:13 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 04:57:12 dispatching child to salvage volume 536871273...
03/20/2011 04:57:12 2 nVolumesInInodeFile 56 
03/20/2011 04:57:12 CHECKING CLONED VOLUME 536871274.
03/20/2011 04:57:12 www_logs.readonly (536871274) updated 03/20/2011 04:55
03/20/2011 04:57:12 Vnode 22: length incorrect; (is 1740719 should be 0)
03/20/2011 04:57:12 SALVAGING VOLUME 536871273.
03/20/2011 04:57:12 www_logs (536871273) updated 03/20/2011 04:57
03/20/2011 04:57:12 Vnode 22: length incorrect; changed from 1740719 to 0
03/20/2011 04:57:12 Vnode 24: length incorrect; changed from 258082 to 0
03/20/2011 04:57:12 totalInodes 246
03/20/2011 04:57:12 Salvaged www_logs (536871273): 238 files, 242135 blocks
03/20/2011 04:58:02 dispatching child to salvage volume 536871273...
03/20/2011 04:58:02 1 nVolumesInInodeFile 28 
03/20/2011 04:58:02 SALVAGING VOLUME 536871273.
03/20/2011 04:58:02 www_logs (536871273) updated 03/20/2011 04:57
03/20/2011 04:58:02 totalInodes 242
03/20/2011 04:58:03 Salvaged www_logs (536871273): 238 files, 244087 blocks
03/20/2011 04:58:03 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 05:02:22 dispatching child to salvage volume 536871273...
03/20/2011 05:02:22 2 nVolumesInInodeFile 56 
03/20/2011 05:02:22 CHECKING CLONED VOLUME 536871274.
03/20/2011 05:02:22 www_logs.readonly (536871274) updated 03/20/2011 05:01
03/20/2011 05:02:22 Vnode 22: length incorrect; (is 1743983 should be 0)
03/20/2011 05:02:22 SALVAGING VOLUME 536871273.
03/20/2011 05:02:22 www_logs (536871273) updated 03/20/2011 05:01
03/20/2011 05:02:22 Vnode 22: length incorrect; changed from 1743983 to 0
03/20/2011 05:02:22 Vnode 24: length incorrect; changed from 258420 to 0
03/20/2011 05:02:22 Vnode 202: length incorrect; changed from 330797 to 0
03/20/2011 05:02:22 Vnode 204: length incorrect; changed from 85938 to 0
03/20/2011 05:02:22 totalInodes 245
03/20/2011 05:02:22 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 05:04:02 dispatching child to salvage volume 536871273...
03/20/2011 05:04:02 1 nVolumesInInodeFile 28 
03/20/2011 05:04:02 SALVAGING VOLUME 536871273.
03/20/2011 05:04:02 www_logs (536871273) updated 03/20/2011 05:03
03/20/2011 05:04:02 totalInodes 242
03/20/2011 05:04:02 Salvaged www_logs (536871273): 238 files, 243771 blocks
03/20/2011 05:04:02 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 05:04:32 dispatching child to salvage volume 536870915...
03/20/2011 05:04:32 2 nVolumesInInodeFile 56 
03/20/2011 05:04:32 CHECKING CLONED VOLUME 536870916.
03/20/2011 05:04:32 root.cell.readonly (536870916) updated 03/20/2011 04:49
03/20/2011 05:04:32 Vnode 264: length incorrect; (is 15834322 should be 0)
03/20/2011 05:04:32 SALVAGING VOLUME 536870915.
03/20/2011 05:04:32 root.cell (536870915) updated 03/20/2011 04:49
03/20/2011 05:04:32 Vnode 264: length incorrect; changed from 15834322 to 0
03/20/2011 05:04:32 totalInodes 190
03/20/2011 05:04:32 Salvaged root.cell (536870915): 183 files, 50649 blocks
03/20/2011 05:06:01 dispatching child to salvage volume 536870915...
03/20/2011 05:06:01 1 nVolumesInInodeFile 28 
03/20/2011 05:06:01 SALVAGING VOLUME 536870915.
03/20/2011 05:06:01 root.cell (536870915) updated 03/20/2011 05:06
03/20/2011 05:06:01 totalInodes 187
03/20/2011 05:06:01 Salvaged root.cell (536870915): 183 files, 66112 blocks
03/20/2011 05:06:01 The volume header file /vicepa/V0536870916.vol is not associated with any actual data (deleted)
03/20/2011 05:10:04 dispatching child to salvage volume 536871273...
03/20/2011 05:10:04 2 nVolumesInInodeFile 56 
03/20/2011 05:10:04 CHECKING CLONED VOLUME 536871274.
03/20/2011 05:10:04 www_logs.readonly (536871274) updated 03/20/2011 05:07
03/20/2011 05:10:04 Vnode 22: length incorrect; (is 1746913 should be 0)
03/20/2011 05:10:04 SALVAGING VOLUME 536871273.
03/20/2011 05:10:04 www_logs (536871273) updated 03/20/2011 05:07
03/20/2011 05:10:04 Vnode 22: length incorrect; changed from 1746913 to 0
03/20/2011 05:10:04 Vnode 24: length incorrect; changed from 258827 to 0
03/20/2011 05:10:04 Vnode 202: length incorrect; changed from 331126 to 0
03/20/2011 05:10:04 Vnode 204: length incorrect; changed from 86064 to 0
03/20/2011 05:10:04 totalInodes 245
03/20/2011 05:10:04 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 05:12:11 dispatching child to salvage volume 536871273...
03/20/2011 05:12:11 1 nVolumesInInodeFile 28 
03/20/2011 05:12:11 SALVAGING VOLUME 536871273.
03/20/2011 05:12:11 www_logs (536871273) updated 03/20/2011 05:11
03/20/2011 05:12:11 totalInodes 242
03/20/2011 05:12:11 Salvaged www_logs (536871273): 238 files, 243522 blocks
03/20/2011 05:12:11 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 05:17:20 dispatching child to salvage volume 536871273...
03/20/2011 05:17:20 2 nVolumesInInodeFile 56 
03/20/2011 05:17:20 CHECKING CLONED VOLUME 536871274.
03/20/2011 05:17:20 www_logs.readonly (536871274) updated 03/20/2011 05:15
03/20/2011 05:17:20 Vnode 22: length incorrect; (is 1750652 should be 0)
03/20/2011 05:17:20 SALVAGING VOLUME 536871273.
03/20/2011 05:17:20 www_logs (536871273) updated 03/20/2011 05:15
03/20/2011 05:17:20 Vnode 22: length incorrect; changed from 1750652 to 0
03/20/2011 05:17:20 Vnode 202: length incorrect; changed from 331721 to 0
03/20/2011 05:17:20 totalInodes 245
03/20/2011 05:17:20 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 05:18:02 dispatching child to salvage volume 536871273...
03/20/2011 05:18:02 1 nVolumesInInodeFile 28 
03/20/2011 05:18:02 SALVAGING VOLUME 536871273.
03/20/2011 05:18:02 www_logs (536871273) updated 03/20/2011 05:17
03/20/2011 05:18:02 totalInodes 242
03/20/2011 05:18:02 Salvaged www_logs (536871273): 238 files, 243525 blocks
03/20/2011 05:18:02 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 05:22:35 dispatching child to salvage volume 536871273...
03/20/2011 05:22:35 2 nVolumesInInodeFile 56 
03/20/2011 05:22:35 CHECKING CLONED VOLUME 536871274.
03/20/2011 05:22:35 www_logs.readonly (536871274) updated 03/20/2011 05:21
03/20/2011 05:22:35 Vnode 22: length incorrect; (is 1753966 should be 0)
03/20/2011 05:22:35 SALVAGING VOLUME 536871273.
03/20/2011 05:22:35 www_logs (536871273) updated 03/20/2011 05:21
03/20/2011 05:22:35 Vnode 22: length incorrect; changed from 1753966 to 0
03/20/2011 05:22:35 Vnode 24: length incorrect; changed from 258958 to 0
03/20/2011 05:22:35 Vnode 202: length incorrect; changed from 333951 to 0
03/20/2011 05:22:35 Vnode 204: length incorrect; changed from 86190 to 0
03/20/2011 05:22:35 totalInodes 245
03/20/2011 05:22:35 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 05:24:02 dispatching child to salvage volume 536871273...
03/20/2011 05:24:02 1 nVolumesInInodeFile 28 
03/20/2011 05:24:02 SALVAGING VOLUME 536871273.
03/20/2011 05:24:02 www_logs (536871273) updated 03/20/2011 05:23
03/20/2011 05:24:02 totalInodes 242
03/20/2011 05:24:02 Salvaged www_logs (536871273): 238 files, 244191 blocks
03/20/2011 05:24:02 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 05:28:26 dispatching child to salvage volume 536871273...
03/20/2011 05:28:26 2 nVolumesInInodeFile 56 
03/20/2011 05:28:26 CHECKING CLONED VOLUME 536871274.
03/20/2011 05:28:26 www_logs.readonly (536871274) updated 03/20/2011 05:28
03/20/2011 05:28:26 Vnode 22: length incorrect; (is 1759827 should be 0)
03/20/2011 05:28:26 SALVAGING VOLUME 536871273.
03/20/2011 05:28:26 www_logs (536871273) updated 03/20/2011 05:28
03/20/2011 05:28:26 Vnode 22: length incorrect; changed from 1759827 to 0
03/20/2011 05:28:26 Vnode 24: length incorrect; changed from 259688 to 0
03/20/2011 05:28:26 Vnode 202: length incorrect; changed from 336904 to 0
03/20/2011 05:28:26 Vnode 204: length incorrect; changed from 86459 to 0
03/20/2011 05:28:26 totalInodes 246
03/20/2011 05:28:26 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 05:30:02 dispatching child to salvage volume 536871273...
03/20/2011 05:30:02 1 nVolumesInInodeFile 28 
03/20/2011 05:30:02 SALVAGING VOLUME 536871273.
03/20/2011 05:30:02 www_logs (536871273) updated 03/20/2011 05:29
03/20/2011 05:30:02 totalInodes 242
03/20/2011 05:30:02 Salvaged www_logs (536871273): 238 files, 244124 blocks
03/20/2011 05:30:02 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 05:35:35 dispatching child to salvage volume 536871273...
03/20/2011 05:35:35 2 nVolumesInInodeFile 56 
03/20/2011 05:35:35 CHECKING CLONED VOLUME 536871274.
03/20/2011 05:35:35 www_logs.readonly (536871274) updated 03/20/2011 05:33
03/20/2011 05:35:35 Vnode 22: length incorrect; (is 1764679 should be 0)
03/20/2011 05:35:35 SALVAGING VOLUME 536871273.
03/20/2011 05:35:35 www_logs (536871273) updated 03/20/2011 05:33
03/20/2011 05:35:35 Vnode 22: length incorrect; changed from 1764679 to 0
03/20/2011 05:35:35 Vnode 24: length incorrect; changed from 260802 to 0
03/20/2011 05:35:35 Vnode 202: length incorrect; changed from 342513 to 0
03/20/2011 05:35:35 Vnode 204: length incorrect; changed from 86594 to 0
03/20/2011 05:35:35 totalInodes 245
03/20/2011 05:35:36 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 05:36:12 dispatching child to salvage volume 536871273...
03/20/2011 05:36:12 1 nVolumesInInodeFile 28 
03/20/2011 05:36:12 SALVAGING VOLUME 536871273.
03/20/2011 05:36:12 www_logs (536871273) updated 03/20/2011 05:33
03/20/2011 05:36:12 totalInodes 242
03/20/2011 05:36:13 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 05:36:13 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 05:40:34 dispatching child to salvage volume 536871273...
03/20/2011 05:40:35 2 nVolumesInInodeFile 56 
03/20/2011 05:40:35 CHECKING CLONED VOLUME 536871274.
03/20/2011 05:40:35 www_logs.readonly (536871274) updated 03/20/2011 05:39
03/20/2011 05:40:35 Vnode 22: length incorrect; (is 1770454 should be 0)
03/20/2011 05:40:35 SALVAGING VOLUME 536871273.
03/20/2011 05:40:35 www_logs (536871273) updated 03/20/2011 05:39
03/20/2011 05:40:35 Vnode 22: length incorrect; changed from 1770454 to 0
03/20/2011 05:40:35 Vnode 24: length incorrect; changed from 261210 to 0
03/20/2011 05:40:35 totalInodes 245
03/20/2011 05:40:35 Salvaged www_logs (536871273): 238 files, 241815 blocks
03/20/2011 05:42:02 dispatching child to salvage volume 536871273...
03/20/2011 05:42:02 1 nVolumesInInodeFile 28 
03/20/2011 05:42:02 SALVAGING VOLUME 536871273.
03/20/2011 05:42:02 www_logs (536871273) updated 03/20/2011 05:41
03/20/2011 05:42:02 totalInodes 242
03/20/2011 05:42:02 Salvaged www_logs (536871273): 238 files, 243545 blocks
03/20/2011 05:42:02 The volume header file /vicepa/V0536871274.vol is not associated with any actual data (deleted)
03/20/2011 06:03:22 dispatching child to salvage volume 536870915...
03/20/2011 06:03:22 2 nVolumesInInodeFile 56 
03/20/2011 06:03:22 CHECKING CLONED VOLUME 536870916.
03/20/2011 06:03:22 root.cell.readonly (536870916) updated 03/20/2011 05:36
03/20/2011 06:03:22 Vnode 264: length incorrect; (is 15834650 should be 0)
03/20/2011 06:03:22 SALVAGING VOLUME 536870915.
03/20/2011 06:03:22 root.cell (536870915) updated 03/20/2011 06:03
03/20/2011 06:03:22 Vnode 264: length incorrect; changed from 15834650 to 0
03/20/2011 06:03:22 totalInodes 191
03/20/2011 06:03:22 Salvaged root.cell (536870915): 183 files, 50650 blocks
03/20/2011 06:04:01 dispatching child to salvage volume 536870915...
03/20/2011 06:04:01 1 nVolumesInInodeFile 28 
03/20/2011 06:04:01 SALVAGING VOLUME 536870915.
03/20/2011 06:04:01 root.cell (536870915) updated 03/20/2011 06:03
03/20/2011 06:04:01 totalInodes 187
03/20/2011 06:04:02 Salvaged root.cell (536870915): 183 files, 66113 blocks
03/20/2011 06:04:02 The volume header file /vicepa/V0536870916.vol is not associated with any actual data (deleted)
03/20/2011 06:08:24 dispatching child to salvage volume 536870915...
03/20/2011 06:08:24 2 nVolumesInInodeFile 56 
03/20/2011 06:08:24 CHECKING CLONED VOLUME 536870916.
03/20/2011 06:08:24 root.cell.readonly (536870916) updated 03/20/2011 06:06
03/20/2011 06:08:24 Vnode 264: length incorrect; (is 15835207 should be 0)
03/20/2011 06:08:24 SALVAGING VOLUME 536870915.
03/20/2011 06:08:24 root.cell (536870915) updated 03/20/2011 06:06
03/20/2011 06:08:24 Vnode 264: length incorrect; changed from 15835207 to 0
03/20/2011 06:08:24 totalInodes 190
03/20/2011 06:08:24 Salvaged root.cell (536870915): 183 files, 50650 blocks
03/20/2011 06:10:01 dispatching child to salvage volume 536870915...
03/20/2011 06:10:01 1 nVolumesInInodeFile 28 
03/20/2011 06:10:01 SALVAGING VOLUME 536870915.
03/20/2011 06:10:01 root.cell (536870915) updated 03/20/2011 06:06
03/20/2011 06:10:01 totalInodes 187
03/20/2011 06:10:02 Salvaged root.cell (536870915): 183 files, 50650 blocks
03/20/2011 06:10:02 The volume header file /vicepa/V0536870916.vol is not associated with any actual data (deleted)
03/20/2011 06:35:25 dispatching child to salvage volume 536870915...
03/20/2011 06:35:25 2 nVolumesInInodeFile 56 
03/20/2011 06:35:25 CHECKING CLONED VOLUME 536870916.
03/20/2011 06:35:25 root.cell.readonly (536870916) updated 03/20/2011 06:29
03/20/2011 06:35:25 Vnode 264: length incorrect; (is 15835941 should be 0)
03/20/2011 06:35:25 SALVAGING VOLUME 536870915.
03/20/2011 06:35:25 root.cell (536870915) updated 03/20/2011 06:29
03/20/2011 06:35:25 Vnode 264: length incorrect; changed from 15835941 to 0
03/20/2011 06:35:25 Vnode 266: length incorrect; changed from 2588241 to 0
03/20/2011 06:35:25 Vnode 280: length incorrect; changed from 777982 to 0
03/20/2011 06:35:25 totalInodes 190
03/20/2011 06:35:26 Salvaged root.cell (536870915): 183 files, 47364 blocks
03/20/2011 06:36:02 dispatching child to salvage volume 536870915...
03/20/2011 06:36:02 1 nVolumesInInodeFile 28 
03/20/2011 06:36:02 SALVAGING VOLUME 536870915.
03/20/2011 06:36:02 root.cell (536870915) updated 03/20/2011 06:29
03/20/2011 06:36:02 totalInodes 187
03/20/2011 06:36:02 Salvaged root.cell (536870915): 183 files, 47364 blocks
03/20/2011 06:36:02 The volume header file /vicepa/V0536870916.vol is not associated with any actual data (deleted)
03/20/2011 07:06:13 dispatching child to salvage volume 536870915...
03/20/2011 07:06:13 2 nVolumesInInodeFile 56 
03/20/2011 07:06:13 CHECKING CLONED VOLUME 536870916.
03/20/2011 07:06:13 root.cell.readonly (536870916) updated 03/20/2011 06:45
03/20/2011 07:06:13 Vnode 264: length incorrect; (is 15836334 should be 0)
03/20/2011 07:06:13 SALVAGING VOLUME 536870915.
03/20/2011 07:06:13 root.cell (536870915) updated 03/20/2011 06:45
03/20/2011 07:06:13 Vnode 264: length incorrect; changed from 15836334 to 0
03/20/2011 07:06:13 Vnode 266: length incorrect; changed from 2588400 to 0
03/20/2011 07:06:13 totalInodes 190
03/20/2011 07:06:13 Salvaged root.cell (536870915): 183 files, 47364 blocks
03/20/2011 07:08:02 dispatching child to salvage volume 536870915...
03/20/2011 07:08:02 1 nVolumesInInodeFile 28 
03/20/2011 07:08:02 SALVAGING VOLUME 536870915.
03/20/2011 07:08:02 root.cell (536870915) updated 03/20/2011 07:07
03/20/2011 07:08:02 totalInodes 187
03/20/2011 07:08:02 Salvaged root.cell (536870915): 183 files, 62829 blocks
03/20/2011 07:08:02 The volume header file /vicepa/V0536870916.vol is not associated with any actual data (deleted)
03/20/2011 07:43:22 dispatching child to salvage volume 536870915...
03/20/2011 07:43:22 2 nVolumesInInodeFile 56 
03/20/2011 07:43:22 CHECKING CLONED VOLUME 536870916.
03/20/2011 07:43:22 root.cell.readonly (536870916) updated 03/20/2011 07:37
03/20/2011 07:43:22 Vnode 264: length incorrect; (is 15836815 should be 0)
03/20/2011 07:43:22 SALVAGING VOLUME 536870915.
03/20/2011 07:43:22 root.cell (536870915) updated 03/20/2011 07:37
03/20/2011 07:43:22 Vnode 264: length incorrect; changed from 15836815 to 0
03/20/2011 07:43:22 totalInodes 190
03/20/2011 07:43:22 Salvaged root.cell (536870915): 183 files, 47364 blocks
03/20/2011 07:44:02 dispatching child to salvage volume 536870915...
03/20/2011 07:44:02 1 nVolumesInInodeFile 28 
03/20/2011 07:44:02 SALVAGING VOLUME 536870915.
03/20/2011 07:44:02 root.cell (536870915) updated 03/20/2011 07:37
03/20/2011 07:44:02 totalInodes 187
03/20/2011 07:44:02 Salvaged root.cell (536870915): 183 files, 47364 blocks
03/20/2011 07:44:02 The volume header file /vicepa/V0536870916.vol is not associated with any actual data (deleted)
03/20/2011 08:05:01 dispatching child to salvage volume 536870915...
03/20/2011 08:05:01 2 nVolumesInInodeFile 56 
03/20/2011 08:05:01 CHECKING CLONED VOLUME 536870916.
03/20/2011 08:05:01 root.cell.readonly (536870916) updated 03/20/2011 07:51
03/20/2011 08:05:01 Vnode 264: length incorrect; (is 15837439 should be 0)
03/20/2011 08:05:01 SALVAGING VOLUME 536870915.
03/20/2011 08:05:01 root.cell (536870915) updated 03/20/2011 07:51
03/20/2011 08:05:01 Vnode 264: length incorrect; changed from 15837439 to 0
03/20/2011 08:05:01 Vnode 266: length incorrect; changed from 2588558 to 0
03/20/2011 08:05:01 totalInodes 190
03/20/2011 08:05:01 Salvaged root.cell (536870915): 183 files, 47364 blocks
03/20/2011 08:06:01 dispatching child to salvage volume 536870915...
03/20/2011 08:06:01 1 nVolumesInInodeFile 28 
03/20/2011 08:06:01 SALVAGING VOLUME 536870915.
03/20/2011 08:06:01 root.cell (536870915) updated 03/20/2011 07:51
03/20/2011 08:06:01 totalInodes 187
03/20/2011 08:06:02 Salvaged root.cell (536870915): 183 files, 47364 blocks
03/20/2011 08:06:02 The volume header file /vicepa/V0536870916.vol is not associated with any actual data (deleted)
03/20/2011 08:36:59 dispatching child to salvage volume 536870915...
03/20/2011 08:36:59 2 nVolumesInInodeFile 56 
03/20/2011 08:36:59 CHECKING CLONED VOLUME 536870916.
03/20/2011 08:36:59 root.cell.readonly (536870916) updated 03/20/2011 08:33
03/20/2011 08:36:59 Vnode 264: length incorrect; (is 15837784 should be 0)
03/20/2011 08:36:59 SALVAGING VOLUME 536870915.