[OpenAFS] Re: busy .backup volumes

[Jason Edgecombe]

Try " vos unlock 1953047510"

On 04/19/2013 08:51 PM, Kristen J. Webb wrote:
> I was able to find a newer version of vos on a new system:
>
> # vos --version
> openafs 1.6.2.1
>
> I reported bos --version (1.4.14), since I figured it would
> represent the version of the volserver ;)
>
> So I tried:
>
> # vos  endtrans fileserver 22 -localauth
> # vos  endtrans fileserver 58191 -localauth
>
> Interestingly, the messages have stopped posting
> to the Volserver log, but I still get:
>
> # vos status fileserver
> Total transactions: 2
> --------------------------------------
> transaction: 58191  created: Fri Apr 19 07:27:14 2013
> lastActiveTime: Fri Apr 19 20:28:48 2013
> attachFlags:  busy
> transactionFlags: delete
> volume: 1953047510  partition: /vicepc  procedure: Dump -> basically a 
> failed vos dump of a .backup volume
> packetRead: 1  lastReceiveTime: Fri Apr 19 10:49:04 2013
> packetSend: 1  lastSendTime: Fri Apr 19 10:56:04 2013
> --------------------------------------
>
> --------------------------------------
> transaction: 22  created: Sun Apr 14 04:43:55 2013
> lastActiveTime: Fri Apr 19 20:28:48 2013
> attachFlags:  busy
> transactionFlags: delete
> volume: 1953054615  partition: /vicepc  procedure: Dump -> basically a 
> failed vos dump of a .backup volume
> packetRead: 1  lastReceiveTime: Tue Apr 16 09:52:25 2013
> packetSend: 1  lastSendTime: Tue Apr 16 09:59:25 2013
> --------------------------------------
>
> I can say for sure that server that issued the vos dump has been 
> rebooted since the transaction started.  The other thing I am 
> observing is that repeated vos status on the fileserver shows the 
> lastActiveTime as current (increasing).
>
> Also, still getting:
>
> # vos ex 1953047510
> **** Volume 1953047510 is busy ****
>
> # vos ex 1953054615
> **** Volume 1953054615 is busy ****
>
> Since they are .backup volumes, this is not impacting
> the users live data, yet, But backups are note completing....
>
> I'm going to check later to see if some other timeout takes effect.  
> I'm curious if there is something more to try in this configuration.  
> Still trying to avoid a bos restart for now.  Oh, wait, after about 20 
> minutes, the Volserver
> log started reporting again, very interesting:
>
> Fri Apr 19 20:25:06 2013 trans 58191 on volume 1953047510 is older 
> than 46650 seconds
> Fri Apr 19 20:25:06 2013 trans 22 on volume 1953054615 is older than 
> 488460 seconds
> Fri Apr 19 20:25:36 2013 trans 22 on volume 1953054615 is older than 
> 488490 seconds
> Fri Apr 19 20:26:06 2013 trans 22 on volume 1953054615 is older than 
> 488520 seconds
> Fri Apr 19 20:48:06 2013 trans 58191 on volume 1953047510 is older 
> than 300 seconds
> Fri Apr 19 20:48:06 2013 trans 22 on volume 1953054615 is older than 
> 300 seconds
> Fri Apr 19 20:48:36 2013 trans 58191 on volume 1953047510 is older 
> than 330 seconds
>
>
> Thanks for the tips Andrew!
>
> Kris
> On 4/19/13 4:11 PM, Andrew Deason wrote:
>> On Fri, 19 Apr 2013 13:06:20 -0600
>> "Kristen J. Webb" <kwebb@teradactyl.com> wrote:
>>
>>> # bos --version
>>> openafs 1.4.14
>>
>> Er, you mean 'vos'? But I assume they're all the same :)
>>
>>> Fri Apr 19 15:02:36 2013 trans 22 on volume 1953054615 is older than
>>> 469110 seconds
>>>
>>> We've been just restarting the server.  Is there a better way to kill
>>> these hung transactions without a sledge hammer?
>>
>> You can try <http://docs.openafs.org/Reference/1/vos_endtrans.html>. It
>> doesn't exist in 1.4, but using the 'vos' from 1.6+ would work fine.
>>
>> However, I thought that specific message indicates that there's still an
>> RPC holding a reference to that transaction (otherwise it would say it's
>> idle). I might not be remembering the details correctly at the moment,
>> but if so, you can't do anything to kill the transaction until you find
>> the executing RPC that's using it and get it to stop. So, you'd need to
>> look at 'vos status', rxdebug, or maybe a stack/core dump to see who's
>> doing that.
>>
>