[OpenAFS] Full disk woes

[Steve Devine]

Hartmut Reuter wrote:
>
> I tried a
>
> /afs/ipp/backups: vos listvldb 1938590434 -cell msu.edu
> vsu_ClientInit: Could not get afs tokens, running unauthenticated.
>
> svc.ml.mdsolids.31
>     RWrite: 1938590433    ROnly: 1938590434    RClone: 1938590434
>     number of sites -> 3
>        server afsfs7.cl.msu.edu partition /vicepa RW Site
>        server afsfs9.cl.msu.edu partition /vicepa RO Site  -- Old release
>        server afsfs7.cl.msu.edu partition /vicepa RO Site  -- New release
> /afs/ipp/backups:
>
> and found out it's your machine afsfs9.cl.msu.edu which does the
> trouble. Then I did a "vos status " to this machine which did not
> respond.
>
> rxdebug "afsfs9.cl.msu.edu 7005" shows a lot of connections in state
> precall with source ports != 7005. That means you have a lot vos
> commands running anywhere. Those you should stop first! Then perhaps
> restart your fileserver to get rid of the old transactions and then
> hopefully everthing is OK again.
>
> Hartmut
>
> Steve Devine wrote:
>> Hartmut Reuter wrote:
>>
>>> Steve Devine wrote:
>>>
>>>> Hartmut Reuter wrote:
>>>>
>>>>
>>>>> Steve Devine wrote:
>>>>>
>>>>>
>>>>>> I committed the cardinal sin of letting a server partition fill up.
>>>>>> I have tried vos remove and vos zap .. I can't get rid of any
>>>>>> vols.Volume management fails on this machine.
>>>>>> Its the old style (non namei) fileserver. It doesn't seem like I can
>>>>>> just "rm the V#####.vol" can I?
>>>>>> Any help?
>>>>>>
>>>>>
>>>>> To remove the small V#####.vol files doesn't help, they are really
>>>>> only 76 bytes long.
>>>>>
>>>>> What happens if you do a "vos remove" or a "vos zap"?
>>>>
>>>> both commands fail. Even when I use force.
>>>
>>>
>>> What says the VolserLog?
>>>
>>>
>>>
>>>>> Go the volumes away and the free space seems as low as before?
>>>>>
>>>>> This can happen, if you only removed readonly and backup volumes
>>>>> which
>>>>> typically can free only the space used by their metadata while the
>>>>> space used by their files and directories is shared between them and
>>>>> the RW volume. But, of course, you don't want to remove your
>>>>> RW-volumes.
>>>>> May be, if you have removed all RO- and BK- volumes you have enough
>>>>> free space for the temporary volume being created when you try to
>>>>> move
>>>>> your smallest RW-volume to another partition/server.
>>>>>
>>>>> There is also a "-live" option for the vos move command which should
>>>>> doe the move without creating a clone. I suppose it has been written
>>>>> for such cases.
>>>>>
>>>>> Good luck,
>>>>> Hartmut
>>>>> -----------------------------------------------------------------
>>>>> Hartmut Reuter                           e-mail reuter@rzg.mpg.de
>>>>>                      phone +49-89-3299-1328
>>>>> RZG (Rechenzentrum Garching)               fax   +49-89-3299-1301
>>>>> Computing Center of the Max-Planck-Gesellschaft (MPG) and the
>>>>> Institut fuer Plasmaphysik (IPP)
>>>>> -----------------------------------------------------------------
>>>>> _______________________________________________
>>>>> OpenAFS-info mailing list
>>>>> OpenAFS-info@openafs.org
>>>>> https://lists.openafs.org/mailman/listinfo/openafs-info
>>>>
>>>>
>>>>
>>>
>> Lot of lines like this ..
>> Fri Jul  6 10:05:18 2007 trans 3811071 on volume 1938590434 is older
>> than 29730 seconds
>> Fri Jul  6 10:05:48 2007 trans 3811072 on volume 1937192577 is older
>> than 28530 seconds
>> Fri Jul  6 10:05:48 2007 trans 3811071 on volume 1938590434 is older
>> than 29760 seconds
>> Fri Jul  6 10:06:18 2007 trans 3811072 on volume 1937192577 is older
>> than 28560 seconds
>> Fri Jul  6 10:06:18 2007 trans 3811071 on volume 1938590434 is older
>> than 29790
>>
>
>
Ok in the end we killed all vos commands to that server and restarted
the Bosserver- volserver.
Then we were able to vos remove the RO vols I stupidly put on there in
the first place.
Ran salvager on some horked vols and so far so good.
Thanks to all that helped.
/sd


-- 
Steve Devine
Network Storage and Printing
Academic Computing & Network Services
Michigan State University

506 Computer Center
East Lansing, MI 48824-1042
1-517-432-7327

Baseball is ninety percent mental; the other half is physical.
- Yogi Berra