[OpenAFS] DAFS dasalvager: cannot be running from cron
Brunckhorst, Ralf
ralf.brunckhorst@hp.com
Wed, 10 Jul 2013 12:38:19 +0000
--_000_E3A882F104F6304D89C62FB119B729800FCCEA2FG5W2743americas_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Thanks for this detailed explanation.
So I have some follow-up question.
Is salvaging of volumes needed afterwards in the following scenarios?
* vos dump -id UNTAG-volume.readonly -time 0 -server SERVER -partition PAR=
T | vos restore -server SERVER -partition PART -name TAG-volume -overwrite =
incremental
* vos dump -id UNTAG-volume.readonly -time 0 -server SERVER -partition PA=
RT | dump-scan_mp | vos restore -server SERVER -partition PART -name TAG-vo=
lume -overwrite full
We are using this 'strange' workflow to have a workaround for the limit in =
openafs of having not more than 11 RO-replicas.
The dump-scan_mp in the pipe is used to change Volumes that have mounpoints=
inside and change those mounpoints so that the point to TAG-volumes.
/Ralf
Am 08.07.2013 um 18:38 schrieb Andrew Deason <adeason@sinenomine.net<mailto=
:adeason@sinenomine.net>>:
On Mon, 8 Jul 2013 08:14:29 +0000
"Brunckhorst, Ralf" <ralf.brunckhorst@hp.com<mailto:ralf.brunckhorst@hp.com=
>> wrote:
Is there any chance to get this also running via cron?
Short answer: dasalvager is apparently unsafe for single-volume
salvages; avoid it for now if you can. Running 'salvageserver -client'
is another way to manually salvage a single volume, which does not have
this problem.
Longer answer:
This is a bug in dasalvager where it is not initializing the structure
properly for locking volumes on disk. So, it thinks it has fd 0 already
open for locking, and tries to use that fd without opening the proper
file. When you run under a terminal, it uses whatever happens to be fd 0
(this is obviously not safe/correct). When you run under cron (or 'at'
or probably a number of other things), fd 0 happens to be closed, so we
fail. It is very fortunate that it does fail, because otherwise I don't
know when we would have discovered this.
Because of all of that, even when dasalvager seems to be running along
fine, it is not accessing volumes in a safe manner (this is a bug; the
same bug). So, in certain edge cases, it would be possible for
dasalvager to cause corruption of volumes. While I haven't completely
thought through the scenarios, I believe this would only cause whole
volumes to become unusable due to volume metadata problems; it wouldn't
corrupt data inside volumes or anything like that. If you don't care, or
if this server is relatively inactive or is otherwise low-risk for some
reason, you could work around this by forcing fd 0 to be open when you
run dasalvager from cron. Obviously, I don't really recommend that, but
it should be possible.
What would probably be better is if you don't use dasalvager for
single-volume salvages at all until we can get this fixed. If you must
manually cause a single-volume salvage, you can run 'salvageserver
-client' instead of dasalvager, which uses the same code paths as
demand-salvages.
>From a developer perspective:
This is due to the DAFS_FS / DAFS_UTIL mess; all of the DAFS stuff in
partition.c should be for _UTIL or _FS instead of just _FS. I started
going through fixing them, but there are so many cases that seem to need
fixing (e.g. all vutil.c references), and it's becoming clear that the
current scheme is becoming increasingly... ridiculously cumbersome and
error-prone. I can think of a few possible ways of improving this:
- Make DAFS_FS imply DAFS_UTIL. We still have to go manually fix all of
the instances to see if they really are _FS or should be _UTIL, but
at least it's less verbose.
- Remove DAFS_UTIL, make all DAFS utilities use DAFS_FS, and fix
existing code to handle the non-pthread DAFS_FS case.
- Just use DAFS_FS everywhere and make all DAFS utilities pthreaded,
reducing the number of different codepaths. Is there any reason we
were avoiding this? It's not like we need LWP DAFS, and any further
granularity of "what type of program is running" should be handled at
runtime by programType anyway.
I'll probably try the latter and see if it's more work than I thought or
if I forgot about some big blocker issue for that. Just mentioning
options here if anyone has opinions on it.
--
Andrew Deason
adeason@sinenomine.net<mailto:adeason@sinenomine.net>
_______________________________________________
OpenAFS-info mailing list
OpenAFS-info@openafs.org<mailto:OpenAFS-info@openafs.org>
https://lists.openafs.org/mailman/listinfo/openafs-info
--_000_E3A882F104F6304D89C62FB119B729800FCCEA2FG5W2743americas_
Content-Type: text/html; charset="us-ascii"
Content-ID: <E8FBF73A78FD7F409B7E420BB52D6A95@Compaq.com>
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)">
<style><!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Times New Roman","serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
<div class=3D"WordSection1">
<p class=3D"MsoNormal">Thanks for this detailed explanation.<o:p></o:p></p>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">So I have some follow-up question.<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal">Is salvaging of volumes needed afterwards in the fol=
lowing scenarios?<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal"> * vos dump –id UNTAG-volume.readonly =
211;time 0 –server SERVER –partition PART | vos restore –=
server SERVER –partition PART –name TAG-volume –overwrite=
incremental<o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal"> * vos dump –id UNTAG-volume.readonly &nb=
sp;–time 0 –server SERVER –partition PART | dump-scan_mp =
| vos restore –server SERVER –partition PART –name TAG-vo=
lume –overwrite full<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">We are using this ‘strange’ workflow to =
have a workaround for the limit in openafs of having not more than 11 RO-re=
plicas.<o:p></o:p></p>
<p class=3D"MsoNormal">The dump-scan_mp in the pipe is used to change Volum=
es that have mounpoints inside and change those mounpoints so that the poin=
t to TAG-volumes.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal">/Ralf<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class=3D"MsoNormal">Am 08.07.2013 um 18:38 schrieb Andrew Deason <<a =
href=3D"mailto:adeason@sinenomine.net">adeason@sinenomine.net</a>>:<o:p>=
</o:p></p>
</div>
<p class=3D"MsoNormal"><br>
<br>
<o:p></o:p></p>
<p class=3D"MsoNormal">On Mon, 8 Jul 2013 08:14:29 +0000<br>
"Brunckhorst, Ralf" <<a href=3D"mailto:ralf.brunckhorst@hp.com=
">ralf.brunckhorst@hp.com</a>> wrote:<br>
<br>
<br>
<o:p></o:p></p>
<p class=3D"MsoNormal">Is there any chance to get this also running via cro=
n?<o:p></o:p></p>
<p class=3D"MsoNormal"><br>
Short answer: dasalvager is apparently unsafe for single-volume<br>
salvages; avoid it for now if you can. Running 'salvageserver -client'<br>
is another way to manually salvage a single volume, which does not have<br>
this problem.<br>
<br>
Longer answer:<br>
<br>
This is a bug in dasalvager where it is not initializing the structure<br>
properly for locking volumes on disk. So, it thinks it has fd 0 already<br>
open for locking, and tries to use that fd without opening the proper<br>
file. When you run under a terminal, it uses whatever happens to be fd 0<br=
>
(this is obviously not safe/correct). When you run under cron (or 'at'<br>
or probably a number of other things), fd 0 happens to be closed, so we<br>
fail. It is very fortunate that it does fail, because otherwise I don't<br>
know when we would have discovered this.<br>
<br>
Because of all of that, even when dasalvager seems to be running along<br>
fine, it is not accessing volumes in a safe manner (this is a bug; the<br>
same bug). So, in certain edge cases, it would be possible for<br>
dasalvager to cause corruption of volumes. While I haven't completely<br>
thought through the scenarios, I believe this would only cause whole<br>
volumes to become unusable due to volume metadata problems; it wouldn't<br>
corrupt data inside volumes or anything like that. If you don't care, or<br=
>
if this server is relatively inactive or is otherwise low-risk for some<br>
reason, you could work around this by forcing fd 0 to be open when you<br>
run dasalvager from cron. Obviously, I don't really recommend that, but<br>
it should be possible.<br>
<br>
What would probably be better is if you don't use dasalvager for<br>
single-volume salvages at all until we can get this fixed. If you must<br>
manually cause a single-volume salvage, you can run 'salvageserver<br>
-client' instead of dasalvager, which uses the same code paths as<br>
demand-salvages.<br>
<br>
>From a developer perspective:<br>
<br>
This is due to the DAFS_FS / DAFS_UTIL mess; all of the DAFS stuff in<br>
partition.c should be for _UTIL or _FS instead of just _FS. I started<br>
going through fixing them, but there are so many cases that seem to need<br=
>
fixing (e.g. all vutil.c references), and it's becoming clear that the<br>
current scheme is becoming increasingly... ridiculously cumbersome and<br>
error-prone. I can think of a few possible ways of improving this:<br>
<br>
- Make DAFS_FS imply DAFS_UTIL. We still have to go manually fix all of<br>
the instances to see if they really are _FS or should be _UTIL,=
but<br>
at least it's less verbose.<br>
<br>
- Remove DAFS_UTIL, make all DAFS utilities use DAFS_FS, and fix<br>
existing code to handle the non-pthread DAFS_FS case.<br>
<br>
- Just use DAFS_FS everywhere and make all DAFS utilities pthreaded,<br>
reducing the number of different codepaths. Is there any reason=
we<br>
were avoiding this? It's not like we need LWP DAFS, and any fur=
ther<br>
granularity of "what type of program is running" shou=
ld be handled at<br>
runtime by programType anyway.<br>
<br>
I'll probably try the latter and see if it's more work than I thought or<br=
>
if I forgot about some big blocker issue for that. Just mentioning<br>
options here if anyone has opinions on it.<br>
<br>
-- <br>
Andrew Deason<br>
<a href=3D"mailto:adeason@sinenomine.net">adeason@sinenomine.net</a><br>
<br>
_______________________________________________<br>
OpenAFS-info mailing list<br>
<a href=3D"mailto:OpenAFS-info@openafs.org">OpenAFS-info@openafs.org</a><br=
>
<a href=3D"https://lists.openafs.org/mailman/listinfo/openafs-info">https:/=
/lists.openafs.org/mailman/listinfo/openafs-info</a><o:p></o:p></p>
</div>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>
--_000_E3A882F104F6304D89C62FB119B729800FCCEA2FG5W2743americas_--