[OpenAFS] Re: Advice on a use case
Ted Creedon
tcreedon@easystreet.net
Tue, 6 Nov 2012 10:49:09 -0800
--20cf307d0626ad78ba04cdd80e9f
Content-Type: text/plain; charset=ISO-8859-1
How about an update R/W or R/O dropdown for windows
tedc
On Tue, Nov 6, 2012 at 8:49 AM, Andrew Deason <adeason@sinenomine.net>wrote:
> On Tue, 6 Nov 2012 00:06:53 -0800
> Timothy Balcer <timothy@telmate.com> wrote:
>
> > I have a need to think about replicating large volumes (multigigabyte)
> > of large number (many terabytes of data total), to at least two other
> > servers besides the read write volume, and to perform these releases
> > relatively frequently (much more than once a day, preferably)
>
> How much more frequently? Hourly? Some people do 4 times hourly (and
> maybe more) successfully.
>
> > Also, these other two (or more) read-only volumes for each read write
> > volume will be remote volumes, transiting across relatively fat, but
> > less than gigabit, pipes (100+ megabits)
>
> Latency may matter more than bandwidth; do you know what it is?
>
> > For the moment what I have decided to experiment with is a simple
> > system. My initial idea is to work the afs read-only volume tree into
> > an AUFS union, with a local read write partition in the mix. This way,
> > writes will be local, but I can periodically "flush" writes to the AFS
> > tree, double check they have been written and released, and then
> > remove them from the local partition.. this should maintain integrity
> > and high availability for the up-to-the-moment recordings, given I
> > RAID the local volume. Obviously, this still introduces a single point
> > of failure... so I'd like to flush as frequently as possible.
> > Incidentally, it seems you can NFS export such a union system fairly
> > simply.
>
> I'm not sure I understand the purpose of this; are you trying to write
> new data from all of the 'remote' locations, and you need those writes
> to 'finish' quickly?
>
> > But, I feel as if I am missing something... it has become clear that
> > releasing is a pretty intensive operation, and if we're talking about
> > multiple gigabytes per release, I can imagine it being extremely
> > difficult. Is there a schema that i can use with OpenAFS that will
> > help alleviate this problem? Or perhaps another approach I am missing
> > that may solve it better?
>
> Eh, some people do that; it just reduces the benefit of the client-side
> caching. Every time you release a volume, the server tells clients that
> for all data in that volume, the client needs to check with the server
> to see if the cached data is different from what's actually in the
> volume. But that may not matter so much, especially for a small number
> of large files.
>
> To improve things, you can maybe try to reduce the number of volumes
> that are changing. That is, if you are adding new data in batches, I
> don't know if it's feasible for you to add that 'batch' of data by
> creating a new volume instead of writing to existing volumes.
>
>
> And, of course, the release process may not be fast enough to actually
> do releases as quickly as you want. There are maybe some ways to ship
> around volume dumps yourself to get around that, and some pending
> improvements to the volserver that would help, but I would only think
> about that after you try the releases yourself.
>
> --
> Andrew Deason
> adeason@sinenomine.net
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>
--20cf307d0626ad78ba04cdd80e9f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
How about an update R/W or R/O dropdown for windows<br><br>tedc<br><br><div=
class=3D"gmail_quote">On Tue, Nov 6, 2012 at 8:49 AM, Andrew Deason <span =
dir=3D"ltr"><<a href=3D"mailto:adeason@sinenomine.net" target=3D"_blank"=
>adeason@sinenomine.net</a>></span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">On Tue, 6 Nov 2012 00:06:53 -0800<br>
Timothy Balcer <<a href=3D"mailto:timothy@telmate.com">timothy@telmate.c=
om</a>> wrote:<br>
<br>
> I have a need to think about replicating large volumes (multigigabyte)=
<br>
> of large number (many terabytes of data total), to at least two other<=
br>
> servers besides the read write volume, and to perform these releases<b=
r>
> relatively frequently (much more than once a day, preferably)<br>
<br>
How much more frequently? Hourly? Some people do 4 times hourly (and<br>
maybe more) successfully.<br>
<br>
> Also, these other two (or more) read-only volumes for each read write<=
br>
> volume will be remote volumes, transiting across relatively fat, but<b=
r>
> less than gigabit, pipes (100+ megabits)<br>
<br>
Latency may matter more than bandwidth; do you know what it is?<br>
<br>
> For the moment what I have decided to experiment with is a simple<br>
> system. =A0My initial idea is to work the afs read-only volume tree in=
to<br>
> an AUFS union, with a local read write partition in the mix. This way,=
<br>
> writes will be local, but I can periodically "flush" writes =
to the AFS<br>
> tree, double check they have been written and released, and then<br>
> remove them from the local partition.. this should maintain integrity<=
br>
> and high availability for the up-to-the-moment recordings, given I<br>
> RAID the local volume. Obviously, this still introduces a single point=
<br>
> of failure... so I'd like to flush as frequently as possible.<br>
> Incidentally, it seems you can NFS export such a union system fairly<b=
r>
> simply.<br>
<br>
I'm not sure I understand the purpose of this; are you trying to write<=
br>
new data from all of the 'remote' locations, and you need those wri=
tes<br>
to 'finish' quickly?<br>
<br>
> But, I feel as if I am missing something... it has become clear that<b=
r>
> releasing is a pretty intensive operation, and if we're talking ab=
out<br>
> multiple gigabytes per release, I can imagine it being extremely<br>
> difficult. =A0Is there a schema that i can use with OpenAFS that will<=
br>
> help alleviate this problem? Or perhaps another approach I am missing<=
br>
> that may solve it better?<br>
<br>
Eh, some people do that; it just reduces the benefit of the client-side<br>
caching. Every time you release a volume, the server tells clients that<br>
for all data in that volume, the client needs to check with the server<br>
to see if the cached data is different from what's actually in the<br>
volume. But that may not matter so much, especially for a small number<br>
of large files.<br>
<br>
To improve things, you can maybe try to reduce the number of volumes<br>
that are changing. That is, if you are adding new data in batches, I<br>
don't know if it's feasible for you to add that 'batch' of =
data by<br>
creating a new volume instead of writing to existing volumes.<br>
<br>
<br>
And, of course, the release process may not be fast enough to actually<br>
do releases as quickly as you want. There are maybe some ways to ship<br>
around volume dumps yourself to get around that, and some pending<br>
improvements to the volserver that would help, but I would only think<br>
about that after you try the releases yourself.<br>
<span class=3D"HOEnZb"><font color=3D"#888888"><br>
--<br>
Andrew Deason<br>
<a href=3D"mailto:adeason@sinenomine.net">adeason@sinenomine.net</a><br>
<br>
_______________________________________________<br>
OpenAFS-info mailing list<br>
<a href=3D"mailto:OpenAFS-info@openafs.org">OpenAFS-info@openafs.org</a><br=
>
<a href=3D"https://lists.openafs.org/mailman/listinfo/openafs-info" target=
=3D"_blank">https://lists.openafs.org/mailman/listinfo/openafs-info</a><br>
</font></span></blockquote></div><br>
--20cf307d0626ad78ba04cdd80e9f--