[OpenAFS] AFS design question: implementing AFS over a highly-distributed, low-bandwidth network

[Chaz Chandler]

Replies in-line:

> general advice:
>=20
> - make sure the network connectivity between your three AFS =22database
> servers=22 is always up...they depend on the network to communicate with
> each other, and if they are always up and always reliable, it will
> enhance the perceived performance of afs

Undertood.  The links are pretty stable.

> - if most of the users do mostly reading, and mostly smallish files, you
> might get by using volume replication at each site, leaving the RW
> volume wherever it is, and relying on some tricks to keep the user
> confused during writes...i think, in general, the idea of moving RW
> volumes all the time is a bad one...it will put big loads on your
> fileservers if you have lots of users

For the most part, yes.  We try to keep the large stuff (databases) =
separately and have the db apps take care of synchronization in their own =
way.

I thought as much, but I'm getting very poor performance even for reading. =
 It seems that the clients don't do a good job of prioritizing the local =
fileserver first.  Some posts mention lower-numbered IP addresses getting =
priority, but I'm not sure how that would work for servers on different =
subnets.  Shouldn't the choice have something to do with lower latency?

> - check the top of your afs tree and make sure root.afs, root.cell and
> probably most other volumes at the top of the tree contain only
> mountpoints, and are all replicated, and don't change them much...in
> other words, you want stability at the top of the tree, to reduce the
> changes that clients have to pay attention to

These are all mounted R/O, right?  Should mostly-static volumes be mounted =
R/O and then only access the R/W vol through /afs/.domain/ for the =
occasional updates?  What about for the software repository scenario below?

The way I understand the volume access is that even if a volume is mounted =
R/W, its R/O vol is accessed until a write operation, in which case all =
further access (including reads) are from the R/W vol.  If so, then a user =
should ideally be able to use the local server for most reads.

We haven't made any changes to the default cache options, so maybe this =
would be a good next step?

> - do your vos operations, backups, etc. in a coordinated way so you can
> predict when they are happening, and don't end up hosing your servers by
> doing multiple simulataneous operations on the same volumes

How do I know when to release?  Do people do this automagically with =
scripting?  With what triggers / schedules?

> how many users? what kinds of files and what kinds of usage? how much
> sharing? these can make a difference in how you set things up, too. (i
> worked on a cell once with clients in sweden, south america, and on the
> european continent...one server at each site, whenever the south
> american site became master db server, followed by transatlantic link
> going down...the european users weren't happy....basically, as long as
> you have good network connectivity between client systems and server
> systems, there is no need for physical proximity...)

We've got about 5 users, so a pretty small setup.  They're mostly very =
non-technical, and expect it to Just Work (tm).

The files regular users are concerned with are mostly documents, images, =
and presentations.  Their heavy sharing is usually confined to a small set =
of small files (usually less that 1MB each); sets change as projects =
change (weekly/monthly).  Still, it takes about 1 minute to transfer 1MB =
across the VPN -- not insignificant.

-Chaz

> Chaz Chandler wrote:
>> Hello all=21
>>=20
>> I am attempting to implement OpenAFS across a VPN with limited bandwidth
>> between sites but relatively mobile users who expect to have their data
>> available (and writable) at whichever site they are currently located.
>>=20
>> The issue I am running up against is how to organize the AFS volume
>> structure so that things like user home dirs, collaboration/group dirs,
>> and writable system items (like a Windows profile, for instance) are
>> optimally maintained in AFS.
>>=20
>> The set-up is:
>> 1) Each site has an OpenAFS server (each with vl, pt, bu, vol,
>> fileservers & salvager).  Currently, v1.4.8 on Linux 2.6.
>> 2) Clients computers are a mix of Windows XP, OpenBSD, and Linux.  1.5
>> clients for windows, 1.4 clients for Linux, and native openbsd clients.
>> 3) All sites are connected in a full mesh VPN (max of about 30KB/s for
>> each link)
>> 4) There's about 600GB of data at the moment.  Although most of it
>> doesn't need to be writable most of the time, things that are frequently
>> written are not currently segregated from static or infrequently-written
>> files/dirs.  Perhaps only a few gigs change on a weekly basis.
>> 5) Users move from site to site, but once there usually spend several
>> weeks.  However, two sites are physically very close and users move
>> between them more frequently (sometimes daily) although the link
>> bandwidth is the same as the others.
>> 6) We have a pretty standard AFS volume layout: separate volumes for
>> each user, a few large volumes with relatively static content, a few
>> volumes for groups to share.
>> 7) Currently, volume releases are done manually.
>> 8) When a user changes locations for a long stretch, we move their R/W
>> user volume to the new location (electronically, not physically), a
>> process which is labor- and time-intensive and usually has at least one
>> snafu along the way.
>> 9) We have been unable to come up with a working implementation of
>> roaming windows profiles on AFS.
>>=20
>> I'm seeking recommendations on:
>> 1) How others have set up a regular release schedule to keep a large
>> amount of data synced over a slow network (custom scripts, I assume, but
>> is there a repository of these things and what are the general mechanics
>> and best practices here?)
>> 2) What sort of volume layout would one recommend, and how should
>> frequently-updated data be stored?  Take, for instance, three examples:
>> - A software repository: large volume with relatively static contents,
>> occasionally has large additions or subtractions when a new piece of
>> software is added or an old one removed.  Ideally, these updates should
>> be able to be accomplished from any location.  Users don't need to write
>> to it, but may need to read from it frequently at LAN speeds.
>> - A collaboration dir: several users read and write a small amount (10s
>> of MBs) on a daily basis from different locations simultaneously, but
>> they expect LAN-type performance.
>> - A user dir: large amounts of data updated from a single location, but
>> user may move to any other site at any time, potentially with up to a
>> day of transit time in which a volume could be moved to the destination
>> site.
>> 3) Any concrete recommendations on how to properly implement windows
>> integration with AFS (especially folder redirection and roaming profiles
>> on AFS).  Yes, I've read the '04 and '05 best practices, however they
>> are now quite old and did not work for me.
>>=20
>> I've been lurking on this list for a while now and have come to the
>> conclusion that while there are a few very knowledgeable and experienced
>> folks in the AFS community, there are not any good, current, and
>> comprehensive AFS information repositories out there.  The list archives
>> are the best option, but I find them almost impossible to use unless I
>> know the exact phrase I'm looking for.  Is there something I'm missing?
>>=20
>> Cheers,
>> -Chaz
>> _______________________________________________
>> OpenAFS-info mailing list
>> OpenAFS-info=40openafs.org
>> https://lists.openafs.org/mailman/listinfo/openafs-info
>>=20
>=20
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info=40openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info