[OpenAFS] Re: OpenAFS-info digest, Vol 1 #2394 - 13 msgs

[Mike Polek]

 > From: Kevin <openafs@gnosys.biz>
 > To: openafs-info@openafs.org
 > Date: Wed, 18 May 2005 20:03:12 -0400
 > Subject: [OpenAFS] Storing system binaries in afs redux (/usr /opt /var)
 >
 >
 > It seems that nobody sees this as worth doing, but from the
 > documentation, it looks as though this concept was factored into the
 > design of afs.  Would anyone care to comment on:
 >
 > a) whether or not they are storing any local machine system binaries in
 > afs-space (obviously, stuff not needed immediately at boot-time
 > like /usr and /opt, maybe /var);

I consider this worth doing. In fact, I have just about everything
in AFS now and use a tmpfs filesystem for the root filesystem.

Filesystem           1K-blocks      Used Available Use% Mounted on
tmpfs                    51200      2796     48404   6% /
AFS                    9000000         0   9000000   0% /afs
AFS                    9000000         0   9000000   0% /bin
AFS                    9000000         0   9000000   0% /etc
AFS                    9000000         0   9000000   0% /lib
AFS                    9000000         0   9000000   0% /opt
AFS                    9000000         0   9000000   0% /sbin
AFS                    9000000         0   9000000   0% /usr
AFS                    9000000         0   9000000   0% /var/lib/rpm
AFS                    9000000         0   9000000   0% /var/lib/scrollkeeper
AFS                    9000000         0   9000000   0% /var/www
none                    192436       216    192220   1% /dev

This is my test machine, which has about 384Mb of main memory.
I use the initial ramdisk to get afs up and running, and
mount --bind the appropriate partitions from there. The tricky part
is getting the system image into afs, as others have said.
My solution is to use cpio, which doesn't preserve hard links, and
pretty much doesn't complain.

Essentially, I build a base OS image from rpms using rpm --root,
then I create a bunch of volumes for the pieces, and cpio the
files in. For my purposes, the suid stuff isn't important.
We use this only for diskless image processors.

/var is the one partition I can think of with a lot of data,
but that normally must be r/w because of logs, etc. The three
heavy-hitter subdirs that I keep in afs are lib/rpm, lib/scrollkeeper,
and www. And I make sure to rotate my logs out to a hard drive
nightly to keep the / partition from filling.

 > b) whether or not they think doing so is a good idea, bad idea, or
 > neutral;

The concept is great. There is a fair amount of work involved in
setting it up. Once it's set up, setting up a new machine is as
simple as setting up the DHCP so it can PXE boot...  Maintaining
the OS image isn't too bad.

 > c) if someone thinks it's a good/bad idea, reasons for this perspective;

good:
1) All partitions are in memory or replicated volumes. There is no
    single-point-of-failure that can come with having networked r/w
    partitions (like using NFS as a root partition).
2) If you have lots of nearly identical systems, the time spent in
    building the first one is divided by the number of system you
    don't have to set up individually.
3) Upgrading can be as simple as rebooting when the next O/S is
    ready to go into production.

bad:
As kernels and OS's change, sometimes the fundamentals change
(like how the system gets booted). This can cause headaches.
Simple things like the way they switched from an ext2 ramdisk to
a cpio archive, using mount --move to mount the root filesystem,
leaving the ramdisk in memory, changing the behavior of modprobe,
etc. All mean rewrites if you're modifying boot scripts.

 > d) thoughts related to permission problems mentioned by Jim Rees that
 > arise from doing this, and the best way (or some ways) to resolve them;

If I want to install an RPM (after appropriate vos backups, of course),
I create an area that looks like the running system with volumes mounted
appropriately, copy the rpms into the tmp area, and then chroot.
Then I just install the rpms. I haven't run into a situation where
I wanted to install something not as root.

For software that is not part of the system, but is needed by the
clients, I just put is some place in afs and symlink from /opt,
so more or less everything looks like it's in /opt. You could
do /usr/local as well, or whatever works for you, as long as it's
self contained. I like to put entire packages into a single volume.
It makes upgrading easier.

 >
 > I've been experimenting with the concept in several Gentoo machines that
 > are all on the same afs cell and in fiddling with an OldWorld Apple
 > machine with very limited local diskspace and running Debian, I'd like
 > to use the concept in order to install a bunch of software that I
 > otherwise would not be able to install due to space constraints.
 >
 > TIA for any comments and apologies for the redux.
 >
 > -Kevin
 >

FYI, My latest build is Fedora Core 3. Everything seems to work ok
on my test machine, but when I moved things over to a dual processor
system, I started having lockup problems. There does seem to be a lot
of "just one more hurdle" kind of thing when doing this sort of setup.
I hope to have all the kinks ironed out soon, and possibly put together
a HOWTO if there is interest.

Mike Polek
Pictage, Inc.