[OpenAFS] Re: Storing system binaries in the /afs tree

Dr A V Le Blanc Dr A V Le Blanc <LeBlanc@mcc.ac.uk>
Thu, 19 May 2005 08:46:06 +0100


On Tue 17 May 2005 at 14:37:53 -0400, Kevin <openafs@gnosys.biz> wrote:
> Any recommendations on storing portions of a Debian system (binaries,
> configs, whatever) in afs-space?  At this point, if I can come up with a
> reliable plan for storing system binaries in afs-space, I'd like to
> fiddle with setting up a more complete Debian system; maybe even try X
> on this old thing.  Perhaps I could put all of /usr in the afs-tree?
> Maybe /opt too?  Putting /var there seems like it might be a bad idea,
> but maybe some directories in /var?  /var seems to get quite large on
> Debian.

I've been running a system here for a year which has nearly everything
in afs space.  I use this to run Linux on public cluster machines and
on teaching machines, which I don't have time to maintain separately.
The system in AFS is a slightly modified Debian sarge.  Each machine
boots from a kernel and ramdisk, which can even be easily loaded over
the network, though I normally have local copies.  The linux on the
ramdisk sets up the hard disk, transfers to it, starts networking and
AFS.  I then run package, which sets up a directory tree with some
files and a lot of symbolic links, including links for /usr and
most files in /lib, /bin, and /var.  Package also cleans up the local
file system.  After this I exec /sbin/init, and the system comes up
as normal.  The interesting thing is that virtually everything is
cached in /var/cache/openafs, and this provides a surprisingly fast
performance, even for KDE applications, except of course on older,
slower machines.  (The system even works on a Pentium I 75mhz machine,
but I don't advise using KDE on that!)

User home directories are mounted using ncpfs; the user IDs are
created dynamically and password file entries added at the first
login, when the passwords are verified using ldap, then the
novell home directory is mounted.

The size of the full system in AFS is about 3.2 gb at present; the
initrd and kernel are together less than 4mb, and I use about
500mb of cache, though of course I can increase that quite easily.
Currently we use 2.6.11.9 with openafs 1.3.82 and a patch.

On Tue 17 May 2005 at 15:29:19 -0400, Jim Rees <rees@umich.edu> wrote:
> The main problem I've had with putting system directories like /usr/bin in
> afs is that when you go to install something it will fail because
> chmod/chown will fail.

I've got round this by simply not installing anything on the client
machines.  I think of the system as a single Linux machine, and I can
install one or another package in it in a more-or-less usual fashion.

> One way around this is to install as root/admin but that scares me.  I have
> modified /usr/bin/install so that if chmod or chown fails, it pretends to
> succeed, and that helps.

I actually maintain the image in an area completely outside of afs, then
use rsync as admin to update the read-write copy.  I have a special
procedure which allows me to test the read-write copy before doing
my vos releases, which can then in principle be scheduled to run
automatically, though in fact I do it by hand early in the morning.

> A smaller problem is no cross-dir links, but that is usually only a problem
> for man pages.

I had problems at the first install with terminfo and with timezone files.

> After things are installed they usually just work.

Yes, but I didn't actually copy all of /var into the system.
Here's the actual sizes:

     4       /var/local
     4       /var/mail
     4       /var/opt
     4       /var/www
     8       /var/lock
     12      /var/games
     12      /var/state
     60      /var/spool
     84      /var/run
     220     /var/lib
     1468    /var/tmp
     1632    /var/backups
     1848    /var/log
     391340  /var/cache

Most of the files that I don't want to have changeable on the systems,
including all of /var/lib/dpkg, is simply in afs with a symlink on
the local disk.  Of course, this sometimes means that I've overlooked
the need to write something or other, and it fails, but I can fix
it by just changing the package files.

I wrote a little utility that takes the hostname and IP address
of the machine and adds some %defines or %undefs at the beginning
of the package files, before macro processing with mpp.  This
allows me to have some things on certain machines only, to
configure some machines differently from others, or even to set up
a system to perform some maintentance task the next time it boots.
And on each machine I install a command which gets run before
init starts; this can be customised, for example to change the
default window manager when a teacher wants something different
for his course.

     -- Owen
     LeBlanc@mcc.ac.uk