[OpenAFS] re: my afs wish list

[Dean Anderson]

> From: Derrick J Brashear <shadow@dementia.org>
>
> On Wed, 30 Apr 2003, Douglas E. Engert wrote:
>
> > Be careful with the wish list, it sounds like you are redesigning DCE and DFS.
> >
> > IMHO DCE failed because it tried to do all things for all people,
> > mixing up authentication, authorization and file services, all into
> > one cell/realm/domain, thus making it way to complicated.

You have to keep in mind this predates working open source operating
systems. In 1990, BSD 4.4 was just released sans much kernel code. (OSF
provided 1/3 of the funding for BSD 4.4, the rest coming from HP and DEC).
The free part of Mach didn't do anything interesting for a non-OS
researcher. The unix-like part of Mach required a Unix source license.
386BSD had just started to rewrite the missing code. (I also came across
some 386BSD 0.1 tapes).  Linux hadn't even started.  Minix was around, but
just a kiddie toy.  Only hardware vendor companies and some universities
had source licences to Unix, and used either SysV or BSD, usually with a
lot of proprietary modifications.  You had to get these companies to buy
into DCE, else it wasn't going to happen.

DCE was practically a complete re-write of everything, all at once.
Porting it to new platforms suffered almost the same problems as AFS vs
NFS administration: Developers have to learn a lot of new things before
they can even get started.  How do you know if you introduced a bug if you
don't know how it is supposed to work?  Unlike OSF/1 and Motif, there were
no reference platforms*** for DCE. (OSF/1 would have been a good choice,
but the DCE group wanted to go with SysV.4. This was insane because the
V.4 people were (at the time) trying to make OSF go away.  (Scott McNealy
of Sun promised Motif would never run on SunOS). AT&T was the evil empire
that OSF was formed to fight.  They were the Microsoft of the 80's.

Back then, we were afraid that AT&T _would_ do with control of Unix what
Microsoft actually did with control of the desktop. This was at the heart
of the open source movement.  "Good thinking", that the SysV.4 vendor
companies will stick their neck out for DCE. Right.  In what is surely a
God-inspired, truly remarkable irony, today the Open Group owns the Unix
Trademark, and SCO (partly owned by Microsoft) owns the SysV.4 code base
and is suing IBM over Linux.  But Microsoft rules the world, for now.
Attila the Hun reportedly said to Aetius (Roman general who allied with
the Huns to defeat the Visigoths): "You won by deceit. That is not my
way". Aetius responded: "Which way rules the world?".

There was a great deal of internal fighting between the OSF/1 group and
the DCE group that proved self-defeating. (mostly around the question of
whether ".:" violated POSIX. My first experience with "mail won't go
through because the spool is full" happened during one of the more
vitrolic of these arguments. 100K+ long messages to thousands of people,
involving many in the Unix community, like Dennis Ritchie and others. A
big disk at the time was only 600Meg.  The DCE group said they weren't
going to port DCE to OSF/1.  Members and Vendors were getting conflicting
messages from each group.  The (unsurprising) response is to wait until
things settle out. That was the worst thing possible.

So it was hard to get vendors to pickup both DCE and OSF/1.  Also making
things difficult was the dot-zero nature of the changes. Like most new or
radically altered code bases, the code wasn't stable, and its hard to get
commercial vendors to ship something that might not be stable, but if they
don't, bugs won't get found and fixed.  Pricing constraints from the
component vendors (Transarc mostly, and HP) made it hard to get things out
to the "wanna play with it" crowd. The "early adopters", the people who
love to play with new stuff, don't have 10 or 20K to spend on what might
be a toy until they prove otherwise. It was expensive and had to be sold
as "mission critical" to justify the price.  Of course, we saw what
happened at the Olympic Games to "mission critical". Sigh.

I think IBM (before transarc purchase, too) was the only vendor to
actually ship Episode.  Other venders shipped DFS but only with a VFS
exporter.  Most DCE sites never used DFS at all. They used only the RPC
and cell directory services. (which offered much more than say, XDR).
Companies like NYSE and Citibank just developed applications that used DCE
rpc to fail-over to other servers.  This was pre-CORBA, and CORBA still
doesn't really compare well with DCE rpc services.  Though there is no
reason that fail-over and CDS like services can't be implemented for
CORBA, it just isn't. Still.

Like was mentioned before, if you use a VFS exporter, half or more of the
benefits of DFS/AFS are lost because you can't move or backup filesets. So
it was hard to see any benefit to it.  It was kind of slow (due to extra
VFS+ layer, 50Mhz CPU and memory and disk challenged machines at the time,
and extra byte locking traffic as compared with AFS) and untuned, and a
little fragile.  You had to know a lot to make it work and keep it
working.  Also, unlike AFS, DFS didn't allow unauthenticated outsiders to
get far. There weren't many DFS sites, and no global CellServDB to find
the few that there were.

In quite a lot of ways, DCE was ahead of its time, and was then shot in
the head by its creators.

> I don't think it helped that the pieces weren't always well-integrated.
>
> I recall that a DCE RPC version of Ubik existed but secd used the
> equivalent of kprop.

Yes, I think that's right. This was a known issue that was to be
resolved/decided later.  How tightly integrated to make Kerberos had been
a question for both DCE and AFS. Many people wanted to run generic KDCs.
You couldn't possibly do that if you changed the protocols used by
Kerberos. As I recall, you couldn't use a generic KDC with either DCE or
Transarc AFS, but I think they wanted to have that capability some day.
Looks like OpenAFS can just about do that.



***Reference platforms were frequently purchased by vendors, and used
along with porting guides to test and compare operation. OSF/1 had the DEC
pmax(risc), the i386 pc(cisc), and the Encore multimax(multiprocessor) as
reference platforms.  One of the early characteristics of risc machines
was the "unaligned access trap". Risc gets some of it simplicity by not
allowing words to be unaligned. An integer has to be stored on a word
boundary in memory. In C structures, though, things are not aligned this
way. In the old days, porting some software to a risc machine was plagued
with "Unaligned Access violations". You had to find the structure, and add
padding, or rewrite the code if the structure was a disk file, to make it
work. This is a PITA.

So I wrote a trap handler to handle the "unaligned access" trap, which
usually resulted in program termination. My handler decoded the
instruction that caused the trap (either a read or write word), and used
byte reads and shifts to get the word into or out of the proper register.
Then I just adjusted the program counter to skip that instruction and
resumed the program as though nothing happened.  This took a toll on
performance, so we put a kernel printf to warn the user.  With some
cajoling, and the printf, I got this into the mainline OSF/1 code.  DEC
then picked it up for the PMAX, and ported it to the Alpha. (I remember
being thrilled when I found my Digital Unix Alpha box had this code. The
trap handler was in assembly, so it wasn't just overlooked and shipped).
HP, IBM, and SGI apparently noticed, and did the same. Finally, Sun came
round in Solaris.  I don't know of any vendor now that will kill programs
on an unaligned access.  DEC even took out the kernel printf, or made it
controlled by a kernel variable.  Reference platforms can have a profound
effect on things.  OpenAFS has reference platforms in the form of
pre-existing AFS cells. When it works with these other cells, its probably
working right.

		--Dean