[OpenAFS-devel] Progress on Linux in-kernel RxRPC library

[Jeffrey Hutzelman]

On Monday, March 21, 2005 07:01:11 PM -0500 Kyle Moffett 
<mrmacman_g4@mac.com> wrote:

> On Mar 21, 2005, at 18:20, Jeffrey Hutzelman wrote:
>> On Sunday, March 20, 2005 07:52:35 PM -0500 Kyle Moffett
>> <mrmacman_g4@mac.com> wrote:
>> This is really a bit naive.  It _might_ be a good model for the
>> cache manager, but it's not reasonable to assume it is OK to
>> combine connections belonging to arbitrary user processes.
>
> If the two calls are using the same key instance, IE: not only
> the same data but the same identifier, then they may be combined.
> This seems identical to the current OpenAFS model.

I think we might be talking about two different things here.  For RPC's 
made by the cache manager in order to access files on behalf of a user 
process, yes, this is exactly the same as the current model, and it will 
work fine.

My concern is about calls made directly by user-mode processes (presumably 
via the sockets interface).  While there are almost certainly times when it 
would be desirable to share connections between different user-mode 
processes with the same credentials, there are also going to be times when 
that's not a good idea.  And it's almost certainly _not_ a good idea for 
user-mode processes to share connections with the cache manager, because 
the fileserver tracks information about clients, including their 
capabilities and callback state, partly by connection (really, it only just 
occurred to me how bad that problem is -- yesterday I was just concerned 
that it might not be a good idea; now I know it's problematic).



>> It also totally ignores the fact that in OpenAFS, some accesses
>> are done on behalf of users by background processes, which must
>> have control over the credentials used for calls they make.
>
> You still have control over your credentials.  The RxRPC library
> in the kernel would search the process' keyring to find a key
> which it would use when using connections.  If you want to use
> a different set of credentials, just create a new thread, then
> create a new, empty thread-keyring.  Once you have that thread,
> put the credential for the user-process in the thread-keyring
> and make your RPC calls.  When done, terminate the thread. The
> same procedure works for processes and process-keyrings.

Ugh.  "Create a thread to make an RPC" seems a little heavyweight, 
especially for a process like the AFS background threads, or an NFS or HTTP 
or SMB protocol translator, which is making lots of calls with lots of 
different sets of credentials.  I think it needs to be possible to be able 
to make a call with a specific set of credentials, rather than using 
whatever keys the process has lying around.



>> It also completely ignores the issue of the axscache, which
>> stores cached access rights.  For this to work, the cache
>> manager must know to what PAG a process belongs, so in fact,
>> you have _not_ eliminated the need for PAG's.
>
> The axscache stores cached access rights, which are applied
> based on the identity (credentials) of the process, no?  To get
> the identity of the process just search the keyrings with the
> standard API for an OpenAFS token.  Take the token it finds,
> (The one in the most-local scope) and use it to check access.

Yeah.  This is essentially what Derrick's (as yet unreleased) code does. 
I'm sorry if I was unclear here -- it's not that automatic connection 
tracking won't _work_; it just doesn't save the cache manager from figuring 
out what PAG a process is in.



>> As Derrick points out, this is a chunk of fairly complex,
>> mostly platform-independent code that we will sill have to
>> maintain for other platforms.  So unless _every_ OS vendor
>> does this, _and_ they all interoperate well, it gains us
>> nothing but headaches.
>
> I've acknowledged that fact repeatedly, however:
>
>>> It would abstract you away from
>>> much of the nitty-gritty of the Linux networking internals
>>> (at least for future versions) and provide a smaller ABI
>>> that is less likely to break on upgrades.
>>
>> Let's not kid ourselves.  Linux does not provide a kernel ABI.
>
> Well, it's an ABI that is broken incompatibly nearly every
> revision.

And also anytime someone looks cross-eyed at the kernel config.

> I think that once the Rx layer in the Linux kernel
> has become reasonably well tested, its API is likely to remain
> more stable than the overall Linux kernel networking API.

I haven't been following the status of the Linux kernel networking API, but 
yes, I suspect that's a reasonably safe bet.  Whether it matters depends on 
what the Rx interface looks like, and how independent it is from the kernel 
sockets API.  Right now we basically create and bind a UDP socket, and then 
send and receive messages on it.  I think gaining greater stability would 
require using an Rx-specific API, rather than sockets.  I have to admit I 
haven't looked at the rxrpc module; does it provide such an interface?


>> I would very much like to see multiple interoperable
>> implementations of Rx. However, it's a fairly complex protocol
>> with a lot of subtle nuances (and more than a few warts), and
>> getting there will take a lot of effort.
>
> I completely agree.  It is my opinion that by writing such a
> system as part of the Linux kernel it is likely to get many
> more eyes looking at it and more potential usage, checking and
> verification than if it is developed externally (Although
> probably not initially).
>
> One other thing to consider is that a number of other OSen
> watch Linux for feature ideas.  If a simple user-space API
> comes along for Rx sockets, other OSes may implement it.

Don't I wish. :-)

>> Unless and until the Linux kernel's implementation is quite
>> stable and very well-tested, I'll be happy to debug problems,
>> but I won't suggest that anyone run it in production.
>
> We still only run OpenAFS 1.2 at my school because we hit too
> many problems with 1.3.

A recurring theme for us.  I'm not the gatekeeper, but I think we're 
probably pretty close to a 1.4.0 release.  If you've seen problems we don't 
know about yet, it would be good to hear about them.  And if you've seen 
problems we think we've fixed, it would be good to know if you still have 
them with the latest 1.3 release (without destabilizing your production 
environment, of course).


> Decent integration with the Linux Rx
> layer is not likely to occur until OpenAFS 1.5 or later, but
> it never hurts to work on long-term goals. :-D

I can't argue with that...

-- Jeffrey T. Hutzelman (N3NHS) <jhutz+@cmu.edu>
   Sr. Research Systems Programmer
   School of Computer Science - Research Computing Facility
   Carnegie Mellon University - Pittsburgh, PA