[OpenAFS-devel] Progress on Linux in-kernel RxRPC library

[Kyle Moffett]

On Mar 22, 2005, at 14:25, Jeffrey Hutzelman wrote:
> I think we might be talking about two different things here.  For
> RPC's made by the cache manager in order to access files on behalf
> of a user process, yes, this is exactly the same as the current
> model, and it will work fine.
>
> My concern is about calls made directly by user-mode processes
> (presumably via the sockets interface).  While there are almost
> certainly times when it would be desirable to share connections
> between different user-mode processes with the same credentials,
> there are also going to be times when that's not a good idea.

I suppose that that could be considered a bug, but I'm not entirely
familiar with all the Rx-related quirks in various applications, so
I may be wrong.  In any case, the hack would be to create a new key
with a different ID but the same data.  That would force the kernel
Rx layer to use a new connection for anything using that key.

> And it's almost certainly _not_ a good idea for user-mode
> processes to share connections with the cache manager, because the
> fileserver tracks information about clients, including their
> capabilities and callback state, partly by connection.

Hrm, that really would seem to be bug-like, although if it keeps
track of capabilities by connection it would be nearly impossible to
work around.  I don't suppose there is any way to have the
fileserver keep track of such data any other way?

> (really, it only just occurred to me how bad that problem is --
> yesterday I was just concerned that it might not be a good idea;
> now I know it's problematic).

Well, that's good to know, at least.  I suppose the cache manager
could always make a copy of the keys before using them, that way
it would never use the same connections, although sometimes that
connection sharing would be useful.

>>> It also totally ignores the fact that in OpenAFS, some accesses
>>> are done on behalf of users by background processes, which must
>>> have control over the credentials used for calls they make.
>>
>> You still have control over your credentials.  The RxRPC library
>> in the kernel would search the process' keyring to find a key
>> which it would use when using connections.  If you want to use
>> a different set of credentials, just create a new thread, then
>> create a new, empty thread-keyring.  Once you have that thread,
>> put the credential for the user-process in the thread-keyring
>> and make your RPC calls.  When done, terminate the thread. The
>> same procedure works for processes and process-keyrings.
>
> Ugh.  "Create a thread to make an RPC" seems a little heavyweight,
> especially for a process like the AFS background threads, or an
> NFS or HTTP or SMB protocol translator, which is making lots of
> calls with lots of different sets of credentials.  I think it
> needs to be possible to be able to make a call with a specific
> set of credentials, rather than using whatever keys the process
> has lying around.

Well, other models are:
Create a thread to handle a specific connection from a specific
user, including all the RPCs they ever need to make. Or, create a
bunch of worker threads beforehand and just have them switch keys
whenever they need to change which user they act as.

>>> It also completely ignores the issue of the axscache, which
>>> stores cached access rights.  For this to work, the cache
>>> manager must know to what PAG a process belongs, so in fact,
>>> you have _not_ eliminated the need for PAG's.
>>
>> The axscache stores cached access rights, which are applied
>> based on the identity (credentials) of the process, no?  To get
>> the identity of the process just search the keyrings with the
>> standard API for an OpenAFS token.  Take the token it finds,
>> (The one in the most-local scope) and use it to check access.
>
> Yeah.  This is essentially what Derrick's (as yet unreleased)
> code does. I'm sorry if I was unclear here -- it's not that
> automatic connection tracking won't _work_; it just doesn't save
> the cache manager from figuring out what PAG a process is in.

Ah, ok, I think the confusion here is that the "user" identity
would no longer be called a "PAG" by the Linux kernel, it would
instead be called an "afstoken" key id.

>> I think that once the Rx layer in the Linux kernel
>> has become reasonably well tested, its API is likely to remain
>> more stable than the overall Linux kernel networking API.
>
> I haven't been following the status of the Linux kernel
> networking API, but yes, I suspect that's a reasonably safe
> bet.  Whether it matters depends on what the Rx interface looks
> like, and how independent it is from the kernel sockets API.
> Right now we basically create and bind a UDP socket, and then
> send and receive messages on it.  I think gaining greater
> stability would require using an Rx-specific API, rather than
> sockets.  I have to admit I haven't looked at the rxrpc module;
> does it provide such an interface?

It provides these kinds of functions for kernel-space users:

rxrpc_add_service
rxrpc_call_abort
rxrpc_call_handle_error
rxrpc_call_read_data
rxrpc_call_write_data
rxrpc_conn_handle_error
rxrpc_create_call
rxrpc_create_connection
rxrpc_create_transport
rxrpc_del_service
rxrpc_incoming_call
rxrpc_krxiod_* <== These control the kernel Rx IO daemon
rxrpc_krxsecd_* <== These control the kernel Rx security daemon
rxrpc_krxtimod_* <== These control the kernel Rx timeout daemon
rxrpc_peer_lookup
rxrpc_put_call
rxrpc_put_connection
rxrpc_put_peer
rxrpc_put_transport

There are more, and I haven't tinkered with my quick grep&sed&awk
script to give function arguments, but you should get the idea.

>>> Unless and until the Linux kernel's implementation is quite
>>> stable and very well-tested, I'll be happy to debug problems,
>>> but I won't suggest that anyone run it in production.
>>
>> We still only run OpenAFS 1.2 at my school because we hit too
>> many problems with 1.3.
>
> A recurring theme for us.  I'm not the gatekeeper, but I think
> we're probably pretty close to a 1.4.0 release.  If you've seen
> problems we don't know about yet, it would be good to hear about
> them.  And if you've seen problems we think we've fixed, it
> would be good to know if you still have them with the latest 1.3
> release (without destabilizing your production environment, of
> course).

Hmm, ok, we'll have another go at it.  We're currently rebuilding
our setup on new servers in a highly-available configuration
(As per my messages on the Openafs-info list).  We'll try using
OpenAFS 1.3 on Linux 2.6 for a bit, and see how that works.  The
one part I have the hardest time with is getting decent OpenAFS
1.3 packages for Debian, but I think I may have those issues
resolved now.  We'll try it and see.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r  
!y?(-)
------END GEEK CODE BLOCK------