[AFS3-std] Hackathon Summary

[Simon Wilkinson]

A group of us - Jeffery Altman, Matt Benjamin, Derrick Brashear,  
Alistair Ferguson, Christof Hanke, Tom Keiser, Hartmut Reuter, Marcus  
Watts, Rod Widdowson and myself met for 3 days in Edinburgh at the end  
of last month. We discussed a wide variety of AFS issues. A jabber  
chat log of our discussions is available at http://conference.openafs.org/hackathon@conference.openafs.org/ 
  (see the 2009-09-22 and 2009-09-24 files logs)

I'm not going to attempt to summarise our discussions in much detail.  
However, I have noted below the topics we discussed, and any  
conclusions that I believe we reached. Where we identified the next  
steps to be taken, I've also noted this in the hope that we can keep  
things moving forwards.

Extended Callbacks
------------------

Matt presented his extended callbacks draft. The discussion ranged  
between protocol, implementation and code management issues. On the  
protocol front, it was felt that the draft was approaching consensus,  
but that concerns remained around the changes to the behaviour of  
callback breaks (in particular, whether an RPC can return before all  
callbacks have been broken). Given the lack of consensus on this  
topic, we agreed that the draft would drop mention of callback  
coalescing entirely. Other issues included the behaviour of clients  
which receive extended
callbacks over untrusted channels, and the risk of deploying extended  
callbacks on servers which only have a small number configured. Matt  
will produce a draft addressing these issues, and then we will attempt  
to move forwards with a consensus call.

We will attempt to address the asynchrony issue at a later date. Given  
that this change is arguably a modification to afs3 semantics, we'll  
attempt to engage a wider body of the community in this discussion

In discussing the implementation, we considered the range of xcb  
dependencies, in particular those on mcas and libosi. It was felt that  
blocking extended callbacks on getting libosi into the tree was  
undesirable - in particular, a desire was expressed for the xcb code  
to use the existing pthread implementation, rather than pulling in  
osi's thread abstractions. We agreed that we would not enable, or  
expose the callback coalescing code in the OpenAFS implementation,  
pending further discussion of this issue. We didn't resolve the issue  
of whether we should use MCAS's native atomics everywhere, or whether  
we should prefer atomic operations provided by the operating system.

Code management concerns were expressed on a number of occasions  
throughout the meeting. I'll summarise these in a section of their own  
towards the end.

We agreed the following:

Matt will publish a new version of the draft which:
1) Removes any mention asynchronous behaviour for callback breaks
2) Extends the security considerations section to state that if the  
client receives an XCB for metadata on an untrusted connection, it  
should treat it as a normal callback break.
3) Adds an implementation note on the risks for servers with a small  
number of callbacks

Simon (in the absence of a chair for afs3-stds) will issue a call for  
consensus on the updated
draft.

Matt will then:
1) Remove as many OSI dependencies from xcb implementation as possible
2) Remove asynchronous callback breaks from the visible implementation  
(off by default, no switch to enable)
3) Push changes to gerrit (separate patches for Windows CM, Unix CM,  
and fileserver)

All will then take time to review these changes.

rx/osd
------

Hartmut and Christof presented their protocol documentation for rx/ 
osd, available online at http://pfanne.rzg.mpg.de/trac/openAFS-OSD/wiki/Specs

A desire was expressed to not use an IP address to identify OSD  
servers, but use a UUID instead, and register OSDs in an extended vldb  
which also knows about ports. We decided that this could be fixed in a  
later protocol version, but that for now servers should be expressed  
as a union of IP address and UUID so we don't have to rev the RPC  
later on.

Where 'expires' is used in the structures it will become an absolute  
time, and become a 64 bit value.

After a discussion of the consistency issues in the current mirroring  
implementation (if some mirrors go offline, then you can end up with  
multiple OSDs with different versions of the data), we decided that  
mirroring would be out of scope for the initial rxosd integration.

UUID will be removed from getOSDLocation, StartAsyncFetch and  
StartASyncStore, as the filserver already knows the UUID (from  
connection establishment)

Times in general will become 64bits

Hartmut and Christof will publish a revised set of protocol  
specifications addressing these, and will continue to split out the  
code into chunks and submit them to gerrit.

RPCRefresh
----------

This was roughly split into topic headings as follows. Simon agreed to  
edit a document proposing these changes.

UUIDs
We had a general discussion of how UUIDs might fit into the AFS  
protocol, beginning with an expressed desire to include client UUIDS  
in every call, to minimise the issues with using IP addresses to  
locate clients. After discussion, this approach was rejected, because  
client information needs to be available before an RPC has been  
decoded. Instead, we proposed making our new security classes exchange  
UUID information as part of the challenge/response connection  
establishment. Tom will specify and develop a new 'clear' class which  
will exchange UUIDs, and be a drop in replacement for the current null  
class.

We then discussed the issue where, through a race condition when  
servers change IP addresses, an RPC may arrive at a server other than  
the one it is destined for and, in some rare situations, mutate data  
incorrectly. It was felt that this was too rare an occurrence to  
justify adding a server UUID to every data mutating RPC. Tom's new  
clear class could also be used to address this case.

Jeffrey discussed ways in which we could use UUIDs (as SIDs) in the  
ptserver. We agreed that this was out of scope for this round, but  
that it was an interesting topic. Jeffrey will write up a proposal.

64bit time
We agreed to change all of the time occurences in AFS RPCS (but not  
any on disk occurences) to be 64bit, with a granularity of 100ns,

RXOSD changes
We'd like to be able to specify a quota value for the number of files  
in a volume (rxosd already implements this, but currently does so by  
using a 'spare' field) - this is particularly relevant for sites which  
are using tape storage. This changes VolIntInfo and those RPCS which  
use it in the volint family, and Fetch/StoreVolumeStatus. Christof  
will provide a detailed list, and suitable language.

We will rename 'ResidencyMask' to 'DataAccessProtocol' in a revised  
version of FetchStatus.

In VolIntInfo we want to add afs_uint32 as 'osdPolicy'

Future proofing FIDs
We agreed to change volumeID, vnode and uniquifier to all be 64 bit  
values, with 0xffffffff and 0 being reserved.

Quotas / Block size
Fields which report quotas and volume block sizes should become 64  
bit, even if we can't use them all now. This affects Store/ 
FetchVolumeStatus, VolIntInfo and VolIntXinfo

Last update time of volume
We'll add a field to FetchStatus to store the last update time of the  
volume, so it can be used to optimise handling of read only volumes in  
the cache manager. Alistair will arrange for this to be implemented

Per file ACLs
We'll define semantics for the new FetchACL and StoreACL commands when  
they are invoked on files
The new FetchStatus will be defined as returning per file ACL  
information.

ACL Extensions
We'll extend ACLs to use 32bits of access data on the wire, and  
reserve all of the new 16 bits for our own use.

FetchStatus cleanups
*) InterfaceVersion will be removed
*) Length and Length_hi will be combined into a 64bit length
*) Dataversion and dataVersionHigh will be a single 64bit value.
*) User and Group ID will become 64bit
*) ParentVnode and ParentUnique will become 64bit (inline with the FID  
changes being made elsewhere)
*) SyncCounter will be removed

Tom: Propose new clear rx security class
Jeffrey: Write proposal for using SIDs within pts
Jeffrey: Provide langauge for 64bit time
Christof: Provide language for file quotas
Simon: Edit this into a manageable whole
Ali: Arrange for code to make use of the 'volume last update time'  
field to be written.

SRV records
-----------

Use them to replace AFSDB - standardise supporting for vlserver and  
ptserver _afs3-vlserver._udp.<cellname>, SRV priority matches to rank,  
weight should be used as input to the server selection randomisation  
function.

Jeffrey will write a short I-D describing how AFS uses SRV records.

rxk5
----

Marcus presented his rxk5 document - /afs/umich.edu/group/itd/build/ 
mdw/openafs/patches/rxk5-1.pdf

A lively discussion ensued. In particular, we discussed the initial  
packet problem at great length. This is where a client sends a packet  
containing valuable data to a server which only wants to accept  
encrypted connections. However, because the server tells the client  
that after it has received the first packet, the client may have just  
sent that data in cleartext. Jeffrey proposed a solution to this  
problem, which Marcus was unconvinced by. Further discussion is  
required.

We also debated the merits of using our locally developed k5ssl,  
against an externally maintained crypto library. Given that Heimdal's  
hcrypto is likely to be imported into the OpenAFS tree to support  
other AFS uses of crypto, the opinion was expressed that rxk5 should  
probably be built upon that.

A discussion of the problems of ubik's hard coded assumption that  
there will only ever be 3 security classes took place. It was agreed  
that a new, dynamic, interface will be defined to handle this.

We discussed Marcus's new cache manager properties list, which  
provides a sysctl like mechanism for exchanging configuration strings  
between cache manager and user space. The meeting was unable to reach  
consensus on this design, and we agreed that it should be discussed  
further on list.

Other agreed changes were:
*) The authenticator will be extended to support more than 4 calls per  
connection
*) Space will be added for an application level binding (AFS wants  
this to assert the client UUID, but we want to make it generic)
*) rkx5 will be modfied to use the Kerberos PRF+, rather than MD5
*) The cellname length in the new tokens pioctl will be extended to 256

Matt & Marcus: Update draft to reflect changes, break code into chunks  
and submit to gerrit
Marcus: Raise sysctl-style properties interface on openafs-devel
Simon: Import hcrypto into OpenAFS tree (as part of the rxgk work)

Generic Quotas
--------------

Christof had raised the issue of providing a more generic quota  
mechanism, which allows more
flexible definitions of what quota might be (rxosd would like to be  
able to apply a separate quota to files under a certain size, for  
example)

We discussed implementing this as a set of tag value pairs, with each  
pair having a globally defined meaning. Individual tags need not be  
implemented on every fileserver - there should be an RPC by which  
clients can determine which tags a fileserver supports. We want to  
implement this by revising existing RPCs which take quota values, and  
use it to replace the quota values that those RPCs already contain.

Christof will write a document describing this, but we won't block the  
RPC refresh on it

Volume State
------------

Tom wants to be able to communicate to the client the type of  
fileserver its talking to, and provide a 'raw' and a 'mapped'  
indicator of the volume status. (raw is implementation dependent,  
mapped uses globally defined error codes). We agreed that fileserver  
type could, for now, be expressed as a capability bit, and that the  
volume state fields should be new parameters within VolIntInfo

Tom will write an I-D describing this. Again, we won't hold RPC  
refresh up for these changes

rx/udp improvements
-------------------

Jeffrey discussed changes he is making to RTT calculations such that  
the algorithms better reflect Phil Karn's findings from 1987. This  
seemed uncontentious - Jeffrey will put a patch into gerrit.

Derrick discussed larger window support, which he will test, and  
discuss his findings further

Derrick discussed improvements he wants to make to RX negotation, by  
adding elements to the existing negotiation packet. In theory this  
should be backwards compatible, because existing clients use the  
packet size to determine the version of the structure they are  
receiving. We discussed the mechanism for progressing RX  
modifications, as there isn't an obvious body to do it in. The  
conclusion was to use afs3-stds, but make a deliberate attempt to  
reach out to those people who we know are using RX in other  
applications.

Derrick will write up an I-D describing this, and solicit feedback

rx/gk
-----

Simon presented his write-up of the current rx/gk protocol

Jeffrey expressed a desire for the first packet problem to be  
resolved, and for client uuids to be a part of the authenticator.

We had a long discussion of bytelife, and of key agility. The  
conlusion that was reached was that each packet should have a security  
header containing a key number that was used to encrypt that packet.  
Providing this key number is input to the PRF used to derive the  
transport key, keys can be revised at either the client, or server's,  
request. bytelife will remain advisory, however.

We decided that the ivec should be determined from the pseudo header,  
to solve the packet ordering problem.

We decided that the same pseudo header should be used as rxk5

Marcus pointed out that rxgk uses the first version of the rxkad  
authenticator, and strongly suggested that CITI's recommendations that  
lead to the rxkadv2 authenticator be studied and followed.

Marcus noted that using different numbers than rxkad for security  
levels will only cause implementation pain. We'll use the same as rxkad.

Simon will update the rxgk protocol document, and create a new one  
which describes its implementation within AFS.

miniosi
-------

Tom talked us through the current libosi, with a view to splitting it  
into chunks that we can start to pull into the tree.

We agreed that we will only pull in changes that are going to be used  
in the code, and import
libosi as follows:

Phase 0: A build framework
Phase 1: buildenv, compiler, types
Phase 2: platform/datamodel.h
Phase 3: Time

Each of these phases should update the rest of the tree to use the new  
functionality, so that
we don't, for example, end up with two descriptions of each platforms  
data model in the tree.

This is on the critical path for extended callbacks, so Tom and Matt  
will seek to move this forwards, with Tom doing the work and Matt  
making sure that it gets done.

Tom: Submit these chunks of libosi, and tree integration patches
Matt: nag Tom

Directory RPCs
--------------
http://michigan-openafs-lists.central.org/archives/afs3-standardization/2009September/000423.html

We discussed Matt's proposal for an explicit directory listing RPC  
(posted to afs3-stds during the hackathon). We were unable to achieve  
consensus on any of the issues this presents, beyond determining that  
server side sorting was unachievable.

DAFS
----

Tom would like a way of a server changing its advertised capabilities,  
without having to do an InitCallbackState3. Jeffrey proposed a new  
TellMeAboutYourself RPC which will take the capabilities as an IN  
parameter.

Tom will go away and think about this, and write something up.

There was agreement that DAFS (including the changes to the vnode and  
volume packages) needed to be properly documented.  Tom and Ali will  
get this done.

PRDB extensions
---------------

Derrick presented the original Swedish Hackathon work:
http://web.archive.org/web/20060211111127/http://www.afsig.se/snipsnap/space/prdb+extensions
We agreed that we need a way of creating multiple names in a single  
RPC, and provide a RenameAuthName RPC which takes a vector of triples  
of (type, old_opaque, new_opaque)

Derrick will produce an I-D documenting the new RPCs, and an  
implementation.

Code management
---------------

We spent a lot of time discussing issues of code management, and how  
large changes can get into the OpenAFS source code. It's hard to  
summarise what was a wide ranging and contradictory discussion, but  
the following points were made and broadly agreed with
*) Both clear protocol, and implementation documentation can hugely  
help with code review
*) Design and implementation discussions, during the development  
process, are hugely valuable
*) For 'large' projects we will merge by creating a git branch onto  
which changes will be reviewed. We can the flip the switch by doing a  
final merge commit, safe in the knowledge that the individual commits  
have already been reviewed.