[OpenAFS-devel] Out of band FetchData/StoreData

[Andrew Deason]

Hi,

[I will try to keep this concise, in order to get more people to read
this and to try to make it not take as much time. Thus, I'm skipping
over some details; just ask if you want me to explain something in more
depth.]

Last year, Sine Nomine developed a private speed enhancement to openafs
for a customer. Due to some of the more stringent requirements of the
request, we found that no known implementations or planned features
(RxOSD, RxTCP) would be viable, so we quickly developed something
in-house to improve RXAFS client<->server throughput.

What we have is a simple "ftp-like" control/data model, where we have
new RxUDP RPCs that initiate a transfer, and the actual payload is
transferred out of band. Currently the implementation is for a simple
ipv4/tcp protocol, but it could be used for other transports, too.

This model is not perfect; there is some suboptimal behavior, but we
made some shortcuts in order to hasten development time. If people want
this, I would want to work with the community on the architecture and
standardizing the wire protocol to make it useable for everyone.
However, I am not sure what people want to do, or how this fits into the
larger picture with, say, RxTCP.

While ideally running Rx over TCP entirely is a better solution (and
solves more problems than just speed), my impression is that that is
still quite some time away from being at all useful. In addition, that
does not have the ability (at least, I don't think it is the intention)
to allow for arbitrary out-of-band transports to be used, which may be
desirable. The OOB approach described is also much simpler in many
respects, and sidesteps some of the problems that make RxTCP difficult.

But in any case, at the end of the day we have "a" solution right now
that does appear to work pretty well. So, what would people prefer that
I do? Without being pushed in another direction, what I am planning on
doing is just submitting the protocol spec to afs3-stds, but I wanted to
ask here first for any general/strategic/future-direction issues or
discussion, and to solicit general opinions.


I am also currently not releasing the code publicly, because the
protocol used in it is a nonstandard extension (although using reserved
RPC code points), and I don't want someone running this in a public
cell. But am I just being overly cautious and silly about that? I have
no problem with code review (though it seems a bit early for that) or
people experimenting with it or anything, but I just imagine what
happens if Joe Q Admin sees "here's some code to make afs go faster"...



A little footnote: I'm trying to avoid technical details since I don't
think they're too important at this stage. But if anyone wants to
know... we can currently achieve throughput in excess of 5gbps
(memcache) or 6gbps (cache bypass) on a client/server pair that can only
reach up to about 6.5 gbps with iperf, so I have reason to believe it
can go much faster. I don't have better 'benchmark' statistics or
anything, since I don't have access myself to the machines running the
code.

-- 
Andrew Deason
adeason@sinenomine.net