[OpenAFS-devel] Solaris 10 predicament update

[Dale Ghent]

I'm using this email to report on the problem, what I've found, and  
lay out what our options are.

Background
With the advent of Solaris 10 8/07 (aka s10u4), the internal Private  
kernel interfaces AFS used to access network interface properties  
changed due to the integration of the pfhooks/netstack feature.  
Specifically, a argument was added to the ill_* functions to  
accommodate the netstack changes. This aspect negates their use vis a  
vis  maintaining AFS driver binary compatibility across all  
permutations of the Solaris kernel.

Situation
The AFS driver code uses the ILL_* macros and functions (defined in  
<inet/ip.h>) to walk a list of network interfaces and, as is the case  
in SetServerPrefs() in src/afs/afs_server.c, pick the best interface  
to bind to in order to talk to the AFS server holding the cell's root  
volume. They are also used in src/rx/SOLARIS/rx_knet.c to gather MTU  
settings of the interface a rx packet was received on, and uses the  
retrieved value to adjust the RX UDP packet size to prevent  
fragmentation.

My research has concluded that there are no straight-forward Public  
interfaces in the Solaris kernel which exist all the way back to  
Solaris 10 FCS. Also, there are no Private interfaces which directly  
address our needs and are stable back to Solaris 10 FCS.

What to do?
There are a few alternatives we can consider, and I'd like to present  
them for discussion... ordered from "most likely" to "least likely":

1) We can mimic what we've traditionally done and instead of using  
ILL_*,  use the Public ldi_ioctl() interface to make sockio calls to / 
dev/udp and fill Private structs with returned network interface  
information. While this may be alright to do in the case of  
SetServerPrefs(), it would be a huge performance impact in the rx  
code. When a rx UDP is received via , the call stack looks like this:

rxi_ReceivePacket->rxi_FindConnection->rxi_FindPeer- 
 >rxi_InitPeerParams()->rxi_FindIfMTU()->rxi_GetIFInfo()

Both rxi_FindIfMTU() and rxi_GetIFInfo() walk the ILL structs to get  
interface address and MTU and from what I can tell, it does this for  
*every* *received* *packet*. So, being that AFS seems rather  
obsessive about staying up-to-date on a interface's MTU, it would  
mean that we would be doing ioctls on a file (/dev/udp) for every rx  
packet we get. This would be hellishly expensive. Would this be a  
correct assumption?


2) Option 2 would be to use the above mentioned ioctl-based method,  
but to remove it entirely from the critical code path. We could, at  
AFSinit() time, create a worker thread which would periodically  
update a global struct of interface telemetry. The worker thread  
would wake up every, say, 30 seconds (tunable), lock the struct via  
mutex, update it, unlock, and return to sleep. The RX and  
ServerPredfs code can read their desired values from this struct when  
they need it, spinning if need be.


3) This is Rob's idea, so blame him if you reel back in horror. We  
find a conditional by testing for a netstack symbol in the kernel ip  
module. If TRUE, we have a pointer function that points to the new  
ILL_ functions with the extra argument. If FALSE, we point to the old  
ones. Yum. This would certainly involve the least amount of code.


4) We toss caution to the wind and let modern routers deal with UDP  
frags the way they should be and dispense with the UDP packet size  
adjustments based on MTU, or at least nail them to 1500. If you're  
still using AFS over a PPP connection... well... sorry 'bout that. We  
also let the kernel routing table do its job and dispense with  
selecting interfaces. I don't think even the NFS code jumps through  
these kinds of hoops. Is there a reason we should be? I admit I'm not  
too familiar with the inner details and history of things here, so  
feel free to gently clue me in.


5) Continue to use the ILL method and release OpenAFS 1.4.5 with the  
code being compatible with s10u4. We simply tell people that if you  
want to run OpenAFS client version 1.4.5 or greater, you also need to  
run Solaris KU 120012-14 (x86) or whatever the analog is if you're  
running SPARC.

6) Any other idears?

/dale


--
Dale Ghent
Specialist, Storage and UNIX Systems
UMBC - Office of Information Technology
ECS 201 - x51705