[AFS3-std] Unlimited array lengths

[Andrew Deason]

In the current AFS RPC-L definitions, there are several places where we
have an array with no upper bound on its length. While implementations
can and do cap this to prevent 32-bit rollover while allocating memory,
this still means that we can cause an endpoint to allocate up to about
2G memory per relevant structure for a single call before the RPC
handler is even reached (at least, the way that rxgen works in OpenAFS
right now). On systems that overcommit memory, and even to some extent
on systems that don't, that can be a serious DoS problem.

Many instances of this are a part of RPCs that are usually only run by
administrators, such as AFSVolCreateVolume's "name", so I don't think
that's as much of an urgent concern. However, some are part of RPCs that
are run by unauthenticated users, such as namelist/idlist in PT_NameToID
and PT_IDToName, which were recently involved in an OpenAFS security
advisory. And of course, many are involved in a server's response to a
client, potentially allowing a server (or an attacker impersonating a
server) to use up a ton of client memory.

In OpenAFS, one proposed way to improve this is just to modify our .xg
files to put limits on all variable-length arrays (openafs gerrit 9588
is one such possible change). We could also have rxgen have an arbitrary
universal cap, but I'm not sure about that, since what a "reasonable"
max length is potentially varies so much between different
structures/RPCs.

Since modifying the RPC-L means modifying the protocol, it was mentioned
that this issue should be brought up here. While in the short term, I
think OpenAFS may just implement some limits to ensure server/client
stability, it seems like we should agree on array limits here, at least
for the long term (do we need an I-D just to specify a bunch of max
lengths?). I'm also not clear on if it's considered acceptable to just
put some limits in for existing structures/RPCs, since that would change
the signature of the RPC.

Ideally, I think I would want to modify OpenAFS' rxgen to not allow
unbounded variable arrays; if something really warrants ~2G allocations,
it should be explicitly specified. But I'm not clear on if that's
allowed, or how we get there.

-- 
Andrew Deason
adeason@sinenomine.net