[OpenAFS] Network becomes terribly slow when cache manager flushes updates over xDSL

[Buhrmaster, Gary]

> The standard is ICMP WOULD FRAGMENT. People who block that don't
> deserve functional apps. Don't cater to them.

It appears you wish the RFCs were strictly implemented.
That is simply not true in the real world Internet, and
if we want to operate optimally in that world, we have to
deal with the reality.

In the real world, for real enterprises and service provides,
the high performant routers and switches have long ago
separated the forwarding plane (aka data plane) from the
management plane (aka control plane).  Packets that can just
be sent on their way use a high speed (usually hardware) path
to be sent out the next interface.  Packets that require
exception processing (for example, are too big, or the ttl
has expired), are punted to the route engine for additional
work.  The route engine processor is never powerful enough
to handle all the packets in software.  To protect its
critical functions (management of the device and forwarding
plane), one or more rate limiters (hardware and/or software)
are put in place to prevent too many exceptions from having
to be handled.  When the number of exceptions occurring is
low, you may get your TTL-expired, or your fragmentation-needed,
or your no-route ICMP message.  But if the rate directed to
the route engine is high you will not, those exception
packets being rate limited.  And I have seen cases on real
networks where on particular paths on particular days where
you consistently get "lucky", and would only occasionally
get that TTL-expired message (used by traceroute) if at all.
I presume intentionally or by accident, that particular router
is experiencing excessive control plane load (since it goes
back to normal later).  It continued to perform its important
packet forwarding functions at all times as it was designed to,
at the cost of the low priority exception processing.

Rate limiting protects the infrastructure, but it does mean that
exceptions will be treated with a very low priority.  And
ICMP generation is an exception.  I think one should accept
and act on fragmentation-needed, but it would also be
unreasonable to use that as your only hint/clue, since
you may never see one.  That is one reason blackhole PMTU
discovery was implemented.  Reality sucks.  Unfortunately,
we have to get over it to move forward.

Gary