[OpenAFS-devel] Re: sanity check: client vldb cache time

[Jeffrey Hutzelman]

On Thu, 2014-01-23 at 14:54 -0600, Andrew Deason wrote:

> > > Also, just from looking at the code, I can't actually see where the
> > > volcache entry is invalided for RW volumes. Both of the cases I
> > > mentioned (afs_CheckCallbacks -> afs_ResetVolumeInfo, and
> > > afs_CheckVolumeNames -> afs_ResetVolumeInfo), are only for RO volumes.
> > 
> > No, you're right; mere expiry doesn't cause non-RW entries to be
> > rechecked, ever.
> 
> I feel like there's maybe a typo or something here, but I'm not sure
> where. It seems like you're agreeing with me, but that last part isn't
> what I said :)

s/non-RW/non-RO/


> The expiry of the volcache entry should cause the entry to be looked up
> again if we ever need to hit the fileserver for something (since we'll
> look up the vol struct to get the list of servers, fail, and call the
> VLDB to get the info). I'm not sure if you're saying something about how
> expiring the RW doesn't affect the non-RW, or if you're just saying
> "yeah, this doesn't work for RWs", or ...?

Expiry of a volcache entry on its own (that is, the mere fact that the
expiration time has passed) does nothing.  The only things that look at
the expiration time are the callback code that updates it, and the code
we've been discussing that resets expired entries, but only for RO
volumes.


> That, or the fileserver is renumbered.

Yes, or that.  Though we ought to be able to deal with that without
rechecking each VLDB entry.  Perhaps when a server goes down we should
call VL_GetAddrsU for that server.




> But it might be possible
> to mitigate that by just checking the VLDB once per down event, and if
> the volume is still where we expect it to be, we assume the server is
> just down. I'm not even beginning to think of how to implement it,
> though, ugh.

Yeah, "ugh" is right.

If the server is up, then we'll find out about volumes that have been
moved from it, and no further work is necessary.

If the server is actually just down and no volumes have been moved from
it, then it is sufficient to notice the server is down.  Once the server
comes back up, it will be happy to tell us VNOVOL for any volume that
has been moved.  Of course, getting VNOVOL for hundreds of volumes
because they were all moved will result in throttling, which is no fun.

If the server has been renumbered, we're never going to get a response.
But rechecking even one volume on that server will get us the new
addresses, at which point we won't have any trouble accessing the other
volumes, without ever having to invalidate the volcache entries.

If the server has permanently been shut down, then we're going to need
to discover new VLDB entries for each volume.  There's nothing for that
but to fetch each and every one from the VLDB.


Unfortunately, the trouble is that even if a fileserver is down, there
might be volumes we think are on that server which really aren't,
because they were moved before the server went down and we never found
out about it.  This can happen pretty easily, since once we use a non-RO
volume we remember about it ~forever, never update the volcache, and are
never told about it moving except if we happen to talk to the old
server.

The only sane solution to that problem is for volcache entries to
expire.




> Another way of helping with this would be to allow the fileserver to
> signal clients when it's shutting down.


>  Either an ICBS3 (very
> heavyweight for DAFS), or potentially a new RXAFSCB RPC. I don't mean
> doing this for every shutdown, just ones where the administrator
> explicitly somehow says "notify all the clients".

The only problem with notifying on shutdown is you can wait a long time
for clients that are unreachable, and the notification only goes to
clients the fileserver hasn't forgotten about.

-- Jeff