[OpenAFS] Re: fs process doesn't exit until I send a signal 9 (Hartmut Reuter)

[Mike Polek]

 >For what it's worth, I have servers that have thousands of volumes
 >> on each partition. (Ok... maybe a poor design choice, but I didn't
 >> know the single threaded volume server would be an issue when I did
 >> the design...) After 30 minutes, the bosserver assumes that the
 >> fileserver isn't going to stop, and does a kill -9 to stop it.
 >> I'm pretty sure it's just because of the sheer number of volumes
 >> to unmount.
 >>
 >> 1) Is there an easy way to change the timeout value? I'm not sure
 >>    yet if it's faster to do the kill -9 one minute into the shutdown
 >>    and just let the salvager do it's thing, or if it's better to
 >>    let the shutdown take an hour. I can say that it would be helpful
 >>    to have an emergency procedure that won't corrupt volumes for when
 >>    the shutdown is triggered by a power failure.  :-)


 > I think it's unsane if the shutdown takes that long. There must be a
 > problem with your clients, perhaps switched off PCs, that the callback
 > has to wait for timeouts. The writing of the volume information to disk
 > never should take that long even if you have 10000 volumes on a server.
 > If you have compiled with --enable-fast-restart you can kill your
 > fileserver after a while (after all active RPCs have finished) and the
 > only disadvantage at restart may be that the uniquifier is too low.

OK.. Perhaps my understanding is a bit off, and I'm making an assumption.
When a fileserver with a lot of volumes starts up, it has to attach
all of the volumes, as evidenced by the logs. As far as I can tell,
it attaches at about the rate of 20 volumes per second.

Fri Feb 25 23:31:25 2005 File server starting
Fri Feb 25 23:31:25 2005 afs_krb_get_lrealm failed, using pictage.com.
Fri Feb 25 23:31:40 2005 Set thread id 17 for FSYNC_sync
Fri Feb 25 23:49:23 2005 Partition /vicepa: attached 21294 volumes; 0 
volumes not attached
Sat Feb 26 00:00:08 2005 Partition /vicepb: attached 13655 volumes; 0 
volumes not attached
Sat Feb 26 00:10:38 2005 Partition /vicepc: attached 13233 volumes; 0 
volumes not attached
Sat Feb 26 00:21:14 2005 Partition /vicepd: attached 13404 volumes; 0 
volumes not attached
Sat Feb 26 00:21:14 2005 Set thread id 18 for 'FiveMinuteCheckLWP'
Sat Feb 26 00:21:14 2005 Set thread id 19 for 'HostCheckLWP'
Sat Feb 26 00:21:14 2005 Getting FileServer name...
Sat Feb 26 00:21:14 2005 FileServer host name is 'XXXXXXXX'
Sat Feb 26 00:21:14 2005 Getting FileServer address...
Sat Feb 26 00:21:14 2005 FileServer XXXXXXXX has address 192.168.XXX.YYY 
(0xf07a8c0 or 0xc0a8070f in host byte order)
Sat Feb 26 00:21:14 2005 File Server started Sat Feb 26 00:21:14 2005

Now, you're telling me that during a shutdown, it does not go
through and "detach" all of those volumes? Rather the problem
is callbacks and PC's that have been turned off? If that's the case,
how long will the fileserver keep trying before it gives up?
For me, the problem only seems to happen for servers with large numbers
of volumes, so I assumed that the 50 minutes it takes for the
fileserver to start would be mirrored on shutdown.

Thu Feb 24 01:17:14 2005: bos shutdown: fileserver failed to shutdown within 
1800 seconds

So, how would I know what's really going on? Will sending TSTP to the
fileserver process and upping the debug level help? Does afsmonitor
tell me how many callbacks are outstanding? And what exactly is it
doing when it is "attaching" a volume?

And here is a silly question, if during the attach process the fileserver
is not modifying anything on disk, then of course it could shut down without
detaching anything. BUT... if it IS marking the volumes ON-LINE in some way,
then didn't it have to mark the OFF-LINE during the shutdown process?

Does anybody else have servers with around 4T of info and over 10,000
per partition and NOT have this issue? If so, I'd love to hear from
you. :-)


 >>
 >> 2) I noticed that in the 1.3 branch, the volume server is multi-
 >>    threaded. (THANK YOU!!!) Does anybody know how this affects
 >>    shutdown/startup time? Should I still be looking for a way to
 >>    reduce the number of volumes on a server?


 > The volserver has nothing to do with the time needed by the fileserver
 > to shutdown. The volserver only does volume operations such as move,
 > backup or release.

Bummer. So it's the fileserver process itself that is responsible for
the whole "attach every volume on startup?" I'm using the -L option
to the fileserver to have more threads. Anything else I can do to
speed up the attach process? Is it something where more threads
helps? or is that irrelevant?

 >>
 >> 3) I've seen references to a "NoSalvage" option. Is that also new
 >>    in 1.3? or is it some sort of patch? Anybody have a really good
 >>    way of dealing with lots of volumes on a server? We currently
 >>    have almost 60T of storage, and it's growing. I like the idea
 >>    of having things well organized into finite volumes... it works
 >>    for our setup.


 > Is your NoSalvage option the same as --enable-fast-restart?

My guess is yes. Anything I need to do besides re-compile with that
option? I.e. any parameters to the salvager? When was it introduced?
I'm currently running openafs 1.2.13 on a mixture of RedHat versions
(7.3, 8.0, 9).

 >  if so, this
 > I introduced to avoid hours of salvaging after a crash. My experieance
 > was that the log contained nearly never a real error message. I think
 > it's better to let the fileserver automatically take a volume off-line
 > when he detects an inconsistency than to have to wait hours for a restart.
 >
 > Hartmut

I agree... and thanks!!!


Mike Polek