[OpenAFS] [1.2.7] Strange file server meltdown

Russ Allbery rra@stanford.edu
Fri, 13 Dec 2002 12:06:52 -0800


Nickolai Zeldovich <kolya@MIT.EDU> writes:
> Russ Allbery <rra@stanford.edu> wrote:

>> (2) From the rxdebug output, I can see that there are several clients that
>>     have multiple connections open to the server.  Over time, as the
>>     server continues to have problems, they all continue opening more
>>     connections (about 20 per system over the course of ten minutes).  All
>>     of the clients that have this property appear to be Windows machines;
>>     none of the Unix clients seem to be doing this.

> Out of curiosity, could you send the rxdebug output that pertains to the
> problem client?  In particular, I'm interested whether there are client
> connections from the fileserver to the client that are waiting for the
> client to reply, for each server connection that's attached.

There were about 30 problem clients.  Here's one of them, chosen rather at
random.  Our registration database claims that this is a Windows XP client
(although be warned that that information isn't checked or guaranteed to
be accurate).

Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c12544
  serial 7,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: eof, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c12568
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: eof, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c1256c
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: eof, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c12570
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: receiving, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c12574
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: eof, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c12578
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: eof, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c1257c
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: eof, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c12580
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: receiving, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c12584
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: receiving, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c12588
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: receiving, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c1258c
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: receiving, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c12590
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: receiving, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c12594
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: error, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c12598
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: error, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c1259c
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: receiving, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c125a0
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: receiving, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c125a4
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: unknown, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c125a8
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: unknown, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c125ac
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: unknown, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c125b0
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: unknown, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c125b4
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: unknown, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c125b8
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: unknown, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c125bc
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: unknown, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c125c0
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: unknown, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c125c4
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: unknown, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c125c8
  serial 21,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: unknown, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized
Connection from host 128.12.173.130, port 7001, Cuid 92100687/f7c125cc
  serial 1,  natMTU 1444, security index 0, server conn
    call 0: # 1, state precall, mode: unknown, flags: waiting_for_process receive_done, has_input_packets
    call 1: # 0, state not initialized
    call 2: # 0, state not initialized
    call 3: # 0, state not initialized

-- 
Russ Allbery (rra@stanford.edu)             <http://www.eyrie.org/~eagle/>