[OpenAFS] Weird issue with AFS hang on a couple of legacy RHEL 6 servers

[Benjamin Kaduk]

Anything relevant from `echo 't' | sudo tee /proc/sysrq-trigger`?

-Ben

On Tue, Sep 08, 2020 at 05:31:22PM +0000, Sebby, Brian A. wrote:
> Hi,
> 
> I have a few legacy RHEL 6 servers that are still running an older 1.6.x series DKMS client, which were recently patched and rebooted.  On a couple of them, access to AFS is now just hanging – and I cannot figure out why.  They have the same kernel release and kernel module as some other systems on the same network that are working, so it doesn’t look like it would be any sort of firewall issue.  Would anyone have any suggestions on how to debug this?  I don’t know if I can upgrade them to 1.8 since we are trying to keep our legacy systems fairly static.
> 
> Some information:
> 
> % uname -a
> Linux ###### 2.6.32-754.31.1.el6.x86_64 #1 SMP Mon Jun 15 08:08:31 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
> % rxdebug localhost 7001 -v
> Trying 127.0.0.1 (port 7001):
> AFS version:  OpenAFS 1.6.20.1 built  2020-08-05
> % rxdebug localhost 7001
> Trying 127.0.0.1 (port 7001):
> Free packets: 167/179, packet reclaims: 379, calls: 647, used FDs: 64
> not waiting for packets.
> 0 calls waiting for a thread
> 1 threads are idle
> 0 calls have waited for a thread
> Connection from host ######, port 7000, Cuid 97239d4d/7cab1088
>   serial 77,  natMTU 1444, security index 0, client conn
>     call 0: # 1, state active, mode: receiving, flags: reader_wait, has_input_packets
>     call 1: # 0, state not initialized
>     call 2: # 0, state not initialized
>     call 3: # 0, state not initialized
> Done.
> 
> I don’t see anything in the system log that would point to any particular issues.
> 
> It looks like it’s gotten some response, but has not processed them.  The symptom is that when you try to go to anything under /afs/<cell>/, it just hangs – whether doing an ls or trying to read a file, etc.
> 
> 
> Thanks,
> 
> Brian
> 
> --
> Brian Sebby  (sebby@anl.gov<mailto:sebby@anl.gov>)  |  Information Technology Infrastructure
> Phone: +1 630.252.9935        |  Business Information Services
> Cell:  +1 630.921.4305        |  Argonne National Laboratory
>