[OpenAFS] afs memcache tuning... lockups in afs_cv_wait

[Mike Polek]

Mike Garrison wrote:
>> afsd -memcache -blocks 655360 -chunksize 14 -stat 50000 -daemons 6
>>     -volumes 512 -nosettime
> 
> 
> I'd suggest tweaking -rxpck to be higher than the default, it may  
> actually help with the issue you're running into. We use 2000 for it.  
> My mind slips me as to why we picked that number at this point.

Aha... I didn't know you could use that param with afsd. It's not
in the main docs on the website, and it's not in the comments that
list the params in afsd.c.

Note to any concerned: Can the comments in the file that say what
    the available parameters are be updated when a parameter is added?
    I know the web docs lag behind a bit... but updating the param
    processing loop could trigger a comment update. It's in the same file.

What is the default value? It looks like 80, I think....
I'm a little unclear if that 80 means create 80 rx packets,
or create 80 "extra" rx packets. I see a total of 130 when
I boot... is that normal?  How high can that param go?
And how much memory and what kind does it chew up?

> 
> Only 9 free packets. Ew!
> 

> 
> 
> noBuffers.. you've had to discard some rx packets because you're out  of 
> rx packets.
> 

> 
> Even more noBuffers .. and again, the alloc-failures for send are huge.

> 
> The only things that really stick out to me is the low number of rx  
> packets, I'd try increasing rxpck and seeing if that helps.  
> Unfortunately, I don't have much experience with the memcache, but I  
> have a strong feeling that you shouldn't be seeing such a high number  
> of alloc failures for sending..

I'd say that looks like the underlying issue. Thanks for the help!
I always like to have a pair of eyeballs other than my own see
these things after I've stared at them for too long...  I miss things.

I'll do some testing and see how it goes.


Thanks!
Mike