[OpenAFS] afs memcache tuning... lockups in afs_cv_wait

[Jason Edgecombe]

Mike Polek wrote:
> Mike Garrison wrote:
>>
>> On Feb 13, 2008, at 9:17 PM, Mike Polek wrote:
>>> Mike Garrison wrote:
>> I'd suggest tweaking -rxpck to be higher than the default, it may  
>> actually help with the issue you're running into. We use 2000 for 
>> it.  My mind slips me as to why we picked that number at this point.
>
>> The only things that really stick out to me is the low number of rx  
>> packets, I'd try increasing rxpck and seeing if that helps.  
>> Unfortunately, I don't have much experience with the memcache, but I  
>> have a strong feeling that you shouldn't be seeing such a high 
>> number  of alloc failures for sending..
>>
>> -- 
>> Mike Garrison
>> _______________________________________________
>> OpenAFS-info mailing list
>> OpenAFS-info@openafs.org
>> https://lists.openafs.org/mailman/listinfo/openafs-info
>
>
> Well... The alloc-failures went away, and the
> noBuffers count is at 0. But the behavior didn't change
> initially at -rxpck 2000
>
>   502 afs_cv_wait
>   510 afs_osi_SleepSig
>   512 afs_osi_SleepSig
>   514 afs_osi_SleepSig
>   516 afs_osi_SleepSig
>   518 afs_osi_SleepSig
>   520 afs_osi_SleepSig
>  2708 afs_cv_wait
>  2709 afs_cv_wait
>  [...]
>  2904 afs_cv_wait
>  2905 afs_cv_wait
>
>
> So... I cranked -rxpck up to 4000, and the behavior did change.
> I could see it allocating thousands of packets by watching the rxstats,
> so it definitely appears that was the resource I was looking for.
>
> After pushing rxpck up to 10000, the bottleneck has now moved.
> I'm not 100% certain whether it's the back end AFS servers, or
> that I'm saturating what the NIC is capable of, or if I've hit
> the limit on the CPUs. I'm getting over 500Mb/s of bandwidth
> usage, which is plenty... far more than I thought possible.
> This blows the lid off my performance concerns.  And the important
> thing is that things degrade gracefully as expected.
>
> Thanks for all the help! Once I get things fully tuned up,
> I'll post my findings, in case anyone else can use the info.
Hi There,

I admit that the docs are off, but we are working on it. If someone 
could summarize what -rxpck
 does and how it is used, then I'll add it to the man pages.

So correct me if  I'm wrong:

does -rxpck set the number of pre-allocated packet structures available 
for rx packets? When are you likely to need to tweak this?

Sincerely,
Jason