[OpenAFS] afs memcache tuning... lockups in afs_cv_wait
Sun, 17 Feb 2008 10:14:04 -0500
Mike Polek wrote:
> Mike Garrison wrote:
>> On Feb 13, 2008, at 9:17 PM, Mike Polek wrote:
>>> Mike Garrison wrote:
>> I'd suggest tweaking -rxpck to be higher than the default, it may
>> actually help with the issue you're running into. We use 2000 for
>> it. My mind slips me as to why we picked that number at this point.
>> The only things that really stick out to me is the low number of rx
>> packets, I'd try increasing rxpck and seeing if that helps.
>> Unfortunately, I don't have much experience with the memcache, but I
>> have a strong feeling that you shouldn't be seeing such a high
>> number of alloc failures for sending..
>> Mike Garrison
>> OpenAFS-info mailing list
> Well... The alloc-failures went away, and the
> noBuffers count is at 0. But the behavior didn't change
> initially at -rxpck 2000
> 502 afs_cv_wait
> 510 afs_osi_SleepSig
> 512 afs_osi_SleepSig
> 514 afs_osi_SleepSig
> 516 afs_osi_SleepSig
> 518 afs_osi_SleepSig
> 520 afs_osi_SleepSig
> 2708 afs_cv_wait
> 2709 afs_cv_wait
> 2904 afs_cv_wait
> 2905 afs_cv_wait
> So... I cranked -rxpck up to 4000, and the behavior did change.
> I could see it allocating thousands of packets by watching the rxstats,
> so it definitely appears that was the resource I was looking for.
> After pushing rxpck up to 10000, the bottleneck has now moved.
> I'm not 100% certain whether it's the back end AFS servers, or
> that I'm saturating what the NIC is capable of, or if I've hit
> the limit on the CPUs. I'm getting over 500Mb/s of bandwidth
> usage, which is plenty... far more than I thought possible.
> This blows the lid off my performance concerns. And the important
> thing is that things degrade gracefully as expected.
> Thanks for all the help! Once I get things fully tuned up,
> I'll post my findings, in case anyone else can use the info.
I admit that the docs are off, but we are working on it. If someone
could summarize what -rxpck
does and how it is used, then I'll add it to the man pages.
So correct me if I'm wrong:
does -rxpck set the number of pre-allocated packet structures available
for rx packets? When are you likely to need to tweak this?