[OpenAFS] Re: Apache and Openafs

[Andrew Deason]

On Thu, 26 Dec 2013 07:36:24 -0800 (PST)
Anne Salemme <anne@salemme.net> wrote:

> First thing to check: Is the AFS cache in its own, dedicated
> filesystem partition.

Well, he's using memcache. But anyway:

> From: GALAMBOS Daniel <dancsa@dancsa.hu>
> 
>> So we got problems, i cloned the machine and started testing. I used
>> apache benchmark (ab) against a simple html (w/o pictures or
>> anything, just text). If i downloaded it 2000 times with 50 thread i
>> got high load (18) and ~20 000 context switching in vmstat. The same
>> file with 100 thread generated ~50 load.

To some extent, the performance of accessing stuff in the cache is just
not going to be that great. Right now, I think the linux cache manager
now isn't really architected to serve data from multiple threads
simultaneously very efficiently, and there's not much you can do about
it. If you reduce the number of threads, I would expect that the
response time you get to vastly improve; does it? But regardless, some
things to try:

Remove .htaccess processing in Apache. This can slow down accesses a
lot, especially in AFS. If I recall correctly, you do this with the
"AllowOverride None" directive in Apache's config. Of course, if you
need to use .htaccess files, that doesn't help.

You could try using a disk cache against the local disk instead of
memcache. Memcache may sound "faster", but it's not, really, and disk
cache is a more tested code path. I don't expect that to do much, but if
you're somehow hitting some weird edge case, it's less likely for there
to be obscure issues with disk cache, I think.

You can also try sniffing the network traffic for port 7001 UDP. The
mentioned problems I believe do indicate just bottlenecks in cache
processing, but if you're somehow hitting the net for anything, that
would also be a problem. If you see anything hitting the net from port
7001 UDP on the client besides rx "version" packets, then something is
not getting cached locally.

Oh, and if you can 'strace' a process reading something from /afs while
the Apache server is loaded, it could say what specific syscalls are
being slow. strace'ing Apache itself is probably too much data to look
through, but if you run, say, "cat /afs/path/to/file.html > /dev/null"
and strace'd that, it would be easy to see where things are slowing
down.

-- 
Andrew Deason
adeason@sinenomine.net