[OpenAFS] Re: Openafs performance with Apache/PHP

[Nate Gordon]

Current args to afsd:
AFSD_ARGS="-fakestat-all -chunksize 20 -volumes 400 -dcache 50000
-files 200000 -stat 150000 -daemons 20"

I believe the default chunksize is 16 or 18.  I might try going back
to that tomorrow when I get back into the office.  The majority of my
files are pretty small.

On 8/13/07, Jason Edgecombe <jason@rampaginggeek.com> wrote:
> Hi,
>
> Have you tried tinkering with the chunksize value? It's specified when
> afsd is run and tells the client how big of a chunk of each file to request.
>
> Jason
>
> Nate Gordon wrote:
> > So after having my server having yet another round of fits I'm going
> > to bring this back up.  My cache is currently hanging around 26GB
> > used, which makes some sense since I've probably got ~35GB of actual
> > content.  Is this too large of a cache?  I know previous versions of
> > openafs had issues with this.
> >
> > I would be willing to try devel versions of things for testing.  I
> > would actually like to, except that I'm not a kernel genious and the
> > last time I tried 1.5 (1.5.21) I couldn't get it to compile against a
> > rhel5 kernel.
> >
> > Is there any way to tune the afs client to deal better with lots and
> > lots of small file operations?
> >
> > With the start of classes coming up soon this will mostly likely cause
> > my server to do very bad things on a very regular basis.  I'm willing
> > to contemplate any idea at least once.
> >
> > On 7/27/07, Nate Gordon <nlgordon@gmail.com> wrote:
> >
> >> So I'm finally getting around to querying about some issues I've been
> >> having for a long time with openafs 1.4.4 on rhel (3|4|5).  In short,
> >> the performance of my webservers serving up PHP code is less than
> >> stellar in some cases.  Those cases being defined as the requests per
> >> second getting above 30.  Here is an example/summary of the issue I'm
> >> having:
> >>
> >> I have a php page, that php page includes some other php files and
> >> makes two database queries.  If I point apache bench at that page on
> >> one of my test servers I get results as follows:
> >>
> >> 1 concurrent user: 16 requests per second 60ms per request
> >> 2 concurrent users: 27 requests per second 73ms per request
> >> 5 concurrent users: 32 requests per second 157ms per request
> >> 10 concurrent users: 10 requests per second 934ms per request
> >> 20 concurrent users: 5 requests per second 3666ms per request
> >> 30 concurrent users: 3 requests per second 10899 ms per request
> >>
> >> * requests per second being defined as the aggregate requests per
> >> second across all the concurrent users.  Thus 3 users at 3 requests
> >> per second each yields 9 requests per second aggregate.
> >>
> >> So as you can see performance drops off to an unusable point.  What I
> >> also noted was that while watching vmstat the number of context
> >> switches soars through the roof.  At the 10 users mark the switches
> >> hovers around ~200,000 per second.  At 20 users we go to ~250,000 and
> >> at 30 users we hit ~270,000 per second.  The machine I'm testing on is
> >> a Dell dual xeon 2.8Ghz hyperthreaded machine with 2GB of ram.  The
> >> machine isn't swapping during any of this and general memory usage
> >> remains fairly constant.  Afsd is configured with:
> >> AFSD_ARGS="-fakestat-all -chunksize 20 -volumes 400 -dcache 50000
> >> -files 200000 -stat 150000 -daemons 20"
> >>
> >> I've tried varying the various options to this to try to improve
> >> performance, but things only seem to get worse.  Cache size on this
> >> machine is currently 100MB and the total content being served is <
> >> 1MB.
> >>
> >> For completeness I've also tested this setup on a similar machine
> >> running rhel 5 with arla-current and have come up with some
> >> interesting results:
> >>
> >> 1 user: 22 requests per second 43ms per request
> >> 2 users: 32 requests per second 62ms per request
> >> 5 users: 36 requests per second 138ms per request
> >> 10 users: 36 requests per second 274ms per request
> >> 20 users: 37 requests per second 542ms per request
> >> 30 users: 36 requests per second 821ms per request
> >>
> >> This shows me exactly what I would expect to happen in on the server.
> >> Aggregate requests per second reach a peak performance level which
> >> increases the individual request time.
> >>
> >> Before anyone gets offended by anything I might have said, I want to
> >> make it absolutely clear that I am not trying to say that either
> >> openafs or arla is better/worse than the other.  I needed something to
> >> compare to for the purposes of this discussion.  I realize that the
> >> two are very very differently architected in their interactions with
> >> the kernel.  I would like to stick with openafs on my servers since
> >> that has official support from my systems group.
> >>
> >> So then I start to wonder about those differences in architecture.  Is
> >> there a point in openafs where it is essentially being single
> >> threaded?  Possibly in cache reading/validation?  Varying the number
> >> of daemons I'm running doesn't appear to affect the performance
> >> greatly until I drop down to something like 1 daemon.
> >>
> >> A little bit more about the PHP code I'm running and about PHP in
> >> general.  When you do a require_once in PHP to include a file it
> >> essentially stats every directory component between / and the full
> >> path to the file it is trying to include.  It does this on every
> >> include and does not cache the results.  It also does this in an
> >> attempt to locate the file if the include was done as a relative path,
> >> ie.:
> >> CWD="/afs/cell/folder/virtualhost/directory/page.php"
> >> include_path = ".:/usr/local/lib/php/:/afs/cell/folder/where/my/includes/are/"
> >> require_once("relative/path/include.inc.php");
> >>
> >> This would generate stat system calls in approximately this pattern:
> >> Check first item in include_path for include ("."):
> >> /
> >> /afs
> >> /afs/cell
> >> /afs/cell/folder
> >> /afs/cell/folder/virtualhost
> >> /afs/cell/folder/virtualhost/directory
> >> /afs/cell/folder/virtualhost/directory/relative <-- Fails since node
> >> doesn't exist
> >> Check second item in include_path for include ("/usr/local/lib/php/"):
> >> /
> >> /usr
> >> /usr/local
> >> /usr/local/lib
> >> /usr/local/lib/php
> >> /usr/local/lib/php/relative <-- Fails again for same reason
> >> Check third item in include_path ("/afs/cell/folder/where/my/includes/are"):
> >> /
> >> /afs
> >> /afs/cell
> >> /afs/cell/folder
> >> /afs/cell/folder/where
> >> /afs/cell/folder/where/my
> >> /afs/cell/folder/where/my/includes
> >> /afs/cell/folder/where/my/includes/are
> >> /afs/cell/folder/where/my/includes/are/relative
> >> /afs/cell/folder/where/my/includes/are/relative/path
> >> /afs/cell/folder/where/my/includes/are/relative/path/include.inc.php
> >> <-- Actual fopen
> >>
> >> Everything except the final fopen are lstat64 calls as indicated by strace.
> >>
> >> As you can see this is pretty expensive to do per page hit, especially
> >> when this is done repeatedly for every included file.  The page in
> >> question includes about 20 or so files.  This is one problem.  I've
> >> also put together the same page by pulling in all the code that would
> >> have been included to generate a self contained php file.  The
> >> performance tops out around 50 requests per second and still has the
> >> same dropoff pattern as you increase the number of simultaneous users
> >> hitting the page when using openafs.  So the problem isn't purley in
> >> the excessive amount of stat system calls.  If anything it shows that
> >> there is a problem somewhere else as well.  Possibly that it requires
> >> that the openafs daemon threads be switched in to answer lots of
> >> questions too often or that the threads have to talk to each other for
> >> every query to the afs file system.
> >>
> >> This topic of discussion might be more suited to the dev list, but I
> >> thought I would start here to see if anyone else was serving php files
> >> out of AFS and could reproduce my problems in their environment.
> >>
> >> If any more information is needed I would be glad to provide it.
> >>
> >> Thanks in advance,
> >>
> >> --
> >> -Nathan Gordon
> >>
> >> If the database server goes down and there is no code to hear it, does
> >> it really go down?
> >> <esc>:wq<CR>
> >>
> >>
> >
> >
> >
>
>


-- 
-Nathan Gordon

If the database server goes down and there is no code to hear it, does
it really go down?
<esc>:wq<CR>