[OpenAFS] Improving AFS performance for Joomla and other web sites

[Edgecombe, Jason]

It appears that I wasn't reading things correctly. My 1000 =
requests/second number was for non-2XX responses. :(

Static file from local disk: 158 requests/second
Joomla from local disk: ~50 requests/second
Static file from AFS: 152 requests/second
Joomla from AFS:    20-30 requests/second.


-------------------------------------------------------------------------=
--
Jason Edgecombe | Linux and Solaris Administrator
UNC Charlotte | The William States Lee College of Engineering
9201 University City Blvd. | Charlotte, NC 28223-0001
Phone: 704-687-3514
jwedgeco@uncc.edu | http://www.coe.uncc.edu
-------------------------------------------------------------------------=
--
If you are not the intended recipient of this transmission or a person =
responsible for delivering it to the intended recipient, any disclosure, =
copying, distribution, or other use of any of the information in this =
transmission is strictly prohibited. If you have received this =
transmission in error, please notify me immediately by reply e-mail or =
by telephone at 704-687-3514.=A0 Thank you.


-----Original Message-----
From: Steven Jenkins [mailto:steven.jenkins@gmail.com]=20
Sent: Wednesday, September 08, 2010 6:53 PM
To: Edgecombe, Jason
Cc: openafs-info@openafs.org
Subject: Re: [OpenAFS] Improving AFS performance for Joomla and other =
web sites

On Wed, Sep 8, 2010 at 5:32 PM, Edgecombe, Jason <jwedgeco@uncc.edu> =
wrote:
> Hi everyone,
>
>
>
> We currently server our entire web tree out of AFS. This includes =
Joomla and
> other PHP-based applications. While I was benchmarking our new server, =
I
> noticed that the following rough performance numbers:
>
>
>
> Static files served from AFS: =
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ~200
> requests per second
>
> Joomla home page served out of AFS: =A0 =
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =A0=A0~30 requests per
> second
>
> Joomla home page served from local ext3: =A0=A0=A0=A0=A0=A0=A0=A0 =
=A01000 requests per second.
>
>
>
> I'm running a 8 cores with hyperthreading (16 threads). I'm running =
the
> Apache threaded worker MPM on RHEL5.5 with mod_fcgid and OpenAFS =
1.4.12.1. I
> have Apache configured for ~1000 threads.
>
>
>
> My data (250GB) and stat cache for the AFS client should be big enough =
to
> not have to hit the server.
>
>
>
> My afsd has the following options: " -fakestat -dynroot -chunksize 18 =
-stat
> 500000 -daemons 32 -volumes 2000 -blocks 212713128"
>
>
>
> What is recommended to improve my AFS performance for Joomla?
>

Skipping past a lot of the usual caveats around performance in a
complex system and going (somewhat) straight to the AFS-specific bits:

1- It would be very useful to also compare static files served from
ext3 -- that can give you somewhat of a baseline (i.e., compare the
ratio of static files in ext3 to Joomla in ext3 versus static files in
AFS to Joomla in AFS).  Offhand, the slowdown you're seeing between
static files in AFS vs Joomla in AFS does not seem egregious (based on
my experience tuning other web application servers, but not Joomla
specifically).

2- To see if cache configuration is an issue, look at your hit/miss
rates.  You can use afsmonitor to look at that dynamically.

3- Depending on your ability to do further instrumentation, it would
be very helpful to look at traffic between your web server and AFS
servers (e.g., turn on audit logs) to see exactly what your web server
is requesting from servers.  Alternatively,  you can capture packet
traces with tcpdump/snoop/wireshark.  Don't forget your pt and
vlservers, by the way.  I personally prefer packet traces, as I'm able
to see oddities like bad addresses (and retries) that would not show
up in server logs.

Steven