[OpenAFS] Slow loading of virtually hosted web content
Kendrick Hernandez
kendrick.hernandez@umbc.edu
Wed, 10 Nov 2021 15:27:43 -0500
--0000000000009671c405d0750f8a
Content-Type: text/plain; charset="UTF-8"
Hi all,
We host around 240 departmental and campus web sites (individual afs
volumes) across 6 virtual web servers on AFS storage. The web servers are 4
core, 16G VMs, and the 4 file servers are 4 core 32G VMs. All CentOS 7
systems.
In the past week or so, we've encountered high-load on the web servers
(primary consumers being apache and afsd) during periods of increased
traffic, and we're trying to identify ways to tune performance. After
seeing the following in the logs:
2021 11 08 08:52:03 -05:00 virthost4 [kern.warning] kernel: afs: Warning:
> We are having trouble keeping the AFS stat cache trimmed down under the
> configured limit (current -stat setting: 3000, current vcache usage: 18116).
> 2021 11 08 08:52:03 -05:00 virthost4 [kern.warning] kernel: afs: If AFS
> access seems slow, consider raising the -stat setting for afsd.
I increased the disk cache to 10g and the -stat parameter to 100000, which
has improved things somewhat, but we're not quite there yet. This is the
current client cache configuration from one of the web servers:
Chunk files: 281250
> Stat caches: 100000
> Data caches: 10000
> Volume caches: 200
> Chunk size: 1048576
> Cache size: 9000000 kB
> Set time: no
> Cache type: disk
Has anyone else experienced this? I think the bottleneck is with the cache
manager and not the file servers themselves, because they don't seem to be
impacted much during those periods of high load, and I can access files in
those web volumes from my local client without any noticable lag.
Any guidance on what to look at regarding performance would be much
appreciated.
Thank you!
k-
--
Kendrick Hernandez
*UNIX Systems Administrator*
Division of Information Technology
University of Maryland, Baltimore County
--0000000000009671c405d0750f8a
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hi all,<div><br></div><div>We host around 240 departmental=
and campus web sites (individual afs volumes) across 6 virtual web servers=
on AFS storage. The web servers are 4 core, 16G VMs, and the 4 file server=
s are 4 core 32G VMs. All CentOS 7 systems. <br></div><div><br></div><div>I=
n the past week or so, we've encountered high-load on the web servers (=
primary consumers being apache and afsd) during periods of increased traffi=
c, and we're trying to identify ways to tune performance. After seeing =
the following in the logs:</div><div><br></div><blockquote class=3D"gmail_q=
uote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2=
04);padding-left:1ex"><span style=3D"font-family:monospace">2021 11 08 08:5=
2:03 -05:00 virthost4 [kern.warning] kernel: afs: Warning: We are having tr=
ouble keeping the AFS stat cache trimmed down under the configured limit (c=
urrent -stat setting: 3000, current vcache usage: 18116).</span><br><span s=
tyle=3D"font-family:monospace">2021 11 08 08:52:03 -05:00 virthost4 [kern.w=
arning] kernel: afs: If AFS access seems slow, consider raising the -stat s=
etting for afsd.</span></blockquote><div><br></div><div>I increased the dis=
k cache to 10g and the -stat parameter to 100000, which has improved things=
somewhat, but we're not quite there yet. This is the current client ca=
che configuration from one of the web servers:<br></div><div><br></div><blo=
ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left=
:1px solid rgb(204,204,204);padding-left:1ex"><div><span style=3D"font-fami=
ly:monospace">Chunk files: =C2=A0 281250<br>Stat caches: =C2=A0 100000</spa=
n></div><div><span style=3D"font-family:monospace"></span></div><span style=
=3D"font-family:monospace">Data caches: =C2=A0 10000</span><br><span style=
=3D"font-family:monospace">Volume caches: 200</span><br><span style=3D"font=
-family:monospace">Chunk size: =C2=A0 =C2=A01048576</span><br><span style=
=3D"font-family:monospace">Cache size: =C2=A0 =C2=A09000000 kB</span><br><s=
pan style=3D"font-family:monospace">Set time: =C2=A0 =C2=A0 =C2=A0no</span>=
<br><span style=3D"font-family:monospace">Cache type: =C2=A0 =C2=A0disk</sp=
an></blockquote><div><br></div><div>Has anyone else experienced this? I thi=
nk the bottleneck is with the cache manager and not the file servers themse=
lves, because they don't seem to be impacted much during those periods =
of high load, and I can access files in those web volumes from my local cli=
ent without any noticable lag. <br></div><div><br></div><div>Any guidance o=
n what to look at regarding performance would be much appreciated. <br></di=
v><div><br></div><div>Thank you!</div><div>k-<br></div><div><br>-- <br><div=
dir=3D"ltr" class=3D"gmail_signature" data-smartmail=3D"gmail_signature"><=
div dir=3D"ltr">Kendrick Hernandez<br><i>UNIX Systems Administrator</i><br>=
Division of Information Technology<br><span style=3D"font-family:georgia,se=
rif">University of Maryland, Baltimore County</span><br></div></div></div><=
/div>
--0000000000009671c405d0750f8a--