[OpenAFS] Re: Client Cache Question
Timothy Balcer
timothy@telmate.com
Mon, 24 Jun 2013 15:58:45 -0700
--089e013a07d0d2682604dfee5a42
Content-Type: text/plain; charset=ISO-8859-1
Thanks in advance for your help and patience :)
This particular client is:
- openafs-client-1.6.2-1.el5.x86_64
The OS is:
- Linux xxx.xxx.net 2.6.18-164.15.1.el5xen #1 SMP Wed Mar 17 12:04:23
EDT 2010 x86_64 x86_64 x86_64 GNU/Linux (it is a dom0, but is running no
VMs)
- CentOS 5.4
The server is:
- openafs-fileserver 1.6.1-1 x86_64 on Ubuntu
Server OS is:
- 3.2.0-29-generic #46-Ubuntu SMP
- Ubuntu 12.04 LTS
I'd like to repeal my earlier data.. turns out I didn't wait long enough...
The behavior that is repeatable is this:
- Soon after client restart, rsync is very fast.. less than a second,
compared to rsync modules at 3-5 seconds
- Then, immediately, or after a few iterations, it slows down to 40+
seconds. It stays this way for the duration (days, so far. no change).
- Rsync times to rsync modules on the same destination host do not
change.
- The amount of data is small, as is the number of files (100k or less
per file, and 100 or so files each time)
- The files are always new. They are not maintained on AFS, they are
sync'd TO AFS from a standard file system. They are never there already.
- Network speeds are good
On Mon, Jun 24, 2013 at 1:39 PM, Andrew Deason <adeason@sinenomine.net>wrote:
> On Fri, 21 Jun 2013 16:26:22 -0700
> Timothy Balcer <timothy@telmate.com> wrote:
>
> > This seems counter intuitive... the 100 or so files do not go over the
> > 500,000 block cache size. They are fairly small (10's to 100's of
> > kilobytes). Why would increasing cache size impact performance
> > Negatively in such a case?
>
> When you say 500,000 or 50,000, etc, you mean 50,000... KiB? So, a
> 500MiB vs 50MiB cache? About how big is the entire amount of data pushed
> to AFS compared to the cache size?
>
> Anyway, one _guess_ as to why a larger cache may be slower for that is
> that you're invalidating/overwriting a larger amount of data in the
> cache. That is, for the 50M cache, you're writing and overwriting <=50M
> of data on disk; for the 500M cache, you're writing and ovewriting >50M
> of data, possibly all over the disk as we kick out different things from
> the cache. If we're limited to overwriting 50M of disk data, the disk
> i/o may perform better since our i/o is able to stay inside various
> caches at lower levels (OS page cache, disk or controller caches, etc).
> If you're not actually using the cached data, the cache can easily be a
> hindrance to performance, and a larger cache can make that worse.
>
> That's just a guess, but I think it's one way you could see the larger
> cache seem to perform more slowly. If you want to get more information,
> you could run fstrace while the copies are running and provide that. And
> as Jeffrey said, details of the platforms and versions in question would
> be useful to have, though as I recall, you are running Linux. The
> filesystems in use could be useful to know, too.
>
> --
> Andrew Deason
> adeason@sinenomine.net
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>
--
Timothy Balcer / IT Services
Telmate / San Francisco, CA
Direct / (415) 300-4313
Customer Service / (800) 205-5510
--089e013a07d0d2682604dfee5a42
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div><div><div><div><div><div>Thanks in advance for your h=
elp and patience :)<br><br>This particular client is:<br><br><ul><li>openaf=
s-client-1.6.2-1.el5.x86_64</li></ul><br></div>The OS is:<br></div><div><br=
>
<ul><li>Linux <a href=3D"http://xxx.xxx.net">xxx.xxx.net</a> 2.6.18-164.15.=
1.el5xen #1 SMP Wed Mar 17 12:04:23 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux=
(it is a dom0, but is running no VMs)<br></li><li>CentOS 5.4</li></ul></di=
v>
<br></div><div>The server is:<br><br></div><div><ul><li>openafs-fileserver =
1.6.1-1 x86_64 on Ubuntu</li></ul><br></div><div>Server OS is:<br><br><ul><=
li>3.2.0-29-generic #46-Ubuntu SMP</li><li>Ubuntu 12.04 LTS</li></ul></div>
<div><br></div>I'd like to repeal my earlier data.. turns out I didn=
9;t wait long enough...<br><br></div>The behavior that is repeatable is thi=
s:<br></div><div><br><ul><li>Soon after client restart, rsync is very fast.=
. less than a second, compared to rsync modules at 3-5 seconds</li>
<li>Then, immediately, or after a few iterations, it slows down to 40+ seco=
nds. It stays this way for the duration (days, so far. no change).</li><li>=
Rsync times to rsync modules on the same destination host do not change.</l=
i>
<li>The amount of data is small, as is the number of files (100k or less pe=
r file, and 100 or so files each time)</li><li>The files are always new. Th=
ey are not maintained on AFS, they are sync'd TO AFS from a standard fi=
le system. They are never there already.<br>
</li><li>Network speeds are good<br></li></ul></div></div><br><br><br></div=
><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Mon, Jun =
24, 2013 at 1:39 PM, Andrew Deason <span dir=3D"ltr"><<a href=3D"mailto:=
adeason@sinenomine.net" target=3D"_blank">adeason@sinenomine.net</a>></s=
pan> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div class=3D"im">On Fri, 21 Jun 2013 16:26:=
22 -0700<br>
Timothy Balcer <<a href=3D"mailto:timothy@telmate.com">timothy@telmate.c=
om</a>> wrote:<br>
<br>
> This seems counter intuitive... the 100 or so files do not go over the=
<br>
> 500,000 block cache size. They are fairly small (10's to 100's=
of<br>
> kilobytes). Why would increasing cache size impact performance<br>
> Negatively in such a case?<br>
<br>
</div>When you say 500,000 or 50,000, etc, you mean 50,000... KiB? So, a<br=
>
500MiB vs 50MiB cache? About how big is the entire amount of data pushed<br=
>
to AFS compared to the cache size?<br>
<br>
Anyway, one _guess_ as to why a larger cache may be slower for that is<br>
that you're invalidating/overwriting a larger amount of data in the<br>
cache. That is, for the 50M cache, you're writing and overwriting <=
=3D50M<br>
of data on disk; for the 500M cache, you're writing and ovewriting >=
50M<br>
of data, possibly all over the disk as we kick out different things from<br=
>
the cache. If we're limited to overwriting 50M of disk data, the disk<b=
r>
i/o may perform better since our i/o is able to stay inside various<br>
caches at lower levels (OS page cache, disk or controller caches, etc).<br>
If you're not actually using the cached data, the cache can easily be a=
<br>
hindrance to performance, and a larger cache can make that worse.<br>
<br>
That's just a guess, but I think it's one way you could see the lar=
ger<br>
cache seem to perform more slowly. If you want to get more information,<br>
you could run fstrace while the copies are running and provide that. And<br=
>
as Jeffrey said, details of the platforms and versions in question would<br=
>
be useful to have, though as I recall, you are running Linux. The<br>
filesystems in use could be useful to know, too.<br>
<span class=3D"HOEnZb"><font color=3D"#888888"><br>
--<br>
Andrew Deason<br>
<a href=3D"mailto:adeason@sinenomine.net">adeason@sinenomine.net</a><br>
<br>
_______________________________________________<br>
OpenAFS-info mailing list<br>
<a href=3D"mailto:OpenAFS-info@openafs.org">OpenAFS-info@openafs.org</a><br=
>
<a href=3D"https://lists.openafs.org/mailman/listinfo/openafs-info" target=
=3D"_blank">https://lists.openafs.org/mailman/listinfo/openafs-info</a><br>
</font></span></blockquote></div><br><br clear=3D"all"><br>-- <br><span sty=
le=3D"border-collapse:collapse;color:rgb(102,102,102);font-family:verdana,s=
ans-serif;font-size:x-small">Timothy Balcer / IT Services<br>Telmate / San =
Francisco, CA<br>
Direct / </span><span style=3D"border-collapse:collapse;font-family:verdana=
,sans-serif;font-size:x-small"><font color=3D"#1155cc">(415) 300-4313</font=
><br><font color=3D"#666666">Customer Service /=A0</font><a value=3D"+18002=
055510" style=3D"color:rgb(17,85,204)">(800) 205-5510</a></span>
</div>
--089e013a07d0d2682604dfee5a42--