[OpenAFS] Re: Client Cache Question
Timothy Balcer
timothy@telmate.com
Tue, 25 Jun 2013 11:46:46 -0700
--047d7bdc13da7a27ef04dffef33e
Content-Type: text/plain; charset=ISO-8859-1
It is small (50M) and is not on its own partition.
This is a tiny amount of data being rsync'd. just a few megabytes per
rsync. and a small amount of files. The link is 100Mbit, and standard rsync
finishes to the same destination point in a few seconds.
The behavior is what concerns me, and seems to point to something going on
with the cache, or the client. It doesn't seem to matter if the cache is
large or small. And the underlying I/O on that filesyetem is minimal. This
is a very lightly used system, which is why we are testing this process
from there.
On Mon, Jun 24, 2013 at 5:50 PM, Anne Salemme <anne@salemme.net> wrote:
> what about the cache? how big is it, and is it on its own disk partition?
>
> anne
>
>
> ------------------------------
> *From:* Timothy Balcer <timothy@telmate.com>
> *To:* Andrew Deason <adeason@sinenomine.net>
> *Cc:* openafs-info@openafs.org
> *Sent:* Monday, June 24, 2013 6:58 PM
> *Subject:* Re: [OpenAFS] Re: Client Cache Question
>
> Thanks in advance for your help and patience :)
>
> This particular client is:
>
>
> - openafs-client-1.6.2-1.el5.x86_64
>
>
> The OS is:
>
>
> - Linux xxx.xxx.net 2.6.18-164.15.1.el5xen #1 SMP Wed Mar 17 12:04:23
> EDT 2010 x86_64 x86_64 x86_64 GNU/Linux (it is a dom0, but is running no
> VMs)
> - CentOS 5.4
>
>
> The server is:
>
>
> - openafs-fileserver 1.6.1-1 x86_64 on Ubuntu
>
>
> Server OS is:
>
>
> - 3.2.0-29-generic #46-Ubuntu SMP
> - Ubuntu 12.04 LTS
>
>
> I'd like to repeal my earlier data.. turns out I didn't wait long enough...
>
> The behavior that is repeatable is this:
>
>
> - Soon after client restart, rsync is very fast.. less than a second,
> compared to rsync modules at 3-5 seconds
> - Then, immediately, or after a few iterations, it slows down to 40+
> seconds. It stays this way for the duration (days, so far. no change).
> - Rsync times to rsync modules on the same destination host do not
> change.
> - The amount of data is small, as is the number of files (100k or less
> per file, and 100 or so files each time)
> - The files are always new. They are not maintained on AFS, they are
> sync'd TO AFS from a standard file system. They are never there already.
> - Network speeds are good
>
>
>
>
>
>
> On Mon, Jun 24, 2013 at 1:39 PM, Andrew Deason <adeason@sinenomine.net>wrote:
>
> On Fri, 21 Jun 2013 16:26:22 -0700
> Timothy Balcer <timothy@telmate.com> wrote:
>
> > This seems counter intuitive... the 100 or so files do not go over the
> > 500,000 block cache size. They are fairly small (10's to 100's of
> > kilobytes). Why would increasing cache size impact performance
> > Negatively in such a case?
>
> When you say 500,000 or 50,000, etc, you mean 50,000... KiB? So, a
> 500MiB vs 50MiB cache? About how big is the entire amount of data pushed
> to AFS compared to the cache size?
>
> Anyway, one _guess_ as to why a larger cache may be slower for that is
> that you're invalidating/overwriting a larger amount of data in the
> cache. That is, for the 50M cache, you're writing and overwriting <=50M
> of data on disk; for the 500M cache, you're writing and ovewriting >50M
> of data, possibly all over the disk as we kick out different things from
> the cache. If we're limited to overwriting 50M of disk data, the disk
> i/o may perform better since our i/o is able to stay inside various
> caches at lower levels (OS page cache, disk or controller caches, etc).
> If you're not actually using the cached data, the cache can easily be a
> hindrance to performance, and a larger cache can make that worse.
>
> That's just a guess, but I think it's one way you could see the larger
> cache seem to perform more slowly. If you want to get more information,
> you could run fstrace while the copies are running and provide that. And
> as Jeffrey said, details of the platforms and versions in question would
> be useful to have, though as I recall, you are running Linux. The
> filesystems in use could be useful to know, too.
>
> --
> Andrew Deason
> adeason@sinenomine.net
>
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>
>
>
>
> --
> Timothy Balcer / IT Services
> Telmate / San Francisco, CA
> Direct / (415) 300-4313
> Customer Service / (800) 205-5510
>
>
>
--
Timothy Balcer / IT Services
Telmate / San Francisco, CA
Direct / (415) 300-4313
Customer Service / (800) 205-5510
--047d7bdc13da7a27ef04dffef33e
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div><div>It is small (50M) and is not on its own partitio=
n.<br><br></div>This is a tiny amount of data being rsync'd. just a few=
megabytes per rsync. and a small amount of files. The link is 100Mbit, and=
standard rsync finishes to the same destination point in a few seconds.<br=
>
<br></div>The behavior is what concerns me, and seems to point to something=
going on with the cache, or the client. It doesn't seem to matter if t=
he cache is large or small. And the underlying I/O on that filesyetem is mi=
nimal. This is a very lightly used system, which is why we are testing this=
process from there.<br>
</div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Mon,=
Jun 24, 2013 at 5:50 PM, Anne Salemme <span dir=3D"ltr"><<a href=3D"mai=
lto:anne@salemme.net" target=3D"_blank">anne@salemme.net</a>></span> wro=
te:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div><div style=3D"font-size:12pt;font-famil=
y:times new roman,new york,times,serif"><div><span>what about the cache? ho=
w big is it, and is it on its own disk partition?</span></div>
<div style=3D"font-style:normal;font-size:16px;background-color:transparent=
;font-family:'times new roman','new york',times,serif"><spa=
n><br></span></div><div style=3D"font-style:normal;font-size:16px;backgroun=
d-color:transparent;font-family:'times new roman','new york'=
;,times,serif">
<span>anne</span></div><div style=3D"font-style:normal;font-size:16px;backg=
round-color:transparent;font-family:'times new roman','new york=
',times,serif"><span><br></span></div><div><br></div> <div style=3D"fo=
nt-family:'times new roman','new york',times,serif;font-siz=
e:12pt">
<div style=3D"font-family:'times new roman','new york',tim=
es,serif;font-size:12pt">
<div dir=3D"ltr"> <hr size=3D"1"> <font face=3D"Arial"> <b><span style=3D=
"font-weight:bold">From:</span></b> Timothy Balcer <<a href=3D"mailto:ti=
mothy@telmate.com" target=3D"_blank">timothy@telmate.com</a>><br> <b><sp=
an style=3D"font-weight:bold">To:</span></b> Andrew Deason <<a href=3D"m=
ailto:adeason@sinenomine.net" target=3D"_blank">adeason@sinenomine.net</a>&=
gt; <br>
<b><span style=3D"font-weight:bold">Cc:</span></b> <a href=3D"mailto:openaf=
s-info@openafs.org" target=3D"_blank">openafs-info@openafs.org</a> <br> <b>=
<span style=3D"font-weight:bold">Sent:</span></b> Monday, June 24, 2013 6:5=
8 PM<br>
<b><span style=3D"font-weight:bold">Subject:</span></b> Re: [OpenAFS] Re: =
Client Cache Question<br> </font> </div><div><div class=3D"h5"> <div><br>
<div><div dir=3D"ltr"><div><div><div><div><div><div>Thanks in advance for y=
our help and patience :)<br><br>This particular client is:<br><br><ul><li>o=
penafs-client-1.6.2-1.el5.x86_64</li></ul><br></div>The OS is:<br></div>
<div><br>
<ul><li>Linux <a rel=3D"nofollow" href=3D"http://xxx.xxx.net/" target=3D"_b=
lank">xxx.xxx.net</a> 2.6.18-164.15.1.el5xen #1 SMP Wed Mar 17 12:04:23 EDT=
2010 x86_64 x86_64 x86_64 GNU/Linux (it is a dom0, but is running no VMs)<=
br>
</li><li>CentOS 5.4</li></ul></div>
<br></div><div>The server is:<br><br></div><div><ul><li>openafs-fileserver =
1.6.1-1 x86_64 on Ubuntu</li></ul><br></div><div>Server OS is:<br><br><ul><=
li>3.2.0-29-generic #46-Ubuntu SMP</li><li>Ubuntu 12.04 LTS</li></ul></div>
<div><br></div>I'd like to repeal my earlier data.. turns out I didn=
9;t wait long enough...<br><br></div>The behavior that is repeatable is thi=
s:<br></div><div><br><ul><li>Soon after client restart, rsync is very fast.=
. less than a second, compared to rsync modules at 3-5 seconds</li>
<li>Then, immediately, or after a few iterations, it slows down to 40+ seco=
nds. It stays this way for the duration (days, so far. no change).</li><li>=
Rsync times to rsync modules on the same destination host do not change.</l=
i>
<li>The amount of data is small, as is the number of files (100k or less pe=
r file, and 100 or so files each time)</li><li>The files are always new. Th=
ey are not maintained on AFS, they are sync'd TO AFS from a standard fi=
le system. They are never there already.<br>
</li><li>Network speeds are good<br></li></ul></div></div><br><br><br></div=
><div><br><br><div>On Mon, Jun 24, 2013 at 1:39 PM, Andrew Deason <span dir=
=3D"ltr"><<a rel=3D"nofollow" href=3D"mailto:adeason@sinenomine.net" tar=
get=3D"_blank">adeason@sinenomine.net</a>></span> wrote:<br>
<blockquote style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-l=
eft:1ex"><div>On Fri, 21 Jun 2013 16:26:22 -0700<br>
Timothy Balcer <<a rel=3D"nofollow" href=3D"mailto:timothy@telmate.com" =
target=3D"_blank">timothy@telmate.com</a>> wrote:<br>
<br>
> This seems counter intuitive... the 100 or so files do not go over the=
<br>
> 500,000 block cache size. They are fairly small (10's to 100's=
of<br>
> kilobytes). Why would increasing cache size impact performance<br>
> Negatively in such a case?<br>
<br>
</div>When you say 500,000 or 50,000, etc, you mean 50,000... KiB? So, a<br=
>
500MiB vs 50MiB cache? About how big is the entire amount of data pushed<br=
>
to AFS compared to the cache size?<br>
<br>
Anyway, one _guess_ as to why a larger cache may be slower for that is<br>
that you're invalidating/overwriting a larger amount of data in the<br>
cache. That is, for the 50M cache, you're writing and overwriting <=
=3D50M<br>
of data on disk; for the 500M cache, you're writing and ovewriting >=
50M<br>
of data, possibly all over the disk as we kick out different things from<br=
>
the cache. If we're limited to overwriting 50M of disk data, the disk<b=
r>
i/o may perform better since our i/o is able to stay inside various<br>
caches at lower levels (OS page cache, disk or controller caches, etc).<br>
If you're not actually using the cached data, the cache can easily be a=
<br>
hindrance to performance, and a larger cache can make that worse.<br>
<br>
That's just a guess, but I think it's one way you could see the lar=
ger<br>
cache seem to perform more slowly. If you want to get more information,<br>
you could run fstrace while the copies are running and provide that. And<br=
>
as Jeffrey said, details of the platforms and versions in question would<br=
>
be useful to have, though as I recall, you are running Linux. The<br>
filesystems in use could be useful to know, too.<br>
<span><font color=3D"#888888"><br>
--<br>
Andrew Deason<br>
<a rel=3D"nofollow" href=3D"mailto:adeason@sinenomine.net" target=3D"_blank=
">adeason@sinenomine.net</a><br>
<br>
_______________________________________________<br>
OpenAFS-info mailing list<br>
<a rel=3D"nofollow" href=3D"mailto:OpenAFS-info@openafs.org" target=3D"_bla=
nk">OpenAFS-info@openafs.org</a><br>
<a rel=3D"nofollow" href=3D"https://lists.openafs.org/mailman/listinfo/open=
afs-info" target=3D"_blank">https://lists.openafs.org/mailman/listinfo/open=
afs-info</a><br>
</font></span></blockquote></div><br><br clear=3D"all"><br>-- <br><span sty=
le=3D"border-collapse:collapse;color:rgb(102,102,102);font-family:verdana,s=
ans-serif;font-size:x-small">Timothy Balcer / IT Services<br>Telmate / San =
Francisco, CA<br>
Direct / </span><span style=3D"border-collapse:collapse;font-family:verdana=
,sans-serif;font-size:x-small"><font color=3D"#1155cc"><a href=3D"tel:%2841=
5%29%20300-4313" value=3D"+14153004313" target=3D"_blank">(415) 300-4313</a=
></font><br>
<font color=3D"#666666">Customer Service /=A0</font><a rel=3D"nofollow" sty=
le=3D"color:rgb(17,85,204)">(800) 205-5510</a></span>
</div>
</div><br><br></div> </div></div></div> </div> </div></div></blockquote></=
div><br><br clear=3D"all"><br>-- <br><span style=3D"border-collapse:collaps=
e;color:rgb(102,102,102);font-family:verdana,sans-serif;font-size:x-small">=
Timothy Balcer / IT Services<br>
Telmate / San Francisco, CA<br>Direct / </span><span style=3D"border-collap=
se:collapse;font-family:verdana,sans-serif;font-size:x-small"><font color=
=3D"#1155cc">(415) 300-4313</font><br><font color=3D"#666666">Customer Serv=
ice /=A0</font><a value=3D"+18002055510" style=3D"color:rgb(17,85,204)">(80=
0) 205-5510</a></span>
</div>
--047d7bdc13da7a27ef04dffef33e--