[OpenAFS] Re: Client Cache Question
Anne Salemme
Anne Salemme <anne@salemme.net>
Mon, 24 Jun 2013 17:50:12 -0700 (PDT)
--790511012-1551707293-1372121412=:34016
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
what about the cache? how big is it, and is it on its own disk partition?=
=0A=0Aanne=0A=0A=0A=0A________________________________=0A From: Timothy Bal=
cer <timothy@telmate.com>=0ATo: Andrew Deason <adeason@sinenomine.net> =0AC=
c: openafs-info@openafs.org =0ASent: Monday, June 24, 2013 6:58 PM=0ASubjec=
t: Re: [OpenAFS] Re: Client Cache Question=0A =0A=0A=0AThanks in advance fo=
r your help and patience :)=0A=0AThis particular client is:=0A=0A=0A=09* op=
enafs-client-1.6.2-1.el5.x86_64=0AThe OS is:=0A=0A=0A=0A=09* Linux xxx.xxx.=
net 2.6.18-164.15.1.el5xen #1 SMP Wed Mar 17 12:04:23 EDT 2010 x86_64 x86_6=
4 x86_64 GNU/Linux (it is a dom0, but is running no VMs)=0A=0A=09* CentOS 5=
.4=0A=0AThe server is:=0A=0A=0A=09* openafs-fileserver 1.6.1-1 x86_64 on Ub=
untu=0A=0AServer OS is:=0A=0A=0A=09* 3.2.0-29-generic #46-Ubuntu SMP=0A=09*=
Ubuntu 12.04 LTS=0AI'd like to repeal my earlier data.. turns out I didn't=
wait long enough...=0A=0AThe behavior that is repeatable is this:=0A=0A=0A=
=0A=09* Soon after client restart, rsync is very fast.. less than a second,=
compared to rsync modules at 3-5 seconds=0A=09* Then, immediately, or afte=
r a few iterations, it slows down to 40+ seconds. It stays this way for the=
duration (days, so far. no change).=0A=09* Rsync times to rsync modules on=
the same destination host do not change.=0A=09* The amount of data is smal=
l, as is the number of files (100k or less per file, and 100 or so files ea=
ch time)=0A=09* The files are always new. They are not maintained on AFS, t=
hey are sync'd TO AFS from a standard file system. They are never there alr=
eady.=0A=0A=09* Network speeds are good=0A=0A=0A=0A=0A=0A=0A=0AOn Mon, Jun =
24, 2013 at 1:39 PM, Andrew Deason <adeason@sinenomine.net> wrote:=0A=0AOn =
Fri, 21 Jun 2013 16:26:22 -0700=0A>Timothy Balcer <timothy@telmate.com> wro=
te:=0A>=0A>> This seems counter intuitive... the 100 or so files do not go =
over the=0A>> 500,000 block cache size. They are fairly small (10's to 100'=
s of=0A>> kilobytes). Why would increasing cache size impact performance=0A=
>> Negatively in such a case?=0A>=0A>When you say 500,000 or 50,000, etc, y=
ou mean 50,000... KiB? So, a=0A>500MiB vs 50MiB cache? About how big is the=
entire amount of data pushed=0A>to AFS compared to the cache size?=0A>=0A>=
Anyway, one _guess_ as to why a larger cache may be slower for that is=0A>t=
hat you're invalidating/overwriting a larger amount of data in the=0A>cache=
. That is, for the 50M cache, you're writing and overwriting <=3D50M=0A>of =
data on disk; for the 500M cache, you're writing and ovewriting >50M=0A>of =
data, possibly all over the disk as we kick out different things from=0A>th=
e cache. If we're limited to overwriting 50M of disk data, the disk=0A>i/o =
may perform better since our i/o is able to stay inside various=0A>caches a=
t lower levels (OS page cache, disk or controller caches, etc).=0A>If you'r=
e not actually using the cached data, the cache can easily be a=0A>hindranc=
e to performance, and a larger cache can make that worse.=0A>=0A>That's jus=
t a guess, but I think it's one way you could see the larger=0A>cache seem =
to perform more slowly. If you want to get more information,=0A>you could r=
un fstrace while the copies are running and provide that. And=0A>as Jeffrey=
said, details of the platforms and versions in question would=0A>be useful=
to have, though as I recall, you are running Linux. The=0A>filesystems in =
use could be useful to know, too.=0A>=0A>--=0A>Andrew Deason=0A>adeason@sin=
enomine.net=0A>=0A>_______________________________________________=0A>OpenA=
FS-info mailing list=0A>OpenAFS-info@openafs.org=0A>https://lists.openafs.o=
rg/mailman/listinfo/openafs-info=0A>=0A=0A=0A-- =0ATimothy Balcer / IT Serv=
ices=0ATelmate / San Francisco, CA=0ADirect / (415) 300-4313=0ACustomer Ser=
vice /=A0(800) 205-5510
--790511012-1551707293-1372121412=:34016
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"color:#000; background-color:#fff; font-family:ti=
mes new roman, new york, times, serif;font-size:12pt"><div><span>what about=
the cache? how big is it, and is it on its own disk partition?</span></div=
><div style=3D"color: rgb(0, 0, 0); font-size: 16px; font-family: 'times ne=
w roman', 'new york', times, serif; background-color: transparent; font-sty=
le: normal;"><span><br></span></div><div style=3D"color: rgb(0, 0, 0); font=
-size: 16px; font-family: 'times new roman', 'new york', times, serif; back=
ground-color: transparent; font-style: normal;"><span>anne</span></div><div=
style=3D"color: rgb(0, 0, 0); font-size: 16px; font-family: 'times new rom=
an', 'new york', times, serif; background-color: transparent; font-style: n=
ormal;"><span><br></span></div><div><br></div> <div style=3D"font-family: =
'times new roman', 'new york', times, serif; font-size: 12pt;"> <div style=
=3D"font-family: 'times new roman', 'new york', times, serif; font-size: 12=
pt;">
<div dir=3D"ltr"> <hr size=3D"1"> <font size=3D"2" face=3D"Arial"> <b><sp=
an style=3D"font-weight:bold;">From:</span></b> Timothy Balcer <timothy@=
telmate.com><br> <b><span style=3D"font-weight: bold;">To:</span></b> An=
drew Deason <adeason@sinenomine.net> <br><b><span style=3D"font-weigh=
t: bold;">Cc:</span></b> openafs-info@openafs.org <br> <b><span style=3D"fo=
nt-weight: bold;">Sent:</span></b> Monday, June 24, 2013 6:58 PM<br> <b><sp=
an style=3D"font-weight: bold;">Subject:</span></b> Re: [OpenAFS] Re: Clien=
t Cache Question<br> </font> </div> <div class=3D"y_msg_container"><br>=0A<=
div id=3D"yiv1330811422"><div dir=3D"ltr"><div><div><div><div><div><div>Tha=
nks in advance for your help and patience :)<br><br>This particular client =
is:<br><br><ul><li>openafs-client-1.6.2-1.el5.x86_64</li></ul><br></div>The=
OS is:<br></div><div><br>=0A<ul><li>Linux <a rel=3D"nofollow" target=3D"_b=
lank" href=3D"http://xxx.xxx.net/">xxx.xxx.net</a> 2.6.18-164.15.1.el5xen #=
1 SMP Wed Mar 17 12:04:23 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux (it is a =
dom0, but is running no VMs)<br></li><li>CentOS 5.4</li></ul></div>=0A<br><=
/div><div>The server is:<br><br></div><div><ul><li>openafs-fileserver 1.6.1=
-1 x86_64 on Ubuntu</li></ul><br></div><div>Server OS is:<br><br><ul><li>3.=
2.0-29-generic #46-Ubuntu SMP</li><li>Ubuntu 12.04 LTS</li></ul></div>=0A<d=
iv><br></div>I'd like to repeal my earlier data.. turns out I didn't wait l=
ong enough...<br><br></div>The behavior that is repeatable is this:<br></di=
v><div><br><ul><li>Soon after client restart, rsync is very fast.. less tha=
n a second, compared to rsync modules at 3-5 seconds</li>=0A<li>Then, immed=
iately, or after a few iterations, it slows down to 40+ seconds. It stays t=
his way for the duration (days, so far. no change).</li><li>Rsync times to =
rsync modules on the same destination host do not change.</li>=0A<li>The am=
ount of data is small, as is the number of files (100k or less per file, an=
d 100 or so files each time)</li><li>The files are always new. They are not=
maintained on AFS, they are sync'd TO AFS from a standard file system. The=
y are never there already.<br>=0A</li><li>Network speeds are good<br></li><=
/ul></div></div><br><br><br></div><div class=3D"yiv1330811422gmail_extra"><=
br><br><div class=3D"yiv1330811422gmail_quote">On Mon, Jun 24, 2013 at 1:39=
PM, Andrew Deason <span dir=3D"ltr"><<a rel=3D"nofollow" ymailto=3D"mai=
lto:adeason@sinenomine.net" target=3D"_blank" href=3D"mailto:adeason@sineno=
mine.net">adeason@sinenomine.net</a>></span> wrote:<br>=0A<blockquote cl=
ass=3D"yiv1330811422gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px=
#ccc solid;padding-left:1ex;"><div class=3D"yiv1330811422im">On Fri, 21 Ju=
n 2013 16:26:22 -0700<br>=0ATimothy Balcer <<a rel=3D"nofollow" ymailto=
=3D"mailto:timothy@telmate.com" target=3D"_blank" href=3D"mailto:timothy@te=
lmate.com">timothy@telmate.com</a>> wrote:<br>=0A<br>=0A> This seems =
counter intuitive... the 100 or so files do not go over the<br>=0A> 500,=
000 block cache size. They are fairly small (10's to 100's of<br>=0A> ki=
lobytes). Why would increasing cache size impact performance<br>=0A> Neg=
atively in such a case?<br>=0A<br>=0A</div>When you say 500,000 or 50,000, =
etc, you mean 50,000... KiB? So, a<br>=0A500MiB vs 50MiB cache? About how b=
ig is the entire amount of data pushed<br>=0Ato AFS compared to the cache s=
ize?<br>=0A<br>=0AAnyway, one _guess_ as to why a larger cache may be slowe=
r for that is<br>=0Athat you're invalidating/overwriting a larger amount of=
data in the<br>=0Acache. That is, for the 50M cache, you're writing and ov=
erwriting <=3D50M<br>=0Aof data on disk; for the 500M cache, you're writ=
ing and ovewriting >50M<br>=0Aof data, possibly all over the disk as we =
kick out different things from<br>=0Athe cache. If we're limited to overwri=
ting 50M of disk data, the disk<br>=0Ai/o may perform better since our i/o =
is able to stay inside various<br>=0Acaches at lower levels (OS page cache,=
disk or controller caches, etc).<br>=0AIf you're not actually using the ca=
ched data, the cache can easily be a<br>=0Ahindrance to performance, and a =
larger cache can make that worse.<br>=0A<br>=0AThat's just a guess, but I t=
hink it's one way you could see the larger<br>=0Acache seem to perform more=
slowly. If you want to get more information,<br>=0Ayou could run fstrace w=
hile the copies are running and provide that. And<br>=0Aas Jeffrey said, de=
tails of the platforms and versions in question would<br>=0Abe useful to ha=
ve, though as I recall, you are running Linux. The<br>=0Afilesystems in use=
could be useful to know, too.<br>=0A<span class=3D"yiv1330811422HOEnZb"><f=
ont color=3D"#888888"><br>=0A--<br>=0AAndrew Deason<br>=0A<a rel=3D"nofollo=
w" ymailto=3D"mailto:adeason@sinenomine.net" target=3D"_blank" href=3D"mail=
to:adeason@sinenomine.net">adeason@sinenomine.net</a><br>=0A<br>=0A________=
_______________________________________<br>=0AOpenAFS-info mailing list<br>=
=0A<a rel=3D"nofollow" ymailto=3D"mailto:OpenAFS-info@openafs.org" target=
=3D"_blank" href=3D"mailto:OpenAFS-info@openafs.org">OpenAFS-info@openafs.o=
rg</a><br>=0A<a rel=3D"nofollow" target=3D"_blank" href=3D"https://lists.op=
enafs.org/mailman/listinfo/openafs-info">https://lists.openafs.org/mailman/=
listinfo/openafs-info</a><br>=0A</font></span></blockquote></div><br><br cl=
ear=3D"all"><br>-- <br><span style=3D"border-collapse: collapse; color: rgb=
(102, 102, 102); font-family: verdana, sans-serif; font-size: x-small;">Tim=
othy Balcer / IT Services<br>Telmate / San Francisco, CA<br>=0ADirect / </s=
pan><span style=3D"border-collapse: collapse; font-family: verdana, sans-se=
rif; font-size: x-small;"><font color=3D"#1155cc">(415) 300-4313</font><br>=
<font color=3D"#666666">Customer Service / </font><a rel=3D"nofollow" =
style=3D"color:rgb(17,85,204);" href=3D"">(800) 205-5510</a></span>=0A</div=
>=0A</div><br><br></div> </div> </div> </div></body></html>
--790511012-1551707293-1372121412=:34016--