[OpenAFS] Re: Advice on a use case
Timothy Balcer
timothy@telmate.com
Fri, 9 Nov 2012 15:06:54 -0800
--20cf3071cd08f4aea804ce1801c8
Content-Type: text/plain; charset=ISO-8859-1
On Fri, Nov 9, 2012 at 1:47 PM, Andrew Deason <adeason@sinenomine.net>wrote:
>
> > Yes.. I understand that. I was commenting on the slowness as compared
> > to rsyncing over NFS, for example, which takes 5 hours for the entire
> > tree when done from the top level of the tree. That tree contains 15
> > of the directories that I mentioned in my earlier post. So 15 * 24k
> > dirs.. and to answer the question, 232,974 files of small size for the
> > one subdirectory in question.
>
> I'm getting a little mixed up about when you're switching from talking
> about the 'NFS solution' vs the 'AFS solution'. Are you saying it took
> 1.5 hours to transfer 15 subdirs into AFS, where it took NFS 5 hours to
> transfer 24000 subdirs?
>
I am not being clear about the numbers I am talking about it looks like...
:)
It takes about 5 hours to transfer/rsync the entire tree, with 15 subdirs,
each containing upwards of 200k files and 24k directories, when I am going
from a local volume to an NFSv4 mount. This NFSv4 mount is on the local
network, so there is not a network bottleneck per se. That's the generic
"fastest" case I am working against.
> > > What can possibly make this faster, then, is copying/creating
> > > files/directories in parallel. <snip>
> >
> > Yes, I routinely run 100's of parallel transfers using a combination
> > of tar and rsync.. tar gets the files over in raw form, and rsync mops
> > up behind. The rsync pass is to correct any problems with the tar
> > copy, and is run twice on a fixed list, generated at transfer time. I
> > have found that even when using a tuned rsync process designed to
> > improve transfer speeds, many parallel tar/untar processes from local
> > to NFSv4 followed by a "local" rsync to the same destination works
> > better for new files, when timeliness is important.
>
> So... are you just talking about the NFS transfers, here?
>
Right, however using tar and rsync to manage the transfers.
>
> With AFS, trying to rsync again afterwards is possibly much slower due
> to cache churn if the number of files in question is larger than the
> stat cache (as Jeff said earlier). You also cannot do more than 4
> simultaneous 'things' to the AFS server with current client releases
> (unless you fiddle with PAGs, or use some AFS-specific tools). So 100s
> of parallel transfers aren't really helpful.
>
Thats excellent information! :) And thanks to you too Jeff.
>
> So if you want to write the tooling for it, I think ideally what you
> would want is a cp/rsync-like tool that would copy files/dirs using a
> separate thread for each file/dir up to some configured limit, tracking
> dependencies so parent dirs are created first, possibly launching new
> processes in separate PAGs as you go. Or, you could use utilities or
> APIs that speak to AFS directly without going through the filesystem
> layer (like afscp/afsio), so you wouldn't need separate threads or
> processes.
>
afsio sounds like the perfect fit for this, actually... although I did have
some trouble with writing with it...I checked manually with ls along the
trees to make sure all tokens were working and etc.
#:~/scripts# cat foo.sh | afsio write -file /afs/.
realmname.com/home/timothy/foo.sh
afsio: No such file or directory (is dir /afs/.realmname.com/home/timothy
in AFS?)
an afsio read succeeds from the RO tree:
#:~/scripts# afsio read -file /afs/
realmname.com/home/timothy/OpenAFS-ServerDos.pdf > foo
#:~/scripts# md5sum /afs/realmname.com/home/timothy/OpenAFS-ServerDos.pdf
cd688648171672c1a5100aaa9e3186b5 /afs/
realmname.com/home/timothy/OpenAFS-ServerDos.pdf
#:~/scripts# md5sum foo
cd688648171672c1a5100aaa9e3186b5 foo
But fails from the RW tree:
#:~/scripts# afsio read -file /afs/.
realmname.com/home/timothy/OpenAFS-ServerDos.pdf > foo
afsio: No such file or directory (file not found: /afs/.realmname
.com/home/timothy/OpenAFS-ServerDos.pdf)
And of course, I can't seem to write to the read only tree, unsurprisingly:
#:~/scripts# cat foo.sh| afsio write -file
/afs/realmname.com/home/timothy/foo.sh
afsio: server or network not responding (could not create file foo.sh in
directory 536870922.1.1)
> Am I making any sense, here? What I describe above is the way to
> transfer lots of little files into AFS more quickly (and possibly other
> network filesystems depending on their consistency guarantees)
> regardless of other schemes like OSD in play. Something like that would
> be useful in general for people that want to copy a large tree or untar
> something into AFS. As far as I know it does not already exist, but I
> could be wrong.
You are making a LOAD of sense, Andrew, and.. as always.. your patience is
sincerely appreciated. I do believe I want to write up a system that will
accomplish this and make it relatively generic.. so once I have it done, I
can send it off for people to review and use :) If I could have some help
with getting afsio working properly, I think that will be enough to get me
going.
Thanks also to everyone else as well.. I am learning more about the
underpinnings of file systems working with AFS than I have in a very long
time!
--
Timothy Balcer / IT Services
Telmate / San Francisco, CA
Direct / (415) 300-4313
Customer Service / (800) 205-5510
--
Timothy Balcer / IT Services
Telmate / San Francisco, CA
Direct / (415) 300-4313
Customer Service / (800) 205-5510
--20cf3071cd08f4aea804ce1801c8
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<div class=3D"gmail_quote"><div class=3D"im">On Fri, Nov 9, 2012 at 1:47 PM=
, Andrew Deason <span dir=3D"ltr"><<a href=3D"mailto:adeason@sinenomine.=
net" target=3D"_blank">adeason@sinenomine.net</a>></span> wrote:<br></di=
v><div class=3D"gmail_quote">
<div class=3D"im"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .=
8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><br>
> Yes.. I understand that. I was commenting on the slowness as compared<=
br>
> to rsyncing over NFS, for example, which takes 5 hours for the entire<=
br>
> tree when done from the top level of the tree. That tree contains 15<b=
r>
> of the directories that I mentioned in my earlier post. So 15 * 24k<br=
>
> dirs.. and to answer the question, 232,974 files of small size for the=
<br>
> one subdirectory in question.<br>
<br>
</div>I'm getting a little mixed up about when you're switching fro=
m talking<br>
about the 'NFS solution' vs the 'AFS solution'. Are you say=
ing it took<br>
1.5 hours to transfer 15 subdirs into AFS, where it took NFS 5 hours to<br>
transfer 24000 subdirs?<br></blockquote></div><div><br>I am not being clear=
about the numbers I am talking about it looks like... :)<br><br>It takes a=
bout 5 hours to transfer/rsync the entire tree, with 15 subdirs, each conta=
ining upwards of 200k files and 24k directories, when I am going from a loc=
al volume to an NFSv4 mount. This NFSv4 mount is on the local network, so t=
here is not a network bottleneck per se. That's the generic "faste=
st" case I am working against.<br>
<br><br></div><div class=3D"im"><blockquote class=3D"gmail_quote" style=3D"=
margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><br>
> > What can possibly make this faster, then, is copying/creating<br>
> > files/directories in parallel. <snip><br>
><br>
> Yes, I routinely run 100's of parallel transfers using a combinati=
on<br>
> of tar and rsync.. tar gets the files over in raw form, and rsync mops=
<br>
> up behind. =A0The rsync pass is to correct any problems with the tar<b=
r>
> copy, and is run twice on a fixed list, generated at transfer time. I<=
br>
> have found that even when using a tuned rsync process designed to<br>
> improve transfer speeds, many parallel tar/untar processes from local<=
br>
> to =A0NFSv4 followed by a "local" rsync to the same destinat=
ion works<br>
> better for new files, when timeliness is important.<br>
<br>
</div>So... are you just talking about the NFS transfers, here?<br></blockq=
uote></div><div><br>Right, however using tar and rsync to manage the transf=
ers.<br>=A0<br></div><div class=3D"im"><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
With AFS, trying to rsync again afterwards is possibly much slower due<br>
to cache churn if the number of files in question is larger than the<br>
stat cache (as Jeff said earlier). You also cannot do more than 4<br>
simultaneous 'things' to the AFS server with current client release=
s<br>
(unless you fiddle with PAGs, or use some AFS-specific tools). So 100s<br>
of parallel transfers aren't really helpful.<br></blockquote></div><div=
><br>Thats excellent information! :) And thanks to you too Jeff.<br>=A0<br>=
</div><div class=3D"im"><blockquote class=3D"gmail_quote" style=3D"margin:0=
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
So if you want to write the tooling for it, I think ideally what you<br>
would want is a cp/rsync-like tool that would copy files/dirs using a<br>
separate thread for each file/dir up to some configured limit, tracking<br>
dependencies so parent dirs are created first, possibly launching new<br>
processes in separate PAGs as you go. Or, you could use utilities or<br>
APIs that speak to AFS directly without going through the filesystem<br>
layer (like afscp/afsio), so you wouldn't need separate threads or<br>
processes.<br></blockquote></div><div><br>afsio sounds like the perfect fit=
for this, actually... although I did have some trouble with writing with i=
t...I checked manually with ls along the trees to make sure all tokens were=
working and etc.<br>
<br><span style=3D"font-family:courier new,monospace">#:~/scripts# cat foo.=
sh | afsio write -file /afs/.<a href=3D"http://realmname.com/home/timothy/f=
oo.sh" target=3D"_blank">realmname.com/home/timothy/foo.sh</a><br>afsio: No=
such file or directory (is dir /afs/.</span><span style=3D"font-family:cou=
rier new,monospace"><span style=3D"font-family:courier new,monospace">realm=
name</span>.com/home/timothy in AFS?) </span><br>
<br>an afsio read succeeds from the RO tree:<br><span style=3D"font-family:=
courier new,monospace"><br>#:~/scripts# afsio read -file /afs/<a href=3D"ht=
tp://realmname.com/home/timothy/OpenAFS-ServerDos.pdf" target=3D"_blank">re=
almname.com/home/timothy/OpenAFS-ServerDos.pdf</a> > foo<br>
#:~/scripts# md5sum /afs/</span><span style=3D"font-family:courier new,mono=
space"><span style=3D"font-family:courier new,monospace">realmname</span>.c=
om/home/timothy/OpenAFS-ServerDos.pdf<br>cd688648171672c1a5100aaa9e3186b5=
=A0 /afs/<a href=3D"http://realmname.com/home/timothy/OpenAFS-ServerDos.pdf=
" target=3D"_blank">realmname.com/home/timothy/OpenAFS-ServerDos.pdf</a><br=
>
#:~/scripts# md5sum foo<br>cd688648171672c1a5100aaa9e3186b5=A0 foo<br></spa=
n><br>But fails from the RW tree:<br><br><span style=3D"font-family:courier=
new,monospace">#:~/scripts# afsio read -file /afs/.<a href=3D"http://realm=
name.com/home/timothy/OpenAFS-ServerDos.pdf" target=3D"_blank">realmname.co=
m/home/timothy/OpenAFS-ServerDos.pdf</a> > foo<br>
afsio: No such file or directory (file not found: /afs/.</span><span style=
=3D"font-family:courier new,monospace"><span style=3D"font-family:courier n=
ew,monospace">realmname</span>.com/home/timothy/OpenAFS-ServerDos.pdf)</spa=
n><br>
<br>And of course, I can't seem to write to the read only tree, unsurpr=
isingly:<br><br><span style=3D"font-family:courier new,monospace">#:~/scrip=
ts# cat foo.sh| afsio write -file /afs/</span><span style=3D"font-family:co=
urier new,monospace"><span style=3D"font-family:courier new,monospace"></sp=
an><span style=3D"font-family:courier new,monospace"><span style=3D"font-fa=
mily:courier new,monospace">realmname</span></span>.com/home/timothy/foo.sh=
<br>
afsio: server or network not responding (could not create file foo.sh in di=
rectory 536870922.1.1)</span><br><br></div><div class=3D"im"><blockquote cl=
ass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;p=
adding-left:1ex">
<br>
Am I making any sense, here? What I describe above is the way to<br>
transfer lots of little files into AFS more quickly (and possibly other<br>
network filesystems depending on their consistency guarantees)<br>
regardless of other schemes like OSD in play. Something like that would<br>
be useful in general for people that want to copy a large tree or untar<br>
something into AFS. As far as I know it does not already exist, but I<br>
could be wrong.</blockquote></div><div>=A0<br>You are making a LOAD of sens=
e, Andrew, and.. as always.. your patience is sincerely appreciated. I do b=
elieve I want to write up a system that will accomplish this and make it re=
latively generic.. so once I have it done, I can send it off for people to =
review and use :) If I could have some help with getting afsio working prop=
erly, I think that will be enough to get me going.<br>
<br>Thanks also to everyone else as well.. I am learning more about the und=
erpinnings of file systems working with AFS than I have in a very long time=
!<br></div></div><div class=3D"HOEnZb"><div class=3D"h5"><br clear=3D"all">=
<br>
-- <br><span style=3D"border-collapse:collapse;color:rgb(102,102,102);font-=
family:verdana,sans-serif;font-size:x-small">Timothy Balcer / IT Services<b=
r>
Telmate / San Francisco, CA<br>Direct / </span><span style=3D"border-collap=
se:collapse;font-family:verdana,sans-serif;font-size:x-small"><font color=
=3D"#1155cc"><a href=3D"tel:%28415%29%20300-4313" value=3D"+14153004313" ta=
rget=3D"_blank">(415) 300-4313</a></font><br>
<font color=3D"#666666">Customer Service /=A0</font><a value=3D"+1800205551=
0" style=3D"color:rgb(17,85,204)">(800) 205-5510</a></span><br>
</div></div></div><br><br clear=3D"all"><br>-- <br><span style=3D"border-co=
llapse:collapse;color:rgb(102,102,102);font-family:verdana,sans-serif;font-=
size:x-small">Timothy Balcer / IT Services<br>Telmate / San Francisco, CA<b=
r>
Direct / </span><span style=3D"border-collapse:collapse;font-family:verdana=
,sans-serif;font-size:x-small"><font color=3D"#1155cc">(415) 300-4313</font=
><br><font color=3D"#666666">Customer Service /=A0</font><a value=3D"+18002=
055510" style=3D"color:rgb(17,85,204)">(800) 205-5510</a></span><br>
--20cf3071cd08f4aea804ce1801c8--