[OpenAFS-devel] Re: Backups using commercial products

[Sam Hartman]

>>>>> "Lyle" == Lyle Seaman <lws@spinnakernet.com> writes:

    Lyle> The debate is both "what is easier now", and "what will
    Lyle> continue to be easier in the future."  But then, that varies
    Lyle> from person to person.  It's my opinion, based on my past
    Lyle> experience with RX and other transport/session layer
    Lyle> protocols, that writing protocols, and making them
    Lyle> efficient, is harder than it seems at first glance.

So, I stated something I don't actually believe.  What I should have
said is that perhaps we can significantly improve RX without too much
cost.  But I agree that choosing someone else's transport is the
correct long-term goal.




    Lyle> But you have to be doing a pretty good job of scheduling I/O
    Lyle> in order to pull 12 MB/s randomly off a single disk with a
    Lyle> single thread.  It's no trouble for a single large file, but
    Lyle> when you have to snarf up small files, directories,
    Lyle> symlinks, and to top it off, do timestamp comparisons on
    Lyle> them and potentially skip them entirely...

OK, I was not thinking this through carefully.  It would be nice
though if you got reasonable network utilization for the large files;
that might be easier to achieve.


    Lyle> In other words, if you are getting an ack every millisecond
    Lyle> or two, but an individual seek takes 9 milliseconds to
    Lyle> complete, you've got no chance of having data ready, unless
    Lyle> you're doing enough disk I/O to get your average time down

    Lyle> async I/O and a "sendfile" sort of functionality would help,
    Lyle> no doubt.  And I don't think that "you may use more CPU
    Lyle> because of overhead" will be an issue, as I believe the
    Lyle> existing CPU costs are mostly in RX per-packet processing,
    Lyle> just as they always have been.

By use more CPU I mean than is necessary not than is in the current
implementation.  I need to think about how sendfile would interact
with constructing security headers.  I guess not a big problem for
clear and even for crypt you might avoid a copy.


    Lyle> Go ahead, try it.  It will be interesting and informative,
    Lyle> at least, and you may well be right.

I'll see what I can do.  Limited chance of success because I'm only
willing to expend so much time and because I'm still learning the
code.

Thanks for your thoughts?