[OpenAFS] About AFS performance over WAN
Ted Creedon
tcreedon@easystreet.net
Mon, 1 Dec 2008 08:59:03 -0800
------=_Part_89832_25770605.1228150743139
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Will the ISP reaction to BitTorrent's UDP packets affect AFS on public
networks?
http://www.theregister.co.uk/2008/12/01/richard_bennett_utorrent_udp/
On Mon, Dec 1, 2008 at 1:24 AM, Rainer Toebbicke <rtb@pclella.cern.ch>wrote:
>
> Giovanni Bracco schrieb:
>
> I know that this is a well know problem of the rx protocol, as shown for
>> example by Hartmut Reuter at the last European AFS Conference 2008
>> (see slide 49 from
>> http://www.openafs.at/drupal/files/slides/1Day_03/AFS-OSD.pdf), due to
>> the fixed rx window size and combined with network latencies in the order of
>> tenths of milliseconds.
>>
>> I am aware that an activity was in progress for a tcp version of openafs,
>> which probably could solve some of this problem, but I do not know what is
>> the status of this activity. More generally, what are the plans to increase
>> the AFS performances over WAN, to take advantage of the present day
>> availability of high bandwith connections?
>>
>>
> What RX-over-TCP would bring you is to copy all the improvements that went
> into TCP over the past decade of research into RX. And it will make things
> more familiar for network administrators dealing mainly with TCP
> considerations. What it will not bring you is a bulk transfer protocol.
>
> AFS transfers files chunk-wise, while there is read-ahead the transfer is
> essentially still sequential. Due to the RPC nature of the protocol you will
> have a stop every 64K (or 256K, or whatever you typically set it to). A
> plain port to TCP will not change anything there, worse such a start-stop in
> a single TCP stream could very well challenge the sophisticated techniques
> that went into window heuristics and congestion control.
>
> With unlimited development resources AFS would deserve a better suited
> protocol than TCP, in practice with a little more realism my gut feeling is
> that at least some more brain should be devoted to improving plain RX rather
> than betting on another horse. I occasionally tried over the past years,
> with some improvements that Hartmut tested as well, but my brain being what
> it is and the matter relatively complicated results remain modest.
>
> High latency remains a fierce enemy. Some address it through pre-fetches
> which are a double-sided sword! For read, if the file system had reliable
> knowledge about big files (or series of files) to be transferred in their
> entirety AFS could relatively easily be modified to start chunk pre-fetches
> in parallel, slightly shifted in time, over standard RX, solving the
> start-stop problematic. The key here is to do this only if you're sure
> you're not over-speculating and throwing away most of it soon after.
>
> For writes, here at CERN we already run with mods that start chunk
> transmission early while the file is still being written to. Naively
> thinking that would be vastly easier to improve given that much more is
> known!
>
> --
>
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Rainer Toebbicke
> European Laboratory for Particle Physics(CERN) - Geneva, Switzerland
> Phone: +41 22 767 8985 Fax: +41 22 767 7155
> _______________________________________________
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
>
------=_Part_89832_25770605.1228150743139
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Will the ISP reaction to BitTorrent's UDP packets affect AFS on public networks?<br><br><a href="http://www.theregister.co.uk/2008/12/01/richard_bennett_utorrent_udp/">http://www.theregister.co.uk/2008/12/01/richard_bennett_utorrent_udp/</a><br>
<br><div class="gmail_quote">On Mon, Dec 1, 2008 at 1:24 AM, Rainer Toebbicke <span dir="ltr"><<a href="mailto:rtb@pclella.cern.ch">rtb@pclella.cern.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
Giovanni Bracco schrieb:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I know that this is a well know problem of the rx protocol, as shown for example by Hartmut Reuter at the last European AFS Conference 2008<br>
(see slide 49 from <a href="http://www.openafs.at/drupal/files/slides/1Day_03/AFS-OSD.pdf" target="_blank">http://www.openafs.at/drupal/files/slides/1Day_03/AFS-OSD.pdf</a>), due to the fixed rx window size and combined with network latencies in the order of tenths of milliseconds.<br>
<br>
I am aware that an activity was in progress for a tcp version of openafs, which probably could solve some of this problem, but I do not know what is the status of this activity. More generally, what are the plans to increase the AFS performances over WAN, to take advantage of the present day availability of high bandwith connections?<br>
<br>
</blockquote>
<br>
What RX-over-TCP would bring you is to copy all the improvements that went into TCP over the past decade of research into RX. And it will make things more familiar for network administrators dealing mainly with TCP considerations. What it will not bring you is a bulk transfer protocol.<br>
<br>
AFS transfers files chunk-wise, while there is read-ahead the transfer is essentially still sequential. Due to the RPC nature of the protocol you will have a stop every 64K (or 256K, or whatever you typically set it to). A plain port to TCP will not change anything there, worse such a start-stop in a single TCP stream could very well challenge the sophisticated techniques that went into window heuristics and congestion control.<br>
<br>
With unlimited development resources AFS would deserve a better suited protocol than TCP, in practice with a little more realism my gut feeling is that at least some more brain should be devoted to improving plain RX rather than betting on another horse. I occasionally tried over the past years, with some improvements that Hartmut tested as well, but my brain being what it is and the matter relatively complicated results remain modest.<br>
<br>
High latency remains a fierce enemy. Some address it through pre-fetches which are a double-sided sword! For read, if the file system had reliable knowledge about big files (or series of files) to be transferred in their entirety AFS could relatively easily be modified to start chunk pre-fetches in parallel, slightly shifted in time, over standard RX, solving the start-stop problematic. The key here is to do this only if you're sure you're not over-speculating and throwing away most of it soon after.<br>
<br>
For writes, here at CERN we already run with mods that start chunk transmission early while the file is still being written to. Naively thinking that would be vastly easier to improve given that much more is known!<br>
<br>
-- <br>
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<br><font color="#888888">
Rainer Toebbicke<br>
European Laboratory for Particle Physics(CERN) - Geneva, Switzerland<br>
Phone: +41 22 767 8985 Fax: +41 22 767 7155<br>
_______________________________________________<br>
OpenAFS-info mailing list<br>
<a href="mailto:OpenAFS-info@openafs.org" target="_blank">OpenAFS-info@openafs.org</a><br>
<a href="https://lists.openafs.org/mailman/listinfo/openafs-info" target="_blank">https://lists.openafs.org/mailman/listinfo/openafs-info</a><br>
</font></blockquote></div><br>
------=_Part_89832_25770605.1228150743139--