[AFS3-std] AFS3 support for sparse files (SEEK_HOLE&co)?

Thu, 4 Sep 2014 13:13:22 +0100

--Apple-Mail=_B085B779-63E6-4C46-AA68-C3C712BE5ABC
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=windows-1252

On 4 Sep 2014, at 09:33, Lionel Cons <lionelcons1972@gmail.com> wrote:

> On 4 September 2014 03:37, Jeffrey Altman =
<jaltman@your-file-system.com> wrote:
>> On 9/3/2014 1:29 PM, Lionel Cons wrote:
>>> Will AFS3 include support for sparse files, e.g. files which have =
one
>>> or multiple holes (see POSIX lseek() documentation about SEEK_HOLE =
and
>>> SEEK_DATA) where no data reside?

I don=92t think SEEK_HOLE and SEEK_DATA are in issue 7. There=92s text =
defining them proposed as part of Issue 8 - see =
http://austingroupbugs.net/view.php?id=3D415 for the accepted text.

> There is one very large, and very bad misconception:
> Holes in a sparse file return '\0' (zero) bytes on read(), but it does
> not mean that all data which contain '\0' bytes are holes. This
> misconception can lead to serious data loss situations because
> applications need to be able to differ between holes in a file -
> representing 'no data here' and large ranges containing '\0' bytes -
> which represent valid data containing '\0' bytes.

I=92m surprised that applications can actually get away with making =
these differentiations, given the way that sparse files are typically =
implemented. In particular, there=92s no portable guarantee of the =
granularity of a hole. Your attempt to leave a 12 byte hole may be =
rejected by the filesystem, and be replaced by 12 zero-bytes, and your =
attempt to leave a 5000 byte hole may become a 4096 byte hole, and 4 =
zero bytes.

This is particularly true of the mmap use case where writing a single =
byte is likely to result in the whole page that you=92ve written to =
becoming non-sparse. On filesystems which manage sparse files using disk =
blocks, it will probably also result in the whole block containing that =
byte becoming allocated.

What level of granularity do your applications expect to have for sparse =
holes? What interfaces are they using to create the holes themselves - =
are they just doing seek then write, or are they using things like =
fallocate() with the zero_range or punch_hole options?

Cheers,

Simon

--Apple-Mail=_B085B779-63E6-4C46-AA68-C3C712BE5ABC
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=windows-1252

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dwindows-1252"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: =
after-white-space;"><br><div><div>On 4 Sep 2014, at 09:33, Lionel Cons =
&lt;<a =
href=3D"mailto:lionelcons1972@gmail.com">lionelcons1972@gmail.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><div style=3D"font-size: 12px; font-style: normal; =
font-variant: normal; font-weight: normal; letter-spacing: normal; =
line-height: normal; orphans: auto; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: auto; word-spacing: =
0px; -webkit-text-stroke-width: 0px;">On 4 September 2014 03:37, Jeffrey =
Altman &lt;<a =
href=3D"mailto:jaltman@your-file-system.com">jaltman@your-file-system.com<=
/a>&gt; wrote:<br><blockquote type=3D"cite">On 9/3/2014 1:29 PM, Lionel =
Cons wrote:<br><blockquote type=3D"cite">Will AFS3 include support for =
sparse files, e.g. files which have one<br>or multiple holes (see POSIX =
lseek() documentation about SEEK_HOLE and<br>SEEK_DATA) where no data =
reside?<br></blockquote></blockquote></div></blockquote><div><br></div><di=
v>I don=92t think SEEK_HOLE and SEEK_DATA are in issue 7. There=92s text =
defining them proposed as part of Issue 8 - see&nbsp;<a =
href=3D"http://austingroupbugs.net/view.php?id=3D415">http://austingroupbu=
gs.net/view.php?id=3D415</a> for the accepted text.</div><br><blockquote =
type=3D"cite"><div style=3D"font-size: 12px; font-style: normal; =
font-variant: normal; font-weight: normal; letter-spacing: normal; =
line-height: normal; orphans: auto; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: auto; word-spacing: =
0px; -webkit-text-stroke-width: 0px;">There is one very large, and very =
bad misconception:<br>Holes in a sparse file return '\0' (zero) bytes on =
read(), but it does<br>not mean that all data which contain '\0' bytes =
are holes. This<br>misconception can lead to serious data loss =
situations because<br>applications need to be able to differ between =
holes in a file -<br>representing 'no data here' and large ranges =
containing '\0' bytes -<br>which represent valid data containing '\0' =
bytes.<br></div></blockquote><div><br></div>I=92m surprised that =
applications can actually get away with making these differentiations, =
given the way that sparse files are typically implemented. In =
particular, there=92s no portable guarantee of the granularity of a =
hole. Your attempt to leave a 12 byte hole may be rejected by the =
filesystem, and be replaced by 12 zero-bytes, and your attempt to leave =
a 5000 byte hole may become a 4096 byte hole, and 4 zero =
bytes.</div><div><br></div><div>This is particularly true of the mmap =
use case where writing a single byte is likely to result in the whole =
page that you=92ve written to becoming non-sparse. On filesystems which =
manage sparse files using disk blocks, it will probably also result in =
the whole block containing that byte becoming =
allocated.</div><div><br></div><div>What level of granularity do your =
applications expect to have for sparse holes? What interfaces are they =
using to create the holes themselves - are they just doing seek then =
write, or are they using things like fallocate() with the zero_range or =
punch_hole =
options?</div><div><br></div><div>Cheers,</div><div><br></div><div>Simon</=
div><div><br></div><div><br></div></body></html>=

--Apple-Mail=_B085B779-63E6-4C46-AA68-C3C712BE5ABC--