[OpenAFS] Number of files in an OpenAFS volume...
Matthew Lowy
matthew.lowy@it.ox.ac.uk
Mon, 4 Jul 2016 10:55:36 +0000
--_000_2BF2A3DFFECCD44B9DDF015424A7673EA60665MBX03adoakoxacuk_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hello,
We have a number of OpenAFS volumes that serve as storage for (public) mirr=
ors and one of them is misbehaving when updated from upstream - the error i=
ndicates we've reached the limit of file names allowed in a volume.
The limit I am seeing is not compatible with my understanding of how OpenAF=
S handles file names in a directory. I've seen in the mail list archives th=
e statements about how many file names can fit, that there are 64k slots an=
d a file name < 16 in length occupies one slot, a file name from 16 to 32 c=
haracters long occupies two slots and so on. The earliest reference I've fo=
und is at http://lists.openafs.org/pipermail/openafs-info/2002-September/00=
5812.html
However...
The directory concerned has more than 21,000 files in it, almost all of the=
m have names exceeding 52 characters... as at today there are 1,220,000 cha=
racters in filenames in that directory. Even assuming they pack down perfec=
tly into directory name slots that's over 76,000 slots... and working them =
out using the rule above indicates that the directory is using over 87,000 =
slots. These are both significantly above 64k.
I don't know if I'm misinterpreting the information in the OpenAFS archive =
or if the information is out of date - but I've not found anything that fun=
damentally is different from the information in the archive and I'm looking=
at a volume that seems to break the limits. I'd really benefit from unders=
tanding what's going on ... how we appear to be getting more file name info=
rmation into a directory than should be possible.
/mirror.ox.ac.uk/sites/archive.ubuntu.com/ubuntu/pool/main/l/linux$ ls |wc =
-l
21731
/mirror.ox.ac.uk/sites/archive.ubuntu.com/ubuntu/pool/main/l/linux$ ls |wc =
-c
1250894
This is one directory in a mirror of archive.ubuntu.com so you can see the =
contents from (e.g) https://launchpad.net/ubuntu/+mirror/mirror.ox.ac.uk-ar=
chive which points to the presentation of our mirror. The number of files h=
as recently gone up because of upstream changes.
Thanks
Matthew
--_000_2BF2A3DFFECCD44B9DDF015424A7673EA60665MBX03adoakoxacuk_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html dir=3D"ltr">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
1">
<style id=3D"owaParaStyle" type=3D"text/css">P {margin-top:0;margin-bottom:=
0;}</style>
</head>
<body ocsi=3D"0" fpstyle=3D"1">
<div style=3D"direction: ltr;font-family: Tahoma;color: #000000;font-size: =
10pt;">Hello,<br>
<br>
We have a number of OpenAFS volumes that serve as storage for (public) mirr=
ors and one of them is misbehaving when updated from upstream - the error i=
ndicates we've reached the limit of file names allowed in a volume.<br>
<br>
The limit I am seeing is not compatible with my understanding of how OpenAF=
S handles file names in a directory. I've seen in the mail list archives th=
e statements about how many file names can fit, that there are 64k slots an=
d a file name < 16 in length occupies
one slot, a file name from 16 to 32 characters long occupies two slots and=
so on. The earliest reference I've found is at
<a href=3D"http://lists.openafs.org/pipermail/openafs-info/2002-September/0=
05812.html" target=3D"_blank">
http://lists.openafs.org/pipermail/openafs-info/2002-September/005812.html<=
/a> <br>
<br>
However...<br>
<br>
The directory concerned has more than 21,000 files in it, almost all of the=
m have names exceeding 52 characters... as at today there are 1,220,000 cha=
racters in filenames in that directory. Even assuming they pack down perfec=
tly into directory name slots that's
over 76,000 slots... and working them out using the rule above indicates t=
hat the directory is using over 87,000 slots. These are both significantly =
above 64k.<br>
<br>
I don't know if I'm misinterpreting the information in the OpenAFS archive =
or if the information is out of date - but I've not found anything that fun=
damentally is different from the information in the archive and I'm looking=
at a volume that seems to break
the limits. I'd really benefit from understanding what's going on ... how =
we appear to be getting more file name information into a directory than sh=
ould be possible.<br>
<br>
/mirror.ox.ac.uk/sites/archive.ubuntu.com/ubuntu/pool/main/l/linux$ ls |wc =
-l<br>
21731<br>
/mirror.ox.ac.uk/sites/archive.ubuntu.com/ubuntu/pool/main/l/linux$ ls |wc =
-c<br>
1250894<br>
<br>
This is one directory in a mirror of archive.ubuntu.com so you can see the =
contents from (e.g)
<a href=3D"https://launchpad.net/ubuntu/+mirror/mirror.ox.ac.uk-archive=
" target=3D"_blank">
https://launchpad.net/ubuntu/+mirror/mirror.ox.ac.uk-archive</a> which =
points to the presentation of our mirror. The number of files has recently =
gone up because of upstream changes.<br>
<br>
Thanks<br>
<br>
Matthew<br>
</div>
</body>
</html>
--_000_2BF2A3DFFECCD44B9DDF015424A7673EA60665MBX03adoakoxacuk_--