[OpenAFS] Overview? Linux filesystem choices

[Steve Simmons]

On Sep 28, 2010, at 4:00 PM, Thomas Kula wrote:

> On Tue, Sep 28, 2010 at 12:49:59PM -0700, Russ Allbery wrote:
>> Jeff Blaine <jblaine@kickflop.net> writes:
>>=20
>>> Barring an equivalent, what Linux setup...
>>=20
>>>  a) seems most stable
>>>  b) is fsck-less
>>=20
>>> Even quick grunt responses are appreciated.
>>=20
>> We use ext3.  It isn't the fastest or the most featureful, but it's =
the
>> core file system that everyone uses on Linux and for us it's been =
rock
>> solid.  You're the least likely to run into strange problems.
>=20
> We (umich.edu) also use ext3. We randomly run into issues where=20
> the filesystem half-thinks that things that should be files are
> directories, which, when this happens on a vice paritition, leads
> to interesting problems.=20
>=20
> Other co-workers (some of whom I believe are on this list) follow
> this more, but I think our strategy has been to keep on top of any
> kernel issues and the corresponding userspace tools for dealing with
> ext filesystems and see what those do. I have no idea why we tend
> to run into this with not-frequent-but-too-often-for-me regularity.
>=20
> That said, I'm not sure what else we'd even consider running on
> Linux systems.

*raises hand as 'other co-worker'*

Yep, we're pretty much a 100% ext3 shop. We keep a close eye on other =
things, and zfs has been looking more and more interesting. But given =
the uncertain state of its future (see elsewhere in this thread) our =
caution level has gone up quite a bit. For the really long term I also =
keep an eye on btrfs, but some of it's features aren't as big a win for =
AFS as they are for 'regular' users. Ext4 looks interesting just for the =
fsck speed improvements (just freaking amazing). Extents may also be =
useful, but I strongly suspect other issues bottleneck AFS performance =
before the filesystem speed does. Then again, better speed never hurts.

Tom refers to some ext3 problem with directories suddenly becoming files =
or vice-versa. There are two points worth mentioning here. First, it is =
extremely rare - maybe once every six months. That's in a cell with 26 =
file servers, 64 vice partitions, 260,000 volumes, 180M files, 92TB of =
raw space for AFS with 46TB currently used, compounded growth rate of =
about 45% per year. Big. Second, we are running our own =
linux-from-scratch systems. It's quite possible we have introduced a =
frailty somewhere.

We have several times been bitten in the ass by ext3 bugs, and the =
recovery process has not been pretty. Usually we've traced this down to =
actual ext3 bugs that others have found and fixed; I've not yet recently =
made that chase on our dir-vs-file problem. For all I know, it's oAFS =
hosing up an inode somehow. But without having equivalent data about =
what other sites do and their size/configurations, I can't honestly say =
if our problems are unique to us or just somethings that most folks =
manage to run below the radar on.

But either way we'll be sticking to ext3 for at least the next couple of =
years.

Steve=