[OpenAFS] Which storage technology to use for terabytes of storage with AFS?

Stephen Joyce stephen@physics.unc.edu
Fri, 30 Nov 2007 13:51:25 -0500 (EST)

I don't have money for FC or a SAN, so I've stuck with DAS. I've had good 
experience with building many smallish servers rather than one big or 
expensive one.

I'm currently using cheap Dell PowerEdge servers running linux. I think we 
got them for about $800/ea, and they support console redirection (critical 
when you have lots of physical servers). We added a 2-port 3ware raid1 for 
the OS and a 4-port 3ware for the data (raid1 or raid5 depending on the 
requirements). Right now I'm keeping the servers to around 1TB each, but 
they're capable of hosting 2-4TB each (depending on raid level) with the 
largest current drives.

If money were no object, I'd have opted for hot-swappable drives, but with 
under 1TB of data on each, any time I've needed to replace a drive I've 
just moved the volumes to another server.

These systems are cheap enough (under about $1.5K each for everything) that 
I keep a spare of everything just in case (spare fully configured and 
running server plus spare raid cards and drives on the shelf).

I _strongly_ advise raid. Raid1 for the OS and raid1, 5, or 6 for the data, 
depending on your requirements. I know some people have reported impressive 
results with linux software raid, but I swear by 3ware hardware raid 
controllers; they "just work." Just avoid "fakeraid" controller cards 
(promise, low-end adaptec, etc) like the plague. They're far more trouble 
than they're worth.

I really like solaris, but this setup is MUCH cheaper and faster than our 
old solaris setup.

On Fri, 30 Nov 2007, Jason Edgecombe wrote:

> Hi everyone,
> Traditionally, we have used direct-attached scsi disk packs on Sun Sparc
> servers running Solaria 9 for OpenAFS. This has given us the most bang
> for the buck. We forgo RAID because we have the backup capabilities of AFS.
> What types of storage technologies are other AFS sites using for their
> AFS vicep partitions? We need to figure our future direction for the
> next couple of years. Fibre channel seems all the rage, but it's quite
> expensive. I'm open to any and all feedback. What works? What doesn't?
> What offers the best bang for the buck on an OpenAFS server?
> This is for an academic environment that fills both academic and
> research needs. Researchers are asking for lots of AFS space (200GB+).
> Of course this needs to be backed up as well.
> Thanks,
> Jason

Cheers, Stephen
Stephen Joyce
Systems Administrator                                            P A N I C
Physics & Astronomy Department                         Physics & Astronomy
University of North Carolina at Chapel Hill         Network Infrastructure
voice: (919) 962-7214                                        and Computing
fax: (919) 962-0480                               http://www.panic.unc.edu

Don't judge a book by its movie.