[OpenAFS] Which storage technology to use for terabytes of storage with AFS?

[Steve Simmons]

On Nov 30, 2007, at 10:58 AM, Jason Edgecombe wrote:

> What types of storage technologies are other AFS sites using for their
> AFS vicep partitions? We need to figure our future direction for the
> next couple of years. Fibre channel seems all the rage, but it's quite
> expensive. I'm open to any and all feedback. What works? What doesn't?
> What offers the best bang for the buck on an OpenAFS server?
>
> This is for an academic environment that fills both academic and
> research needs. Researchers are asking for lots of AFS space (200GB+).
> Of course this needs to be backed up as well.

We use SATA raids with 16 disks, configured as 0+1 with two disk sets  
per raid, one of eight and one of six, with two hot spares. Thus the  
400Gx16 raid winds up with two partitions, one at 1.375TB and one at  
1.125TB. Testing that was done before I got here said the performance  
of that device equaled that of similarly configured SCSI-based  
systems at a considerably cheaper $ per GB price.

Ours is an academic environment exactly as you describe.

We have users asking for hundreds of GBs. We've found that somewhere  
in the neighborhood of 100G used, volumes become too unweildy to  
handle during working hours. Vos move, etc, just takes too damned  
long. We therefore work with our users to try and cap individual  
volumes at 50G, and give them multiple volumes to get them up to  
their need point.

I've recently been arguing that the partition sizes are too large,  
and our next set of servers should have smaller ones. On the flip  
side, with lots of 100GB partitions, balancing on smaller partitions  
might become a challenge.

We do our own backups with a huge 'vos dump' script and other backing  
software. I'm in mid-conversion to mysql for part of the meta-data,  
and when done will post it. Somebody from our site will be at the  
next openafs best practices workshop, most likely me, and I'll  
present on it. Interested parties can have a pre-release version from  
before we started the mysql work; that was pretty stable.

Steve