[OpenAFS] Re: vos move speed rates

[Chris Hoy Poy]

Hi all,

I've got a couple of XFS backed servers and a couple of EXT3 backed servers.

My findings: 
The XFS backed servers consistently: 
      - use less CPU in operations
      - occupy less disk space (I have a lot of homedirs on AFS, they use around 10-20% *less* space on AFS) 
      - in general perform better (under my workloads, lots of smallish homedirs, obviously your mileage may vary etc)

Details:

I have these systems virtualised (libvirt/KVM, Ubuntu 10.04) and all the servers regularly move around the hardware. They are Ubuntu 10.04 based systems, upgraded from Ubuntu 8.04

The disk subsystems are mainly Dell MD3000i's, 15x SAS 15k disks and then iSCSI to the VMs. 

I haven't done an extreme amount of testing, but these have been in production for 2+ years with minimal intervention and issues. There are multiple servers so the load is distributed around the VM farm, and to provide some quarantining in case of failure etc.

They are balanced every day based on IO/quota etc. 

during backups, the EXT3 backed servers will often lag a bit - sometimes SNMP and other utilities will lock up for a minute or two. The XFS systems display no such issue.

No real issues, but I definitely recommend XFS based on my experience. Its been on my todo list to migrate the volumes off of the EXT3 systems and rebuild them with XFS, but it hasnt hit the criticality point, although the space issue is one that I would like to research more (of course). Given how often the balancer script churns the volumes, I don't think its anything too dumb in there (though its always a possibility). But when I move a 1GB volume off of an XFS backed server to an EXT3 backed server, it occupies 10-20% more space :) 

These are default XFS / EXT3 based systems, very little tuning. Could be a lot of reasons for the space discrepancy, and I just havent looked hard enough yet ;) 

//Chris


----- Original Message -----
From: "Harald Barth" <haba@kth.se>
To: ecgarris@iu.edu, ecgarris@iupui.edu
Cc: openafs-info@openafs.org
Sent: Thursday, 14 April, 2011 3:14:55 AM
Subject: Re: [OpenAFS] Re: vos move speed rates

> Anyone have experience using XFS or other filesystems instead of ext*?

*waves*

I am using xfs now and plan to use xfs on my new servers. This time I
will try to tune xfs as well. Currently I lean to the following config:

Dell515 with Centos 5

HW mirrored SATA disks for root file system and xfs log
(/dev/rootvg/vicepalog)

md RAID6 over 6 SAS disks (/dev/vicevg0/vicepa) at a time

md tuning:
echo 16384 > /sys/block/md0/md/stripe_cache_size

xfs creation parameters:
mkfs.xfs -f -l
logdev=/dev/rootvg/vicepalog,size=134217728,lazy-count=1,su=4k
/dev/vicevg0/vicepa

xfs mount parameters:
noatime,nodiratime,logbufs=8,logdev=/dev/rootvg/vicepalog

Result so far:

Version 1.03e ------Sequential Output------ --Sequential Input-
--Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
gourami-a 32088M 442872 60 103527 26 561788 73 994.7 3
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 8313 21 +++++ +++ 6130 15 7618 22 +++++ +++ 4533 12
gourami-a,32088M,,,442872,60,103527,26,,,561788,73,994.7,3,16,8313,21,+++++,+++,6130,15,7618,22,+++++,+++,4533,12

I like the 4533 deletes per second. Untuned that number was 650.

If you see any errors in the above or have more suggestions, please
don't be shy.

Harald.
_______________________________________________ OpenAFS-info mailing
list OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info