[OpenAFS] Slow AFS performance

Jason Edgecombe jason@rampaginggeek.com
Thu, 29 Nov 2007 15:26:03 -0500


Dale Ghent wrote:
> On Nov 28, 2007, at 9:19 AM, Jason Edgecombe wrote:
>
>> I'm experiencing poor AFS performance on under Sparc solaris 9 09/05HW
>> running Openafs server 1.4.1 on a Sun StorageTeck 3511 Fibre channel  to
>> SATA array
>>
>> At first, I thought that having UFS logging disabled was the culprit,
>> but I have enabled UFS logging and I am using the namei server, but
>> performance still stinks. It took 1.5 hours to move a 3.2GB volume to
>> the server. Things seem fine except on the Fibre channel disks.
>
> ...
>
>> My bonnie++ performance numbers for vicepa are here:
>> http://www.coe.uncc.edu/~jwedgeco/bonnie.html
>>
>> What could AFS be doing that causes the performance to stink?
>
> Well first, your bonnie run tested sequential IO, so of course you're
> going to get stellar numbers with sequential large writes and reads.
> Arrays and disks in general eat that stuff right on up.
>
> AFS namei does writes in small batches, up to 64k, and all of it
> random (think: lots of users accessing lots of small files) It would
> be advantageous to tun to this environment.
>
> Check out your 3511s first.
>
> 1) Make sure write-back caching is on. This is a global as well as a
> per-Logical Drive setting on the 3510/3511.
>
> 2) Optimized for Random I/O (be sure the latest 3511 fw is applied, it
> betters the performance of this setting)
>
> 3) You didn't mention the type of RAID config you have on the 3511s.
> RAID5? If so, is the stripe size low (64k?)
>
> Some 'iostat -nx 1' output would be helpful, too. The wsvc_t and %b
> columns would be telling.
>
> /dale
>
ok, I'm now some better performance out of my FC array. "vos move" of a
4.4GB volume from one disk in the FC array to another disk in the array
only took 16 minutes (4.6MB/s).

Derrick's suggestion of upgrading to 1.4.5 with namei did the trick.

Unfortunately, I don't think I can upgrade the firmware or tune the 3511
array. It's the 3511 expansion unit without the raid controller. It's a
JBOD.


As requested, here are the statics from "iostat -nx 1"
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0   13.0    0.0    8.0  0.0  0.1    0.0    9.8   0   2 c1t0d0
  481.0    0.0 14905.1    0.0  0.0  0.6    0.0    1.3   0  50 c0t0d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t1d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t3d0
   13.0  133.0  104.0 12082.3  0.0  3.6    0.0   25.0   0  86 c0t2d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t5d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t8d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t7d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t4d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t6d0

...
                    extended device statistics             
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t0d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t0d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t1d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t3d0
    0.0  333.0    0.0  581.0 384.2 19.0 1153.7   57.1 100 100 c0t2d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t5d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t8d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t7d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t4d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t6d0
                    extended device statistics             
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t0d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t0d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t1d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t3d0
    0.0  339.0    0.0  773.0 371.3 19.0 1095.4   56.0 100 100 c0t2d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t5d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t8d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t7d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t4d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t6d0


Jason