[OpenAFS] Openafs 1.4.2 on Debian Etch kernel 2.6.18 slow

Derek Harkness dharknes@umd.umich.edu
Mon, 12 Feb 2007 23:03:35 -0500


This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--Apple-Mail-2--241312735
Content-Type: multipart/alternative; boundary=Apple-Mail-1--241312799


--Apple-Mail-1--241312799
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=US-ASCII;
	delsp=yes;
	format=flowed

Thanks for the help but I've identified the problem.  The problem was  
an interaction between AFS, reiserfs, and the XRaid.  The XRaid has a  
setting that allows the OS to flush the controller and disk cache.   
It appears that AFS and reiserfs were flushing on each block, where  
as JFS flushed the cache at file close.  I disabled that setting and  
the performance of the XRaid jumped to exactly what it should have  
been.  Thanks again for the help.

Derek Harkness
System Administrator
University of Michigan-Dearborn
(313) 593-5089


> I've got a bunch of questions.  Even if you only have time to answer a
> few of them, it will help us to narrow down the root cause.
>
> First and foremost, do local volume package operations (e.g. the
> salvager, vos backup, fileserver startup/shutdown, etc) run slowly, or
> is it only stuff that involves Rx?  What about vos dump foo localhost
> on the ailing fileserver?  The fact that iowait is going through the
> roof may be indicative of an io subsystem problem, so eliminating
> network/Rx problems at the top of the decision tree will be useful.

Salvager and vos backup running locally are slow.  They don't drive  
iowait as high but certainly don't get the throughput they should

Below is an iostat for my AFS partitions, I was salvaging the /vicepa  
partition which had 5 dead clone volumes that were deleted.  I have 4  
partitions on 2 raid devices (sda, sdb) sda1 -> /vicepa, sda2 -> / 
vicepb, sdb1 -> /vicepc, sdb2 - > /vicepd.  vicep[abc] are formated  
with reiserfs and vicepd is format jfs.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.05    0.00    0.00    1.95    0.00   98.00

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               5.19         0.00        31.14          0        156
sda2              0.00         0.00         0.00          0          0
sda1              7.78         0.00        31.14          0        156
sdb               0.00         0.00         0.00          0          0
sdb2              0.00         0.00         0.00          0          0
sdb1              0.00         0.00         0.00          0          0

> I'm not familiar with the Linux iostat utility, but if it supports
> per-disk stats similar similar to the -x option on Solaris, or the -D
> option on AIX, then please post some data while the problem is
> occurring.
>
> * Were you running some well-known benchmark suite?  If so, what
> options did you pass?

I went back and checked and yes my original test method was only  
hitting the page cache.  But here are some new #s.

Read test
hdparm -t /dev/sdb2
Buffered read: 354 MB in  3.01 seconds = 117.54 MB/sec

bonnie++ benchmark tool
Version  1.03
------Sequential Output------ --Sequential Input- --Random--Per Chr-  
--Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine 	Size	K/sec 	%CP 	K/sec 	%CP 	K/sec 	%CP 	K/sec 	%CP 	K/sec 	% 
CP  	K/sec 	%CP
thales		8G		40379  	91 		57465  	24  		3870   	1  		9779  	22 		 
97536  	16 		195.3   	0

------Sequential Create------ --------Random Create---------Create--  
--Read--- -Delete-- -Create-- --Read--- -Delete--
	files  	K/sec 	%CP  	K/sec 	%CP  	K/sec 	%CP  	K/sec 	%CP 	K/sec 	% 
CP  	K/sec 	%CP
	16 		25563  	99 		+++++ 	+++ 	20961 	100 	24985 	100 	+++++ 	+++ 	 
18984 	100

thales,8G, 
40379,91,57465,24,3870,1,9779,22,97536,16,195.3,0,16,25563,99,+++++,++ 
+,20961,100,24985,100,+++++,+++,18984,100

> * Did it involve one file or many?

8 files

> * Were any fsync()s issued?

Yes

> * Did it modify any filesystem metadata, or only file data?

Only file data

> * Was it single threaded or multi-threaded?
Single

> * How much data was read/written?
About 8 gigs

> * How big were the files involved?

1 Gig each

> * Did you do anything to mitigate/bypass caching?



> Other questions that might be useful:
>
> * How deep are the tagged command queues for the xserve lun(s)?

Don't know but I will check

> * Do all the disks pass surface scans?

I will run a surface test over night.

> * Are the disks and/or controllers reporting SMART events?

No smart events, no other raid events reported

> * If this stuff is fabric attached, have you looked at port error
> counts, port performance data, etc?

XRaid is directly connected to the server both

> How have you verified that "network performance" is ok?  What are the
> ethernet port error counts like?  What are the packet retransmit rates
> like?

RX packets:1065818 errors:0 dropped:0 overruns:0 frame:0
TX packets:1257865 errors:0 dropped:0 overruns:0 carrier:0

NPtcp network bandwidth test reports ~90 Mbits/sec average throughput.

> I don't know much of anything about apple's storage line, but if they
> have any sort of performance analysis and/or problem determination
> tools, what do they say?

I will run tonight

>
>> the some problem is the AFS fileserver.
>>
>> Hardware:
>> HP DL380
>> 2x2.8ghz Hyperthreaded Xeon CPU
>> 4 Gigs of RAM
>> Gigabit ethernet
>> MPTFusion fiber channel card
>> Apple XRaid
>>
>> I've got 2 other identical box currently run AFS and working  
>> fine.  The only
>> difference is the other boxes are running an old OS.
>>
>
> Are the machines running the older kernel still running 1.3.x?

Yes, 1.3.81.

> Until we can better understand your testing methodology, I'd have to
> say this could be a hardware problem, a kernel driver problem, an AFS
> problem, or even a network problem.  We need more information to
> narrow it down.
>
> Regards,
>
> -- 
> Tom Keiser
> tkeiser@gmail.com


--Apple-Mail-1--241312799
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=ISO-8859-1

<HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode: space; =
-khtml-line-break: after-white-space; "><DIV><DIV><DIV><SPAN =
class=3D"Apple-style-span" style=3D"border-collapse: separate; =
border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; =
font-size: 12px; font-style: normal; font-variant: normal; font-weight: =
normal; letter-spacing: normal; line-height: normal; text-align: auto; =
-khtml-text-decorations-in-effect: none; text-indent: 0px; =
-apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
white-space: normal; widows: 2; word-spacing: 0px; =
"></SPAN></DIV><DIV><DIV>Thanks for the help but I've identified the =
problem.=A0 The problem was an interaction between AFS, reiserfs, and =
the XRaid.=A0 The XRaid has a setting that allows the OS to flush the =
controller and disk cache.=A0 It appears that AFS and reiserfs were =
flushing on each block, where as JFS flushed the cache at file close.=A0 =
I disabled that setting and the performance of the XRaid jumped to =
exactly what it should have been.=A0 Thanks again for the =
help.</DIV><DIV><BR><DIV><DIV>Derek Harkness</DIV><DIV>System =
Administrator</DIV><DIV>University of Michigan-Dearborn</DIV><DIV>(313) =
593-5089</DIV></DIV></DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><BR><BLOCKQUOTE type=3D"cite"><DIV=
 style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">I've got a bunch of questions.<SPAN =
class=3D"Apple-converted-space">=A0 </SPAN>Even if you only have time to =
answer a</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">few of them, it will help us to =
narrow down the root cause.</DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: =
14px; "><BR></DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">First and foremost, do local =
volume package operations (e.g. the</DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">salvager, vos =
backup, fileserver startup/shutdown, etc) run slowly, or</DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">is it only stuff that involves Rx?<SPAN =
class=3D"Apple-converted-space">=A0 </SPAN>What about vos dump foo =
localhost</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">on the ailing fileserver?<SPAN =
class=3D"Apple-converted-space">=A0 </SPAN>The fact that iowait is going =
through the</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">roof may be indicative of an io =
subsystem problem, so eliminating</DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">network/Rx =
problems at the top of the decision tree will be =
useful.</DIV></BLOCKQUOTE><DIV><BR =
class=3D"khtml-block-placeholder"></DIV>Salvager and vos backup running =
locally are slow.=A0 They don't drive iowait as high but=A0certainly=A0don=
't get the throughput they should<DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>Below is an iostat for my =
AFS partitions, I was salvaging the /vicepa partition which had 5 dead =
clone volumes that were deleted.=A0 I have 4 partitions on 2 raid =
devices (sda, sdb) sda1 -&gt; /vicepa, sda2 -&gt; /vicepb, sdb1 -&gt; =
/vicepc, sdb2 - &gt; /vicepd.=A0 vicep[abc] are formated with reiserfs =
and vicepd is format jfs.</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>avg-cpu:=A0 %user=A0=A0 =
%nice %system %iowait=A0 %steal=A0=A0 %idle</DIV><DIV>=A0 =A0 =A0 =A0 =A0=A0=
 0.05=A0 =A0 0.00=A0 =A0 0.00=A0 =A0 1.95=A0 =A0 0.00=A0=A0 =
98.00</DIV><DIV><BR class=3D"khtml-block-placeholder"></DIV><DIV>Device:=A0=
 =A0 =A0 =A0 =A0 =A0 tps=A0 =A0 kB_read/s=A0 =A0 kB_wrtn/s=A0 =A0 =
kB_read=A0 =A0 kB_wrtn</DIV><DIV>sda=A0 =A0 =A0 =A0 =A0 =A0 =A0=A0 5.19=A0=
 =A0 =A0 =A0=A0 0.00=A0 =A0 =A0 =A0 31.14=A0 =A0 =A0 =A0 =A0 0=A0 =A0 =A0 =
=A0 156</DIV><DIV>sda2=A0 =A0 =A0 =A0 =A0 =A0 =A0 0.00=A0 =A0 =A0 =A0=A0 =
0.00=A0 =A0 =A0 =A0=A0 0.00=A0 =A0 =A0 =A0 =A0 0=A0 =A0 =A0 =A0 =A0 =
0</DIV><DIV>sda1=A0 =A0 =A0 =A0 =A0 =A0 =A0 7.78=A0 =A0 =A0 =A0=A0 0.00=A0=
 =A0 =A0 =A0 31.14=A0 =A0 =A0 =A0 =A0 0=A0 =A0 =A0 =A0 =
156</DIV><DIV>sdb=A0 =A0 =A0 =A0 =A0 =A0 =A0=A0 0.00=A0 =A0 =A0 =A0=A0 =
0.00=A0 =A0 =A0 =A0=A0 0.00=A0 =A0 =A0 =A0 =A0 0=A0 =A0 =A0 =A0 =A0 =
0</DIV><DIV>sdb2=A0 =A0 =A0 =A0 =A0 =A0 =A0 0.00=A0 =A0 =A0 =A0=A0 0.00=A0=
 =A0 =A0 =A0=A0 0.00=A0 =A0 =A0 =A0 =A0 0=A0 =A0 =A0 =A0 =A0 =
0</DIV><DIV>sdb1=A0 =A0 =A0 =A0 =A0 =A0 =A0 0.00=A0 =A0 =A0 =A0=A0 0.00=A0=
 =A0 =A0 =A0=A0 0.00=A0 =A0 =A0 =A0 =A0 0=A0 =A0 =A0 =A0 =A0 =
0</DIV><BR><BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I'm not =
familiar with the Linux iostat utility, but if it supports</DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">per-disk stats similar similar to the -x option on =
Solaris, or the -D</DIV><DIV style=3D"margin-top: 0px; margin-right: =
0px; margin-bottom: 0px; margin-left: 0px; ">option on AIX, then please =
post some data while the problem is</DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
">occurring.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">* Were you running some well-known benchmark =
suite?<SPAN class=3D"Apple-converted-space">=A0 </SPAN>If so, =
what</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">options did you =
pass?</DIV></BLOCKQUOTE><DIV><BR class=3D"khtml-block-placeholder"></DIV>I=
 went back and checked and yes my=A0original=A0test method was only =
hitting the page cache.=A0 But here are some new #s.</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>Read test</DIV><DIV>hdparm =
-t /dev/sdb2</DIV><DIV>Buffered read: 354 MB in=A0 3.01 seconds =3D =
117.54 MB/sec</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>bonnie++ benchmark =
tool</DIV><DIV>Version=A0 1.03=A0 =A0 =A0=A0=A0</DIV><DIV>------Sequential=
 Output------ --Sequential Input- --Random--Per Chr- --Block-- -Rewrite- =
-Per Chr- --Block-- --Seeks--</DIV><DIV>Machine <SPAN =
class=3D"Apple-tab-span" style=3D"white-space:pre">	</SPAN>Size<SPAN =
class=3D"Apple-tab-span" style=3D"white-space:pre">	</SPAN>K/sec =
<SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>%CP <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>K/sec <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>%CP <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>K/sec <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>%CP <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>K/sec <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>%CP <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>K/sec <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>%CP=A0 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>K/sec <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>%CP</DIV><DIV>thales<SPAN class=3D"Apple-tab-span" =
style=3D"white-space:pre">		</SPAN>8G<SPAN =
class=3D"Apple-tab-span" style=3D"white-space:pre">		=
</SPAN>40379=A0 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>91 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">		=
</SPAN>57465=A0 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>24=A0 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">		=
</SPAN>3870=A0=A0 <SPAN class=3D"Apple-tab-span" =
style=3D"white-space:pre">	</SPAN>1=A0 <SPAN class=3D"Apple-tab-span"=
 style=3D"white-space:pre">		</SPAN>9779=A0 <SPAN =
class=3D"Apple-tab-span" style=3D"white-space:pre">	</SPAN>22 <SPAN =
class=3D"Apple-tab-span" style=3D"white-space:pre">		=
</SPAN>97536=A0 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>16=A0<SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">		=
</SPAN>195.3=A0=A0 <SPAN class=3D"Apple-tab-span" =
style=3D"white-space:pre">	</SPAN>0</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>------Sequential =
Create------ --------Random Create---------Create-- --Read--- -Delete-- =
-Create-- --Read--- -Delete--</DIV><DIV><SPAN class=3D"Apple-tab-span" =
style=3D"white-space:pre">	</SPAN>files=A0 <SPAN =
class=3D"Apple-tab-span" style=3D"white-space:pre">	</SPAN>K/sec =
<SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>%CP=A0 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>K/sec <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>%CP=A0 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>K/sec <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>%CP=A0 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>K/sec <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>%CP <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>K/sec <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>%CP=A0 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>K/sec <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>%CP=A0</DIV><DIV><SPAN class=3D"Apple-tab-span" =
style=3D"white-space:pre">	</SPAN>16 <SPAN class=3D"Apple-tab-span" =
style=3D"white-space:pre">		</SPAN>25563=A0 <SPAN =
class=3D"Apple-tab-span" style=3D"white-space:pre">	</SPAN>99 <SPAN =
class=3D"Apple-tab-span" style=3D"white-space:pre">		=
</SPAN>+++++ <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>+++ <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>20961 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>100 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>24985 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>100 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>+++++ <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>+++ <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>18984 <SPAN class=3D"Apple-tab-span" style=3D"white-space:pre">	=
</SPAN>100</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>thales,8G,40379,91,57465,24,3=
870,1,9779,22,97536,16,195.3,0,16,25563,99,+++++,+++,20961,100,24985,100,+=
++++,+++,18984,100</DIV><DIV><BR><BLOCKQUOTE type=3D"cite"><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">* Did it involve one file or =
many?</DIV></BLOCKQUOTE><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>8 =
files</DIV><BR><BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">* Were any =
fsync()s issued?</DIV></BLOCKQUOTE><BR>Yes</DIV><DIV><BR><BLOCKQUOTE =
type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">* Did it modify any filesystem =
metadata, or only file data?</DIV></BLOCKQUOTE><BR><DIV>Only file =
data</DIV><BR><BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">* Was it =
single threaded or =
multi-threaded?</DIV></BLOCKQUOTE><DIV>Single</DIV><BR><BLOCKQUOTE =
type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">* How much data was =
read/written?</DIV></BLOCKQUOTE><DIV>About 8 gigs</DIV><BR><BLOCKQUOTE =
type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">* How big were the files =
involved?</DIV></BLOCKQUOTE><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>1 Gig =
each</DIV><BR><BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">* Did you do =
anything to mitigate/bypass caching?</DIV></BLOCKQUOTE><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><BR><BLOCKQUOTE type=3D"cite"><DIV=
 style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">Other questions that might be useful:</DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">* How =
deep are the tagged command queues for the xserve =
lun(s)?</DIV></BLOCKQUOTE><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>Don't know but I will =
check</DIV><BR><BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">* Do all the =
disks pass surface scans?</DIV></BLOCKQUOTE><DIV><BR =
class=3D"khtml-block-placeholder"></DIV>I will run a surface test over =
night.</DIV><DIV><BR><BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: =
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">* Are =
the disks and/or controllers reporting SMART =
events?</DIV></BLOCKQUOTE><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>No smart events, no other =
raid events reported</DIV><BR><BLOCKQUOTE type=3D"cite"><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">* If this stuff is fabric attached, have you looked =
at port error</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">counts, port performance data, =
etc?</DIV></BLOCKQUOTE><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>XRaid is directly connected =
to the server both=A0</DIV><BR><BLOCKQUOTE type=3D"cite"><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">How have you verified that "network performance" is =
ok?<SPAN class=3D"Apple-converted-space">=A0 </SPAN>What are =
the</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">ethernet port error counts =
like?<SPAN class=3D"Apple-converted-space">=A0 </SPAN>What are the =
packet retransmit rates</DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
">like?</DIV></BLOCKQUOTE><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>RX packets:1065818 errors:0 =
dropped:0 overruns:0 frame:0</DIV><DIV>TX packets:1257865 errors:0 =
dropped:0 overruns:0 carrier:0</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>NPtcp network bandwidth =
test reports ~90 Mbits/sec average throughput.</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><BLOCKQUOTE type=3D"cite"><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">I don't know much of anything about apple's storage =
line, but if they</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">have any sort of performance =
analysis and/or problem determination</DIV><DIV style=3D"margin-top: =
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">tools, =
what do they say?</DIV></BLOCKQUOTE><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>I will run =
tonight</DIV><DIV><BR class=3D"khtml-block-placeholder"></DIV><BLOCKQUOTE =
type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV> =
<BLOCKQUOTE type=3D"cite"><DIV style=3D"margin-top: 0px; margin-right: =
0px; margin-bottom: 0px; margin-left: 0px; ">the some problem is the AFS =
fileserver.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">Hardware:</DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">HP =
DL380</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">2x2.8ghz Hyperthreaded Xeon =
CPU</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; ">4 Gigs of RAM</DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">Gigabit ethernet</DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">MPTFusion =
fiber channel card</DIV><DIV style=3D"margin-top: 0px; margin-right: =
0px; margin-bottom: 0px; margin-left: 0px; ">Apple XRaid</DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: =
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I've got =
2 other identical box currently run AFS and working fine.<SPAN =
class=3D"Apple-converted-space">=A0 </SPAN>The only</DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">difference is the other boxes are running an old =
OS.</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV> =
</BLOCKQUOTE><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">Are the machines running the older kernel still =
running 1.3.x?</DIV></BLOCKQUOTE><DIV><BR =
class=3D"khtml-block-placeholder"></DIV>Yes, 1.3.81.</DIV><DIV><BR =
class=3D"khtml-block-placeholder"><BLOCKQUOTE type=3D"cite"><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">Until we can better understand your testing =
methodology, I'd have to</DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">say this =
could be a hardware problem, a kernel driver problem, an AFS</DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">problem, or even a network problem.<SPAN =
class=3D"Apple-converted-space">=A0 </SPAN>We need more information =
to</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: =
0px; margin-left: 0px; ">narrow it down.</DIV><DIV style=3D"margin-top: =
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
min-height: 14px; "><BR></DIV><DIV style=3D"margin-top: 0px; =
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; =
">Regards,</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV =
style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; =
margin-left: 0px; ">--<SPAN =
class=3D"Apple-converted-space">=A0</SPAN></DIV><DIV style=3D"margin-top: =
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Tom =
Keiser</DIV><DIV style=3D"margin-top: 0px; margin-right: 0px; =
margin-bottom: 0px; margin-left: 0px; "><A =
href=3D"mailto:tkeiser@gmail.com">tkeiser@gmail.com</A></DIV> =
</BLOCKQUOTE></DIV><BR></DIV></DIV></BODY></HTML>=

--Apple-Mail-1--241312799--

--Apple-Mail-2--241312735
content-type: application/pgp-signature; x-mac-type=70674453;
	name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFF0TiXKvmLCyEduhQRAitEAJ94stdQphDLh6iSvB06tTft1LegBgCeIcu9
4eq3tHzBmxIG/bY3fbOhYoU=
=m3TE
-----END PGP SIGNATURE-----

--Apple-Mail-2--241312735--