[OpenAFS] OpenAFS performance
Matthias Gerstner
matthias.gerstner@esolutions.de
Mon, 22 Nov 2010 19:20:30 +0100
Thanks for all your hints regarding file server configuration etc.
I've now boosted up parameters on the server and a number of clients and
run that new setup for about a week. Unfortunately the lags during
high-load scenarious are still quite the same.
My file server parameters now look like this:
parm /usr/libexec/openafs/fileserver -L -busyat 1200 -p 128 -cb 500000 -b 480 -l 1200 -vc 2000 -s 1200 -rxpck 400
My particular client runs with the following:
-fakestat -stat 20000 -dcache 6000 -daemons 6 -volumes 256 -chunksize 20 -files 50000
I've attached some first output from rxdebug and xstat_fs_test. The
output from rxdebug was during the time lags have been occuring. From
what I can see there are plenty ressources now on the server. However
I'm not quite sure about interpreting all the numbers.
The xstat_fs_test shows quite a high number of CBsTimedOut. I don't know
if that might be a problem. Regarding NAT or firewalls with low UDP
timeouts I don't think that there is any of those in the network path
between my servers and clients. But I need to check that in detail to be
sure.
When looking at the server at time of lags I see no lack in in CPU or
memory ressources. I started thinking that simply the I/O to disk may be
the problem. However hdparm gives me sequentials reads of about 80
MB/s. Even when with random access I'm only getting a fraction of that it
should not cause such lags.
If anybody sees something suspicious about the attached output or has
another idea what might be the bottleneck here I'll appreciate it.
Best regards,
Matthias
Output from "rxdebug <server> -rxstats"
------------------------------------------------------------------------
Trying 10.2.32.201 (port 7000):
Free packets: 2954, packet reclaims: 13, calls: 8192237, used FDs: 64
not waiting for packets.
0 calls waiting for a thread
118 threads are idle
rx stats: free packets 2954, allocs 70487333, alloc-failures(rcv 0/0,send 0/0,ack 0)
greedy 0, bogusReads 0 (last from host 0), noPackets 0, noBuffers 0, selects 0, sendSelects 0
packets read: data 47655984 ack 16231022 busy 0 abort 2 ackall 0 challenge 1273 response 986 debug 20056 params 0 unused 0 unused 0 unused 0 version 0
other read counters: data 47655984, ack 16230917, dup 60 spurious 105 dally 0
packets sent: data 21182653 ack 21283872 busy 0 abort 2765 ackall 0 challenge 986 response 1273 debug 0 params 0 unused 0 unused 0 unused 0 version 0
other send counters: ack 21283872, data 42365306 (not resends), resends 1448, pushed 0, acked&ignored 19151459
(these should be small) sendFailed 0, fatalErrors 15
Average rtt is 0.000, with 109098699 samples
Minimum rtt is 0.000, maximum is 1.408
35 server connections, 140 client connections, 17 peer structs, 41 call structs, 19 free call structs
Connection from host 10.2.32.143, port 7001, Cuid b98c1781/3393d8b4
serial 423, natMTU 1444, security index 0, client conn
call 0: # 143, state dally, mode: receiving, flags: receive_done
call 1: # 0, state not initialized
call 2: # 0, state not initialized
call 3: # 0, state not initialized
Connection from host 10.2.32.138, port 7001, Cuid b98c1781/3393d8cc
serial 307, natMTU 1444, security index 0, client conn
call 0: # 104, state dally, mode: receiving, flags: receive_done
call 1: # 0, state not initialized
call 2: # 0, state not initialized
call 3: # 0, state not initialized
Connection from host 10.2.32.203, port 7001, Cuid b98c1781/3393d584
serial 32014, natMTU 1444, security index 0, client conn
call 0: # 16006, state dally, mode: receiving, flags: receive_done
call 1: # 2, state not initialized
call 2: # 0, state not initialized
call 3: # 0, state not initialized
Connection from host 10.2.32.200, port 7001, Cuid b98c1781/3393d5ac
serial 8819, natMTU 1444, security index 0, client conn
call 0: # 3638, state dally, mode: receiving, flags: receive_done
call 1: # 0, state not initialized
call 2: # 0, state not initialized
call 3: # 0, state not initialized
Connection from host 10.2.32.203, port 7001, Cuid ac3cda4e/4766e670
serial 1447428, natMTU 1444, flags pktCksum, security index 2, server conn
rxkad: level clear, flags authenticated pktCksum, expires in 1.7 hours
Received 1146258928 bytes in 891775 packets
Sent 1322282322 bytes in 1006648 packets
call 0: # 418268, state active, mode: receiving, flags: receive_done
call 1: # 44156, state not initialized
call 2: # 29096, state not initialized
call 3: # 19207, state not initialized
Connection from host 10.2.32.171, port 7001, Cuid 9dd5d048/d72f9ff0
serial 12886, natMTU 1444, flags pktCksum, security index 2, server conn
rxkad: level crypt, flags authenticated pktCksum, expires in 1.7 hours
Received 24157384 bytes in 18524 packets
Sent 3578950 bytes in 3890 packets
call 0: # 3624, state active, mode: receiving, flags: receive_done
call 1: # 15, state not initialized
call 2: # 11, state not initialized
call 3: # 4, state not initialized
Connection from host 10.2.32.203, port 7001, Cuid ac3cda4e/4766e698
serial 4302086, natMTU 1444, flags pktCksum, security index 2, server conn
rxkad: level clear, flags authenticated pktCksum, expires in 2.6 hours
Received 700197298 bytes in 4068193 packets
Sent -1651088545 bytes in 2374669 packets
call 0: # 531652, state active, mode: sending, flags: receive_done, has_output_packets
call 1: # 46254, state active, mode: receiving, has_input_packets
call 2: # 15481, state not initialized
call 3: # 7715, state not initialized
Connection from host 10.2.32.171, port 7001, Cuid b98c1781/3393d848
serial 640, natMTU 1444, security index 0, client conn
call 0: # 320, state dally, mode: receiving, flags: receive_done
call 1: # 0, state not initialized
call 2: # 0, state not initialized
call 3: # 0, state not initialized
Done.
------------------------------------------------------------------------
Output from "xstat_fs_test <server> 3 -onceonly
------------------------------------------------------------------------
Starting up the xstat_fs service, no debugging, one-shot operation
------------------------------------------------------------
261434 DeleteFiles
1437017 DeleteCallBacks
1329159 BreakCallBacks
2191577 AddCallBack
0 GotSomeSpaces
12 DeleteAllCallBacks
14605 nFEs
16606 nCBs
500000 nblks
86359 CBsTimedOut
0 nbreakers
0 GSS1
0 GSS2
0 GSS3
0 GSS4
0 GSS5
------------------------------------------------------------------------
--
Matthias Gerstner, Dipl.-Wirtsch.-Inf. (FH), Senior Software Engineer
e.solutions GmbH
Am Wolfsmantel 46, 91058 Erlangen, Germany
Registered Office:
Pascalstr. 5, 85057 Ingolstadt, Germany
Phone +49-8458-3332-672, mailto:Matthias.Gerstner@esolutions.de
Fax +49-8458-3332-20672
e.solutions GmbH
Managing Directors Uwe Reder, Dr. Riclef Schmidt-Clausen
Register Court Ingolstadt HRB 5221