[OpenAFS] Re: bonnie++ on OpenAFS
Achim Gsell
achim.gsell@psi.ch
Mon, 22 Nov 2010 18:53:26 +0100
--Apple-Mail-2-756078029
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
On Nov 22, 2010, at 6:14 PM, Andrew Deason wrote:
> On Mon, 22 Nov 2010 18:00:41 +0100
> Achim Gsell <achim.gsell@psi.ch> wrote:
>=20
>> Cache type: memory
>=20
> I'm not saying this is a problem, but does the same thing happen with =
a
> disk cache?
Same problem ...
>=20
> Some stats might help say what's wrong. 'rxdebug <client> 7001 =
-rxstats'
> and 'rxdebug <server> -rxstats' before and after. Also 'xstat_fs_test
> <server> -collID 3 -onceonly' before and after, but I don't think =
that's
> it.
Output of commands is attached.
>=20
>> Server:
>> Scientific Linux 5, kernel 2.6.18-194.17.4.el5
>> OpenAFS 1.4.12.1
>>=20
>> Server and clients are test-systems without any other load.
>=20
> Server and client parameters?
/usr/vice/etc/afsd -cachedir /var/cache/openafs -blocks 1048576 -rootvol =
root.afs -mountdir /afs -afsdb -dynroot -fakestat-all -nosettime =
-chunksize 18 [-memcache]
/fileserver -p 128 -b 512 -l 2048 -s 2048 -vc 2048 -cb 512000 -rxpck =
2048 -sendsize 2097152 -udpsize 2097152 -syslog -d 0 -auditlog =
/var/opt/openafs-server/fileserver.audit -allow-dotted-principals
So long
Achim
--Apple-Mail-2-756078029
Content-Disposition: attachment;
filename=rxdebug-client.after
Content-Type: application/octet-stream;
name="rxdebug-client.after"
Content-Transfer-Encoding: 7bit
Trying 129.129.190.156 (port 7001):
Free packets: 129, packet reclaims: 1, calls: 5604, used FDs: 64
not waiting for packets.
0 calls waiting for a thread
1 threads are idle
rx stats: free packets 129, allocs -1076888308, alloc-failures(rcv 0/0,send 0/0,ack 0)
greedy 0, bogusReads 0 (last from host 0), noPackets 0, noBuffers 0, selects 0, sendSelects 0
packets read: data 3282193504 ack 4053303469 busy 0 abort 13824 ackall 0 challenge 611 response 0 debug 45 params 0 unused 0 unused 0 unused 0 version 0
other read counters: data 3282193503, ack 4052189516, dup 2 spurious 1113909 dally 45
packets sent: data 3379644563 ack 851210257 busy 0 abort 19 ackall 0 challenge 0 response 611 debug 0 params 0 unused 0 unused 0 unused 0 version 0
other send counters: ack 851210257, data 2464321830 (not resends), resends 2192, pushed 0, acked&ignored 2298233150
(these should be small) sendFailed 0, fatalErrors 6
Average rtt is 0.038, with 690545813 samples
Minimum rtt is 0.000, maximum is 1.167
3 server connections, 2668 client connections, 12 peer structs, 18 call structs, 11 free call structs
Connection from host 192.168.16.55, port 7000, Cuid a1bc7a5c/32e54824
serial 5, natMTU 1444, flags DESTROYED, security index 0, client conn
call 0: # 1, state dally, mode: error
call 1: # 0, state not initialized
call 2: # 0, state not initialized
call 3: # 0, state not initialized
Connection from host 192.168.4.79, port 7000, Cuid a1bc7a5c/32e54828
serial 5, natMTU 1444, flags DESTROYED, security index 0, client conn
call 0: # 1, state dally, mode: error
call 1: # 0, state not initialized
call 2: # 0, state not initialized
call 3: # 0, state not initialized
Connection from host 129.129.189.59, port 7000, Cuid a1bc7a5c/7e64b568
serial 9263364, natMTU 1444, flags pktCksum, security index 2, client conn
rxkad: level clear, flags pktCksum
Received 1574924 bytes in 14530 packets
Sent 201079480 bytes in 9248749 packets
call 0: # 14529, state active, mode: sending, flags: window_send, has_output_packets
call 1: # 0, state not initialized
call 2: # 0, state not initialized
call 3: # 0, state not initialized
Connection from host 129.129.189.59, port 7000, Cuid a1bc7a5c/32e54684
serial 9, natMTU 1444, security index 0, client conn
call 0: # 4, state dally, mode: receiving, flags: receive_done
call 1: # 0, state not initialized
call 2: # 0, state not initialized
call 3: # 0, state not initialized
Connection from host 129.129.190.143, port 7000, Cuid a1bc7a5c/32e54688
serial 8, natMTU 1444, security index 0, client conn
call 0: # 3, state dally, mode: receiving, flags: receive_done
call 1: # 0, state not initialized
call 2: # 0, state not initialized
call 3: # 0, state not initialized
Done.
--Apple-Mail-2-756078029
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=us-ascii
--Apple-Mail-2-756078029
Content-Disposition: attachment;
filename=rxdebug-client.before
Content-Type: application/octet-stream;
name="rxdebug-client.before"
Content-Transfer-Encoding: 7bit
Trying 129.129.190.156 (port 7001):
Free packets: 101, packet reclaims: 1, calls: 5601, used FDs: 64
not waiting for packets.
0 calls waiting for a thread
1 threads are idle
rx stats: free packets 101, allocs -1081504850, alloc-failures(rcv 0/0,send 0/0,ack 0)
greedy 0, bogusReads 0 (last from host 0), noPackets 0, noBuffers 0, selects 0, sendSelects 0
packets read: data 3282179362 ack 4050962249 busy 0 abort 13824 ackall 0 challenge 608 response 0 debug 41 params 0 unused 0 unused 0 unused 0 version 0
other read counters: data 3282179361, ack 4049848541, dup 2 spurious 1113664 dally 45
packets sent: data 3375056294 ack 851196107 busy 0 abort 19 ackall 0 challenge 0 response 608 debug 0 params 0 unused 0 unused 0 unused 0 version 0
other send counters: ack 851196107, data 2455145292 (not resends), resends 2174, pushed 0, acked&ignored 2297837994
(these should be small) sendFailed 0, fatalErrors 6
Average rtt is 0.041, with 641521206 samples
Minimum rtt is 0.000, maximum is 1.167
3 server connections, 2663 client connections, 12 peer structs, 18 call structs, 15 free call structs
Connection from host 129.129.189.59, port 7000, Cuid a1bc7a5c/7e64b568
serial 4661057, natMTU 1444, flags pktCksum, security index 2, client conn
rxkad: level clear, flags pktCksum
Received 53096 bytes in 439 packets
Sent -1991063816 bytes in 4660508 packets
call 0: # 438, state active, mode: sending, flags:, has_output_packets
call 1: # 0, state not initialized
call 2: # 0, state not initialized
call 3: # 0, state not initialized
Done.
--Apple-Mail-2-756078029
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=us-ascii
--Apple-Mail-2-756078029
Content-Disposition: attachment;
filename=rxdebug-server.after
Content-Type: application/octet-stream;
name="rxdebug-server.after"
Content-Transfer-Encoding: 7bit
Trying 129.129.189.59 (port 7000):
Free packets: 2561, packet reclaims: 0, calls: 760470, used FDs: 24
not waiting for packets.
0 calls waiting for a thread
121 threads are idle
rx stats: free packets 2561, allocs 108179310, alloc-failures(rcv 0/0,send 0/0,ack 0)
greedy 0, bogusReads 0 (last from host 0), noPackets 0, noBuffers 0, selects 0, sendSelects 0
packets read: data 96260307 ack 7846812 busy 0 abort 2 ackall 0 challenge 25 response 19 debug 180 params 0 unused 0 unused 0 unused 0 version 0
other read counters: data 96260307, ack 7846810, dup 0 spurious 2 dally 0
packets sent: data 10842645 ack 50011087 busy 0 abort 2 ackall 0 challenge 19 response 25 debug 0 params 0 unused 0 unused 0 unused 0 version 0
other send counters: ack 50011087, data 21685290 (not resends), resends 110, pushed 0, acked&ignored 28594142
(these should be small) sendFailed 0, fatalErrors 0
Average rtt is 0.000, with 108182812 samples
Minimum rtt is 0.000, maximum is 0.959
7 server connections, 68 client connections, 10 peer structs, 11 call structs, 10 free call structs
Connection from host 129.129.190.156, port 7001, Cuid a1bc7a5c/7e64b568
serial 4714610, natMTU 1444, flags pktCksum, security index 2, server conn
rxkad: level clear, flags authenticated pktCksum, expires in 25.2 hours
Received 201271248 bytes in 9248885 packets
Sent 1575032 bytes in 14531 packets
call 0: # 14530, state active, mode: receiving, has_input_packets
call 1: # 0, state not initialized
call 2: # 0, state not initialized
call 3: # 0, state not initialized
Done.
--Apple-Mail-2-756078029
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=us-ascii
--Apple-Mail-2-756078029
Content-Disposition: attachment;
filename=rxdebug-server.before
Content-Type: application/octet-stream;
name="rxdebug-server.before"
Content-Transfer-Encoding: 7bit
Trying 129.129.189.59 (port 7000):
Free packets: 3646, packet reclaims: 0, calls: 746373, used FDs: 19
not waiting for packets.
0 calls waiting for a thread
121 threads are idle
rx stats: free packets 3646, allocs 103576977, alloc-failures(rcv 0/0,send 0/0,ack 0)
greedy 0, bogusReads 0 (last from host 0), noPackets 0, noBuffers 0, selects 0, sendSelects 0
packets read: data 91672077 ack 7832710 busy 0 abort 2 ackall 0 challenge 24 response 18 debug 176 params 0 unused 0 unused 0 unused 0 version 0
other read counters: data 91672077, ack 7832708, dup 0 spurious 2 dally 0
packets sent: data 10828543 ack 47669873 busy 0 abort 2 ackall 0 challenge 18 response 24 debug 0 params 0 unused 0 unused 0 unused 0 version 0
other send counters: ack 47669873, data 21657086 (not resends), resends 110, pushed 0, acked&ignored 28594140
(these should be small) sendFailed 0, fatalErrors 0
Average rtt is 0.000, with 108182810 samples
Minimum rtt is 0.000, maximum is 0.959
5 server connections, 68 client connections, 10 peer structs, 11 call structs, 10 free call structs
Connection from host 129.129.190.156, port 7001, Cuid a1bc7a5c/7e64b568
serial 2359304, natMTU 1444, flags pktCksum, security index 2, server conn
rxkad: level clear, flags authenticated pktCksum, expires in 25.3 hours
Received -1990838672 bytes in 4660667 packets
Sent 53096 bytes in 439 packets
call 0: # 438, state active, mode: receiving, flags: reader_wait
call 1: # 0, state not initialized
call 2: # 0, state not initialized
call 3: # 0, state not initialized
Done.
--Apple-Mail-2-756078029
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=us-ascii
--Apple-Mail-2-756078029
Content-Disposition: attachment;
filename=xstat_fs_test.after
Content-Type: application/octet-stream;
name="xstat_fs_test.after"
Content-Transfer-Encoding: 7bit
Starting up the xstat_fs service, no debugging, one-shot operation
------------------------------------------------------------
32844 DeleteFiles
32908 DeleteCallBacks
703599 BreakCallBacks
87636 AddCallBack
0 GotSomeSpaces
8 DeleteAllCallBacks
9 nFEs
9 nCBs
512000 nblks
0 CBsTimedOut
0 nbreakers
0 GSS1
0 GSS2
0 GSS3
0 GSS4
0 GSS5
--Apple-Mail-2-756078029
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=us-ascii
--Apple-Mail-2-756078029
Content-Disposition: attachment;
filename=xstat_fs_test.before
Content-Type: application/octet-stream;
name="xstat_fs_test.before"
Content-Transfer-Encoding: 7bit
Starting up the xstat_fs service, no debugging, one-shot operation
------------------------------------------------------------
32844 DeleteFiles
32908 DeleteCallBacks
689505 BreakCallBacks
87633 AddCallBack
0 GotSomeSpaces
8 DeleteAllCallBacks
9 nFEs
9 nCBs
512000 nblks
0 CBsTimedOut
0 nbreakers
0 GSS1
0 GSS2
0 GSS3
0 GSS4
0 GSS5
--Apple-Mail-2-756078029--