[OpenAFS-devel] fileserver parameters
Roland Kuhn
rkuhn@e18.physik.tu-muenchen.de
Mon, 20 Jun 2005 15:28:02 +0200
--Apple-Mail-13--767037878
Content-Type: multipart/mixed; boundary=Apple-Mail-12--767038201
--Apple-Mail-12--767038201
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=US-ASCII;
delsp=yes;
format=flowed
Hi Tom!
On Jun 19, 2005, at 5:03 AM, Tom Keiser wrote:
> On 6/16/05, Roland Kuhn <rkuhn@e18.physik.tu-muenchen.de> wrote:
>
>> Dear experts!
>>
>> We are fighting with the fileserver performance since a long time.
>> Once I got the advice to use the single threaded fileserver, which
>> helped, but didn't get me more than 10MB/s. Now we upgraded to Debian
>> sarge (openafs 1.3.81), which comes again with the threaded server.
>> With default settings we get 1MB/s (the underlying RAID can easily
>> deliver >200MB/s, which shows that the VM settings are okay). Now I
>> tried with -L -vc 10000 -cb 100000 -udpsize 12800, which brings it
>> back to about 6MB/s (all numbers with >>1 simultanous clients
>> reading). This is still a factor 30 below the capabilities of the
>> RAID (okay, we only have 1GB/s ethernet ;-) ). I've seen excessive
>> context switch rates (>>100000/s), which obviously don't happen with
>> the single threaded fileserver.
>>
>> So, can anybody comment on these numbers? Those are dual Opteron
>> boxes with enough RAM, so please make some suggestions what options I
>> should try to get more like the real performance of a fileserver...
>>
>
> This is very interesting. On much much older hardware (2x 300MHz sun
> e450 running solaris 10) I can get >15MB/s aggregate off a single
> fc-al disk with >> 1 clients over gigE with absolutely no tweaking of
> fileserver parameters. Of course, there are many performance
> bottlenecks in multithreading that are actually exacerbated by faster
> cpu's, so the results you're seeing are plausible.
>
Interesting indeed. Thanks for the very detailed answer, it'll take
some time for me to digest it ;-)
Right now I'm using the single threaded fileserver with
-L -vc 10000 -udpsize 128000 -rxpck 400 -busyat 200 -cb 100000
and I'm getting 11MB/s using 8 slow and 3 fast clients. The workload
is sequential reading of 1.5GB big files (thousands of those).
> I'd be interested in seeing a comparison of 1.3.81 and 1.3.84
> performance. Several threading patches were integrated between these
> revisions, and it would be interesting to see how they affect your
> problem. I know on sparc they are making a difference, but that
> doesn't necessarily correllate to amd64.
>
We're not using amd64 yet, still on i386 Debian. But I can easily
compile the 1.3.84 fileserver, unless there are problems to be
expected with 1.2.13 clients (I've not seen any so far with the
1.3.81-sarge1 fileserver).
> If you upgrade to 1.3.84, there is another fileserver option that you
> will want to experiment with: -rxpck. Sometime after 1.3.81,
> thread-local packet queues were integrated, and they may reduce your
> context switch rate due to less contention for the global packet queue
> lock. The default value for -rxpck will give you approximately 500
> rx_packet structures. I recommend trying several values in the range
> 1000-5000. At some point, you will reach an optimal tradeoff between
> a small value that fits within your cache hierarchy, and a large value
> that reduces the number of transfers between the thread-local and
> global rx_packet queues. Before submitting the thread-local patch to
> RT, I was only able to test on a few architectures, and I'd like to
> get feedback for amd64.
>
Sorry, the upgrade to amd64 will only be done in a few months :-(
> Another option you might care to experiment with is: -p . IIRC, the
> default will give you 12 worker threads. It sounds like many of your
> worker threads are busy handling calls, but are constantly contending
> over locks, and blocking on i/o. You will need to experiment with
> this, but you may find that reducing the number of worker threads will
> actually improve performance by forcing new calls to queue up, thereby
> allowing your active calls to complete with less contention. Of
> course, this won't alleviate the problems caused by blocking i/o.
> Reducing this value too far is dangerous because some calls have high
> latencies (e.g. some calls make calls to the ptserver).
>
> Have you looked at the xstat results from your servers? afsmonitor is
> a great little tool, and it can even dump these results periodically
> to a log. This data could help us to understand your workload.
> Seeing those numbers would also help us with suggesting changes to
> parameters in the volume package.
>
Well, I have no idea what the numbers mean, but I've attached the
output of "afsmonitor -detailed -fshost <XXX>". In this case the
workload was such that the computation on the client side limited the
throughput to about 8MB/s. The problem is that this fileserver is
very much used from our cluster and I cannot easily introduce
downtime or change the workload.
Thanks for all your help!
Ciao,
Roland
--Apple-Mail-12--767038201
Content-Transfer-Encoding: quoted-printable
Content-Type: application/octet-stream;
x-unix-mode=0644;
name="afs.log"
Content-Disposition: attachment;
filename=afs.log
=0AMon=20Jun=2020=2015:12:56=202005=20XXX=20FS=202=20600=20893=2085915=20=
36=2011542=20600=208869=20146163=20686=2027497=2010000=2085976=200=20120=20=
73322=2045=2038803191=200=200=200=200=200=200=200=200=2025932245=208620=20=
7793505=2017968559=200=207793505=2017968558=20431=201=2025951494=20=
5060389=200=205060389=200=200=200=2051902988=201445=200=2077944762=20=
254118=20420000=200=200=2014=20650000=2011640667=2042=2020=2023=2037=20=
10=2018=201=2018=207=200=2011=2076=202=201502=200=200=200=200=200=200=20=
0=200=200=200=200=200=200=200=200=200=200=200=200=200=200=200=200=200=20=
0=200=200=200=200=200=200=201119267660=200=2054847=2054841=2022760=20=
212545=2017302=20498945=200=208=204=20409999=203=203=200=2043934=200=20=
981=200=20319=200=2025784=201751=201751=202=20529428=200=20204307=200=20=
3=200=20181252=209856=209851=2025438=20793801=201011434=20467369=200=20=
59=2092=20664678=201=201=200=20397=200=200=200=20397=200=20397=209645=20=
9645=200=20331826=200=20873=200=2010=200=2020294=208=208=200=20541631=20=
0=20250163=200=20392=200=20499645=208830=208830=2029=2082916=200=20=
425345=200=20426=200=20193746=201=201=200=20537=200=200=200=20537=200=20=
537=2039=2039=200=20117115=200=20944=200=20382=200=2020918=200=200=200=20=
0=200=200=20999999=200=200=200=20893=20893=202=20326453=200=2048299=200=20=
369=200=2068759=200=200=200=200=200=200=20999999=200=200=200=200=200=200=20=
0=200=200=20999999=200=200=200=200=200=200=200=200=200=20999999=200=200=20=
0=200=200=200=200=200=200=20999999=200=200=200=200=200=200=200=200=200=20=
999999=200=200=200=20556=20556=200=206557=200=200=200=201=200=2053=200=20=
0=200=200=200=200=20999999=200=200=200=2014=2014=200=2069256=200=204767=20=
0=206=200=2069048=200=200=200=200=200=200=20999999=200=200=200=200=200=20=
0=200=200=200=20999999=200=200=200=200=200=200=200=200=200=20999999=200=20=
0=200=20141=20141=200=20860438=200=2060668=200=201=200=20110030=200=200=20=
0=200=200=200=20999999=200=200=200=203=203=200=2056428=200=201695=200=20=
1006=200=2036645=200=200=200=200=200=200=20999999=200=200=200=202=201=20=
0=2014=200=200=200=2014=200=2014=2054841=2054841=2022757=20535102=20=
17300=20894155=200=203=204=20409978=2035261582=200=201048576=20709=200=20=
1113=208=2041=2019730=2037=2033203=200=209851=209851=2025437=20758515=20=
1011404=2012315=200=2046=2092=20664642=2017650917=201=2067108864=201373=20=
1569=203099=20651=20593=20700=20628=20213=201025=20=0A=0A=
AFS_XSTATSCOLL_FULL_PERF_INFO=20(coll=202)=20for=20FS=20XXX=0A[Probe=20=
1,=20Mon=20Jun=2020=2015:12:56=202005]=0A=0A=09=20=20=20=20=20=20=20=20=20=
2=20numPerfCalls=0A=0A=09=20=20=20=20=20=20=20600=20vcache_L_Entries=0A=09=
=20=20=20=20=20=20=20893=20vcache_L_Allocs=0A=09=20=20=20=20=2085915=20=
vcache_L_Gets=0A=09=20=20=20=20=20=20=20=2036=20vcache_L_Reads=0A=09=20=20=
=20=20=2011542=20vcache_L_Writes=0A=0A=09=20=20=20=20=20=20=20600=20=
vcache_S_Entries=0A=09=20=20=20=20=20=208869=20vcache_S_Allocs=0A=09=20=20=
=20=20146163=20vcache_S_Gets=0A=09=20=20=20=20=20=20=20686=20=
vcache_S_Reads=0A=09=20=20=20=20=2027497=20vcache_S_Writes=0A=0A=09=20=20=
=20=20=2010000=20vcache_H_Entries=0A=09=20=20=20=20=2085976=20=
vcache_H_Gets=0A=09=20=20=20=20=20=20=20=20=200=20vcache_H_Replacements=0A=
=0A=09=20=20=20=20=20=20=20120=20dir_Buffers=0A=09=20=20=20=20=2073322=20=
dir_Calls=0A=09=20=20=20=20=20=20=20=2045=20dir_IOs=0A=0A=09=20=20=
38803191=20rx_packetRequests=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_noPackets_RcvClass=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_noPackets_SendClass=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_noPackets_SpecialClass=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_socketGreedy=0A=09=20=20=20=20=20=20=20=20=200=20rx_bogusPacketOnRead=0A=
=09=20=20=20=20=20=20=20=20=200=20rx_bogusHost=0A=09=20=20=20=20=20=20=20=
=20=200=20rx_noPacketOnRead=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_noPacketBuffersOnRead=0A=09=20=2025932245=20rx_selects=0A=09=20=20=20=20=
=20=208620=20rx_sendSelects=0A=09=20=20=207793505=20=
rx_packetsRead_RcvClass=0A=09=20=2017968559=20rx_packetsRead_SendClass=0A=
=09=20=20=20=20=20=20=20=20=200=20rx_packetsRead_SpecialClass=0A=09=20=20=
=207793505=20rx_dataPacketsRead=0A=09=20=2017968558=20rx_ackPacketsRead=0A=
=09=20=20=20=20=20=20=20431=20rx_dupPacketsRead=0A=09=20=20=20=20=20=20=20=
=20=201=20rx_spuriousPacketsRead=0A=09=20=2025951494=20=
rx_packetsSent_RcvClass=0A=09=20=20=205060389=20rx_packetsSent_SendClass=0A=
=09=20=20=20=20=20=20=20=20=200=20rx_packetsSent_SpecialClass=0A=09=20=20=
=205060389=20rx_ackPacketsSent=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_pingPacketsSent=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_abortPacketsSent=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_busyPacketsSent=0A=09=20=2051902988=20rx_dataPacketsSent=0A=09=20=20=20=
=20=20=201445=20rx_dataPacketsReSent=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_dataPacketsPushed=0A=09=20=2077944762=20rx_ignoreAckedPacket=0A=09=20=20=
=20=20254118=20rx_totalRtt_Sec=0A=09=20=20=20=20420000=20=
rx_totalRtt_Usec=0A=09=20=20=20=20=20=20=20=20=200=20rx_minRtt_Sec=0A=09=20=
=20=20=20=20=20=20=20=200=20rx_minRtt_Usec=0A=09=20=20=20=20=20=20=20=20=
14=20rx_maxRtt_Sec=0A=09=20=20=20=20650000=20rx_maxRtt_Usec=0A=09=20=20=
11640667=20rx_nRttSamples=0A=09=20=20=20=20=20=20=20=2042=20=
rx_nServerConns=0A=09=20=20=20=20=20=20=20=2020=20rx_nClientConns=0A=09=20=
=20=20=20=20=20=20=2023=20rx_nPeerStructs=0A=09=20=20=20=20=20=20=20=20=
37=20rx_nCallStructs=0A=09=20=20=20=20=20=20=20=2010=20=
rx_nFreeCallStructs=0A=0A=09=20=20=20=20=20=20=20=2018=20=
host_NumHostEntries=0A=09=20=20=20=20=20=20=20=20=201=20host_HostBlocks=0A=
=09=20=20=20=20=20=20=20=2018=20host_NonDeletedHosts=0A=09=20=20=20=20=20=
=20=20=20=207=20host_HostsInSameNetOrSubnet=0A=09=20=20=20=20=20=20=20=20=
=200=20host_HostsInDiffSubnet=0A=09=20=20=20=20=20=20=20=2011=20=
host_HostsInDiffNetwork=0A=09=20=20=20=20=20=20=20=2076=20=
host_NumClients=0A=09=20=20=20=20=20=20=20=20=202=20host_ClientBlocks=0A=0A=
=091119267660=20epoch=0A=20=20=20=20=20=20FetchData:=2054847=20ops=20=
(54841=20OK);=20sum=3D22760.212545,=20min=3D0.000008,=20max=3D4.409999=0A=
=20=20=20=20=20=20=20FetchACL:=203=20ops=20(3=20OK);=20sum=3D0.043934,=20=
min=3D0.000319,=20max=3D0.025784=0A=20=20=20=20FetchStatus:=201751=20ops=20=
(1751=20OK);=20sum=3D2.529428,=20min=3D0.000003,=20max=3D0.181252=0A=20=20=
=20=20=20=20StoreData:=209856=20ops=20(9851=20OK);=20sum=3D25438.793801,=20=
min=3D0.000059,=20max=3D92.664678=0A=20=20=20=20=20=20=20StoreACL:=201=20=
ops=20(1=20OK);=20sum=3D0.000397,=20min=3D0.000397,=20max=3D0.000397=0A=20=
=20=20=20StoreStatus:=209645=20ops=20(9645=20OK);=20sum=3D0.331826,=20=
min=3D0.000010,=20max=3D0.020294=0A=20=20=20=20=20RemoveFile:=208=20ops=20=
(8=20OK);=20sum=3D0.541631,=20min=3D0.000392,=20max=3D0.499645=0A=20=20=20=
=20=20CreateFile:=208830=20ops=20(8830=20OK);=20sum=3D29.082916,=20=
min=3D0.000426,=20max=3D0.193746=0A=20=20=20=20=20=20=20=20=20Rename:=20=
1=20ops=20(1=20OK);=20sum=3D0.000537,=20min=3D0.000537,=20max=3D0.000537=0A=
=20=20=20=20=20=20=20=20Symlink:=2039=20ops=20(39=20OK);=20sum=3D0.117115,=
=20min=3D0.000382,=20max=3D0.020918=0A=20=20=20=20=20=20=20=20=20=20=20=
Link:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20=
max=3D0.000000=0A=20=20=20=20=20=20=20=20MakeDir:=20893=20ops=20(893=20=
OK);=20sum=3D2.326453,=20min=3D0.000369,=20max=3D0.068759=0A=20=20=20=20=20=
=20RemoveDir:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20=
min=3D999999.000000,=20max=3D0.000000=0A=20=20=20=20=20=20=20=20SetLock:=20=
0=20ops=20(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20=
max=3D0.000000=0A=20=20=20=20=20ExtendLock:=200=20ops=20(0=20OK);=20=
sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=20=20=20=20=
ReleaseLock:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20=
min=3D999999.000000,=20max=3D0.000000=0A=20=20GetStatistics:=200=20ops=20=
(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=
GiveUpCallbacks:=20556=20ops=20(556=20OK);=20sum=3D0.006557,=20=
min=3D0.000001,=20max=3D0.000053=0A=20=20GetVolumeInfo:=200=20ops=20(0=20=
OK);=20sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=
GetVolumeStatus:=2014=20ops=20(14=20OK);=20sum=3D0.069256,=20=
min=3D0.000006,=20max=3D0.069048=0ASetVolumeStatus:=200=20ops=20(0=20=
OK);=20sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=20=20=
GetRootVolume:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20=
min=3D999999.000000,=20max=3D0.000000=0A=20=20=20=20=20CheckToken:=200=20=
ops=20(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=
=20=20=20=20=20=20=20=20GetTime:=20141=20ops=20(141=20OK);=20=
sum=3D0.860438,=20min=3D0.000001,=20max=3D0.110030=0A=20NGetVolumeInfo:=20=
0=20ops=20(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20=
max=3D0.000000=0A=20=20=20=20=20BulkStatus:=203=20ops=20(3=20OK);=20=
sum=3D0.056428,=20min=3D0.001006,=20max=3D0.036645=0A=20=20=
XStatsVersion:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20=
min=3D999999.000000,=20max=3D0.000000=0A=20=20=20=20=20=20GetXStats:=202=20=
ops=20(1=20OK);=20sum=3D0.000014,=20min=3D0.000014,=20max=3D0.000014=0A=
FetchData:=2054841=20xfers=20(54841=20OK),=20time=20sum=3D22757.535102,=20=
min=3D0.000003,=20max=3D4.409978=0A=09[bytes:=20sum=3D35261582,=20min=3D0,=
=20max=3D1048576]=0A=09[buckets:=200:=20709,=201:=200,=202:=201113,=203:=20=
8,=204:=2041,=205:=2019730=206:=2037,=207:=2033203,=208:=200]=0A=
StoreData:=209851=20xfers=20(9851=20OK),=20time=20sum=3D25437.758515,=20=
min=3D0.000046,=20max=3D92.664642=0A=09[bytes:=20sum=3D17650917,=20=
min=3D1,=20max=3D67108864]=0A=09[buckets:=200:=201373,=201:=201569,=202:=20=
3099,=203:=20651,=204:=20593,=205:=20700=206:=20628,=207:=20213,=208:=20=
1025]=0A=0AMon=20Jun=2020=2015:13:57=202005=20XXX=20FS=203=20600=20893=20=
86543=2036=2011542=20600=208869=20147419=20686=2027504=2010000=2086604=20=
0=20120=2073322=2045=2039347362=200=200=200=200=200=200=200=200=20=
26286759=208716=207892854=2018223190=200=207892854=2018223189=20431=201=20=
26337378=205119347=200=205119347=200=200=200=2052674756=201466=200=20=
78718256=20258973=20480000=200=200=2014=20650000=2011811342=2041=2020=20=
23=2037=208=2018=201=2018=207=200=2011=2077=202=201502=200=200=200=200=20=
0=200=200=200=200=200=200=200=200=200=200=200=200=200=200=200=200=200=20=
0=200=200=200=200=200=200=200=200=201119267660=200=2055467=2055462=20=
23165=20481670=2017624=20791657=200=208=204=20409999=203=203=200=2043934=20=
0=20981=200=20319=200=2025784=201751=201751=202=20529428=200=20204307=20=
0=203=200=20181252=209864=209858=2025643=20516273=201019263=20322379=200=20=
59=2092=20664678=201=201=200=20397=200=200=200=20397=200=20397=209645=20=
9645=200=20331826=200=20873=200=2010=200=2020294=208=208=200=20541631=20=
0=20250163=200=20392=200=20499645=208830=208830=2029=2082916=200=20=
425345=200=20426=200=20193746=201=201=200=20537=200=200=200=20537=200=20=
537=2039=2039=200=20117115=200=20944=200=20382=200=2020918=200=200=200=20=
0=200=200=20999999=200=200=200=20893=20893=202=20326453=200=2048299=200=20=
369=200=2068759=200=200=200=200=200=200=20999999=200=200=200=200=200=200=20=
0=200=200=20999999=200=200=200=200=200=200=200=200=200=20999999=200=200=20=
0=200=200=200=200=200=200=20999999=200=200=200=200=200=200=200=200=200=20=
999999=200=200=200=20558=20558=200=206573=200=200=200=201=200=2053=200=20=
0=200=200=200=200=20999999=200=200=200=2014=2014=200=2069256=200=204767=20=
0=206=200=2069048=200=200=200=200=200=200=20999999=200=200=200=200=200=20=
0=200=200=200=20999999=200=200=200=200=200=200=200=200=200=20999999=200=20=
0=200=20148=20148=200=20900492=200=2062271=200=201=200=20110030=200=200=20=
0=200=200=200=20999999=200=200=200=203=203=200=2056428=200=201695=200=20=
1006=200=2036645=200=200=200=200=200=200=20999999=200=200=200=203=202=20=
0=2025=200=200=200=2011=200=2014=2055462=2055462=2023162=20787867=20=
17623=20166031=200=203=204=20409978=2035788046=200=201048576=20709=200=20=
1113=208=2041=2019844=2037=2033710=200=209858=209858=2025642=20480682=20=
1019232=20848976=200=2046=2092=20664642=2017796845=201=2067108864=201373=20=
1569=203099=20651=20593=20700=20628=20214=201031=20=0A=0A=
AFS_XSTATSCOLL_FULL_PERF_INFO=20(coll=202)=20for=20FS=20XXX=0A[Probe=20=
2,=20Mon=20Jun=2020=2015:13:57=202005]=0A=0A=09=20=20=20=20=20=20=20=20=20=
3=20numPerfCalls=0A=0A=09=20=20=20=20=20=20=20600=20vcache_L_Entries=0A=09=
=20=20=20=20=20=20=20893=20vcache_L_Allocs=0A=09=20=20=20=20=2086543=20=
vcache_L_Gets=0A=09=20=20=20=20=20=20=20=2036=20vcache_L_Reads=0A=09=20=20=
=20=20=2011542=20vcache_L_Writes=0A=0A=09=20=20=20=20=20=20=20600=20=
vcache_S_Entries=0A=09=20=20=20=20=20=208869=20vcache_S_Allocs=0A=09=20=20=
=20=20147419=20vcache_S_Gets=0A=09=20=20=20=20=20=20=20686=20=
vcache_S_Reads=0A=09=20=20=20=20=2027504=20vcache_S_Writes=0A=0A=09=20=20=
=20=20=2010000=20vcache_H_Entries=0A=09=20=20=20=20=2086604=20=
vcache_H_Gets=0A=09=20=20=20=20=20=20=20=20=200=20vcache_H_Replacements=0A=
=0A=09=20=20=20=20=20=20=20120=20dir_Buffers=0A=09=20=20=20=20=2073322=20=
dir_Calls=0A=09=20=20=20=20=20=20=20=2045=20dir_IOs=0A=0A=09=20=20=
39347362=20rx_packetRequests=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_noPackets_RcvClass=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_noPackets_SendClass=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_noPackets_SpecialClass=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_socketGreedy=0A=09=20=20=20=20=20=20=20=20=200=20rx_bogusPacketOnRead=0A=
=09=20=20=20=20=20=20=20=20=200=20rx_bogusHost=0A=09=20=20=20=20=20=20=20=
=20=200=20rx_noPacketOnRead=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_noPacketBuffersOnRead=0A=09=20=2026286759=20rx_selects=0A=09=20=20=20=20=
=20=208716=20rx_sendSelects=0A=09=20=20=207892854=20=
rx_packetsRead_RcvClass=0A=09=20=2018223190=20rx_packetsRead_SendClass=0A=
=09=20=20=20=20=20=20=20=20=200=20rx_packetsRead_SpecialClass=0A=09=20=20=
=207892854=20rx_dataPacketsRead=0A=09=20=2018223189=20rx_ackPacketsRead=0A=
=09=20=20=20=20=20=20=20431=20rx_dupPacketsRead=0A=09=20=20=20=20=20=20=20=
=20=201=20rx_spuriousPacketsRead=0A=09=20=2026337378=20=
rx_packetsSent_RcvClass=0A=09=20=20=205119347=20rx_packetsSent_SendClass=0A=
=09=20=20=20=20=20=20=20=20=200=20rx_packetsSent_SpecialClass=0A=09=20=20=
=205119347=20rx_ackPacketsSent=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_pingPacketsSent=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_abortPacketsSent=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_busyPacketsSent=0A=09=20=2052674756=20rx_dataPacketsSent=0A=09=20=20=20=
=20=20=201466=20rx_dataPacketsReSent=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_dataPacketsPushed=0A=09=20=2078718256=20rx_ignoreAckedPacket=0A=09=20=20=
=20=20258973=20rx_totalRtt_Sec=0A=09=20=20=20=20480000=20=
rx_totalRtt_Usec=0A=09=20=20=20=20=20=20=20=20=200=20rx_minRtt_Sec=0A=09=20=
=20=20=20=20=20=20=20=200=20rx_minRtt_Usec=0A=09=20=20=20=20=20=20=20=20=
14=20rx_maxRtt_Sec=0A=09=20=20=20=20650000=20rx_maxRtt_Usec=0A=09=20=20=
11811342=20rx_nRttSamples=0A=09=20=20=20=20=20=20=20=2041=20=
rx_nServerConns=0A=09=20=20=20=20=20=20=20=2020=20rx_nClientConns=0A=09=20=
=20=20=20=20=20=20=2023=20rx_nPeerStructs=0A=09=20=20=20=20=20=20=20=20=
37=20rx_nCallStructs=0A=09=20=20=20=20=20=20=20=20=208=20=
rx_nFreeCallStructs=0A=0A=09=20=20=20=20=20=20=20=2018=20=
host_NumHostEntries=0A=09=20=20=20=20=20=20=20=20=201=20host_HostBlocks=0A=
=09=20=20=20=20=20=20=20=2018=20host_NonDeletedHosts=0A=09=20=20=20=20=20=
=20=20=20=207=20host_HostsInSameNetOrSubnet=0A=09=20=20=20=20=20=20=20=20=
=200=20host_HostsInDiffSubnet=0A=09=20=20=20=20=20=20=20=2011=20=
host_HostsInDiffNetwork=0A=09=20=20=20=20=20=20=20=2077=20=
host_NumClients=0A=09=20=20=20=20=20=20=20=20=202=20host_ClientBlocks=0A=0A=
=091119267660=20epoch=0A=20=20=20=20=20=20FetchData:=2055467=20ops=20=
(55462=20OK);=20sum=3D23165.481670,=20min=3D0.000008,=20max=3D4.409999=0A=
=20=20=20=20=20=20=20FetchACL:=203=20ops=20(3=20OK);=20sum=3D0.043934,=20=
min=3D0.000319,=20max=3D0.025784=0A=20=20=20=20FetchStatus:=201751=20ops=20=
(1751=20OK);=20sum=3D2.529428,=20min=3D0.000003,=20max=3D0.181252=0A=20=20=
=20=20=20=20StoreData:=209864=20ops=20(9858=20OK);=20sum=3D25643.516273,=20=
min=3D0.000059,=20max=3D92.664678=0A=20=20=20=20=20=20=20StoreACL:=201=20=
ops=20(1=20OK);=20sum=3D0.000397,=20min=3D0.000397,=20max=3D0.000397=0A=20=
=20=20=20StoreStatus:=209645=20ops=20(9645=20OK);=20sum=3D0.331826,=20=
min=3D0.000010,=20max=3D0.020294=0A=20=20=20=20=20RemoveFile:=208=20ops=20=
(8=20OK);=20sum=3D0.541631,=20min=3D0.000392,=20max=3D0.499645=0A=20=20=20=
=20=20CreateFile:=208830=20ops=20(8830=20OK);=20sum=3D29.082916,=20=
min=3D0.000426,=20max=3D0.193746=0A=20=20=20=20=20=20=20=20=20Rename:=20=
1=20ops=20(1=20OK);=20sum=3D0.000537,=20min=3D0.000537,=20max=3D0.000537=0A=
=20=20=20=20=20=20=20=20Symlink:=2039=20ops=20(39=20OK);=20sum=3D0.117115,=
=20min=3D0.000382,=20max=3D0.020918=0A=20=20=20=20=20=20=20=20=20=20=20=
Link:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20=
max=3D0.000000=0A=20=20=20=20=20=20=20=20MakeDir:=20893=20ops=20(893=20=
OK);=20sum=3D2.326453,=20min=3D0.000369,=20max=3D0.068759=0A=20=20=20=20=20=
=20RemoveDir:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20=
min=3D999999.000000,=20max=3D0.000000=0A=20=20=20=20=20=20=20=20SetLock:=20=
0=20ops=20(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20=
max=3D0.000000=0A=20=20=20=20=20ExtendLock:=200=20ops=20(0=20OK);=20=
sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=20=20=20=20=
ReleaseLock:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20=
min=3D999999.000000,=20max=3D0.000000=0A=20=20GetStatistics:=200=20ops=20=
(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=
GiveUpCallbacks:=20558=20ops=20(558=20OK);=20sum=3D0.006573,=20=
min=3D0.000001,=20max=3D0.000053=0A=20=20GetVolumeInfo:=200=20ops=20(0=20=
OK);=20sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=
GetVolumeStatus:=2014=20ops=20(14=20OK);=20sum=3D0.069256,=20=
min=3D0.000006,=20max=3D0.069048=0ASetVolumeStatus:=200=20ops=20(0=20=
OK);=20sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=20=20=
GetRootVolume:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20=
min=3D999999.000000,=20max=3D0.000000=0A=20=20=20=20=20CheckToken:=200=20=
ops=20(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=
=20=20=20=20=20=20=20=20GetTime:=20148=20ops=20(148=20OK);=20=
sum=3D0.900492,=20min=3D0.000001,=20max=3D0.110030=0A=20NGetVolumeInfo:=20=
0=20ops=20(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20=
max=3D0.000000=0A=20=20=20=20=20BulkStatus:=203=20ops=20(3=20OK);=20=
sum=3D0.056428,=20min=3D0.001006,=20max=3D0.036645=0A=20=20=
XStatsVersion:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20=
min=3D999999.000000,=20max=3D0.000000=0A=20=20=20=20=20=20GetXStats:=203=20=
ops=20(2=20OK);=20sum=3D0.000025,=20min=3D0.000011,=20max=3D0.000014=0A=
FetchData:=2055462=20xfers=20(55462=20OK),=20time=20sum=3D23162.787867,=20=
min=3D0.000003,=20max=3D4.409978=0A=09[bytes:=20sum=3D35788046,=20min=3D0,=
=20max=3D1048576]=0A=09[buckets:=200:=20709,=201:=200,=202:=201113,=203:=20=
8,=204:=2041,=205:=2019844=206:=2037,=207:=2033710,=208:=200]=0A=
StoreData:=209858=20xfers=20(9858=20OK),=20time=20sum=3D25642.480682,=20=
min=3D0.000046,=20max=3D92.664642=0A=09[bytes:=20sum=3D17796845,=20=
min=3D1,=20max=3D67108864]=0A=09[buckets:=200:=201373,=201:=201569,=202:=20=
3099,=203:=20651,=204:=20593,=205:=20700=206:=20628,=207:=20214,=208:=20=
1031]=0A=0AMon=20Jun=2020=2015:14:57=202005=20XXX=20FS=204=20600=20893=20=
87220=2036=2011544=20600=208871=20148723=20688=2027535=2010000=2087281=20=
0=20120=2073337=2045=2039882515=200=200=200=200=200=200=200=200=20=
26635608=208789=207985897=2018478391=200=207985897=2018478390=20438=201=20=
26722224=205176645=200=205176645=200=200=200=2053444448=201489=200=20=
79548070=20263789=20610000=200=200=2014=20650000=2011981653=2043=2020=20=
23=2037=2011=2018=201=2018=207=200=2011=2079=202=201502=200=200=200=200=20=
0=200=200=200=200=200=200=200=200=200=200=200=200=200=200=200=200=200=20=
0=200=200=200=200=200=200=200=200=201119267660=200=2056105=2056099=20=
23572=20234239=2017959=20907539=200=208=204=20409999=203=203=200=2043934=20=
0=20981=200=20319=200=2025784=201760=201760=202=20699738=200=20218072=20=
0=203=200=20181252=209892=209887=2026042=20284335=201041656=20229751=200=20=
59=2092=20664678=201=201=200=20397=200=200=200=20397=200=20397=209645=20=
9645=200=20331826=200=20873=200=2010=200=2020294=208=208=200=20541631=20=
0=20250163=200=20392=200=20499645=208832=208832=2029=20165590=200=20=
429181=200=20426=200=20193746=201=201=200=20537=200=200=200=20537=200=20=
537=2039=2039=200=20117115=200=20944=200=20382=200=2020918=200=200=200=20=
0=200=200=20999999=200=200=200=20893=20893=202=20326453=200=2048299=200=20=
369=200=2068759=200=200=200=200=200=200=20999999=200=200=200=200=200=200=20=
0=200=200=20999999=200=200=200=200=200=200=200=200=200=20999999=200=200=20=
0=200=200=200=200=200=200=20999999=200=200=200=200=200=200=200=200=200=20=
999999=200=200=200=20558=20558=200=206573=200=200=200=201=200=2053=200=20=
0=200=200=200=200=20999999=200=200=200=2014=2014=200=2069256=200=204767=20=
0=206=200=2069048=200=200=200=200=200=200=20999999=200=200=200=200=200=20=
0=200=200=200=20999999=200=200=200=200=200=200=200=200=200=20999999=200=20=
0=200=20148=20148=200=20900492=200=2062271=200=201=200=20110030=200=200=20=
0=200=200=200=20999999=200=200=200=203=203=200=2056428=200=201695=200=20=
1006=200=2036645=200=200=200=200=200=200=20999999=200=200=200=204=203=20=
0=2039=200=200=200=2011=200=2014=2056099=2056099=2023569=20524060=20=
17958=20261374=200=203=204=20409978=2036309899=200=201048576=20709=200=20=
1123=208=2045=2019964=2038=2034212=200=209887=209887=2026041=20247507=20=
1041625=20722312=200=2046=2092=20664642=2018033863=201=2067108864=201375=20=
1569=203099=20651=20593=20700=20633=20226=201041=20=0A=0A=
AFS_XSTATSCOLL_FULL_PERF_INFO=20(coll=202)=20for=20FS=20XXX=0A[Probe=20=
3,=20Mon=20Jun=2020=2015:14:57=202005]=0A=0A=09=20=20=20=20=20=20=20=20=20=
4=20numPerfCalls=0A=0A=09=20=20=20=20=20=20=20600=20vcache_L_Entries=0A=09=
=20=20=20=20=20=20=20893=20vcache_L_Allocs=0A=09=20=20=20=20=2087220=20=
vcache_L_Gets=0A=09=20=20=20=20=20=20=20=2036=20vcache_L_Reads=0A=09=20=20=
=20=20=2011544=20vcache_L_Writes=0A=0A=09=20=20=20=20=20=20=20600=20=
vcache_S_Entries=0A=09=20=20=20=20=20=208871=20vcache_S_Allocs=0A=09=20=20=
=20=20148723=20vcache_S_Gets=0A=09=20=20=20=20=20=20=20688=20=
vcache_S_Reads=0A=09=20=20=20=20=2027535=20vcache_S_Writes=0A=0A=09=20=20=
=20=20=2010000=20vcache_H_Entries=0A=09=20=20=20=20=2087281=20=
vcache_H_Gets=0A=09=20=20=20=20=20=20=20=20=200=20vcache_H_Replacements=0A=
=0A=09=20=20=20=20=20=20=20120=20dir_Buffers=0A=09=20=20=20=20=2073337=20=
dir_Calls=0A=09=20=20=20=20=20=20=20=2045=20dir_IOs=0A=0A=09=20=20=
39882515=20rx_packetRequests=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_noPackets_RcvClass=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_noPackets_SendClass=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_noPackets_SpecialClass=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_socketGreedy=0A=09=20=20=20=20=20=20=20=20=200=20rx_bogusPacketOnRead=0A=
=09=20=20=20=20=20=20=20=20=200=20rx_bogusHost=0A=09=20=20=20=20=20=20=20=
=20=200=20rx_noPacketOnRead=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_noPacketBuffersOnRead=0A=09=20=2026635608=20rx_selects=0A=09=20=20=20=20=
=20=208789=20rx_sendSelects=0A=09=20=20=207985897=20=
rx_packetsRead_RcvClass=0A=09=20=2018478391=20rx_packetsRead_SendClass=0A=
=09=20=20=20=20=20=20=20=20=200=20rx_packetsRead_SpecialClass=0A=09=20=20=
=207985897=20rx_dataPacketsRead=0A=09=20=2018478390=20rx_ackPacketsRead=0A=
=09=20=20=20=20=20=20=20438=20rx_dupPacketsRead=0A=09=20=20=20=20=20=20=20=
=20=201=20rx_spuriousPacketsRead=0A=09=20=2026722224=20=
rx_packetsSent_RcvClass=0A=09=20=20=205176645=20rx_packetsSent_SendClass=0A=
=09=20=20=20=20=20=20=20=20=200=20rx_packetsSent_SpecialClass=0A=09=20=20=
=205176645=20rx_ackPacketsSent=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_pingPacketsSent=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_abortPacketsSent=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_busyPacketsSent=0A=09=20=2053444448=20rx_dataPacketsSent=0A=09=20=20=20=
=20=20=201489=20rx_dataPacketsReSent=0A=09=20=20=20=20=20=20=20=20=200=20=
rx_dataPacketsPushed=0A=09=20=2079548070=20rx_ignoreAckedPacket=0A=09=20=20=
=20=20263789=20rx_totalRtt_Sec=0A=09=20=20=20=20610000=20=
rx_totalRtt_Usec=0A=09=20=20=20=20=20=20=20=20=200=20rx_minRtt_Sec=0A=09=20=
=20=20=20=20=20=20=20=200=20rx_minRtt_Usec=0A=09=20=20=20=20=20=20=20=20=
14=20rx_maxRtt_Sec=0A=09=20=20=20=20650000=20rx_maxRtt_Usec=0A=09=20=20=
11981653=20rx_nRttSamples=0A=09=20=20=20=20=20=20=20=2043=20=
rx_nServerConns=0A=09=20=20=20=20=20=20=20=2020=20rx_nClientConns=0A=09=20=
=20=20=20=20=20=20=2023=20rx_nPeerStructs=0A=09=20=20=20=20=20=20=20=20=
37=20rx_nCallStructs=0A=09=20=20=20=20=20=20=20=2011=20=
rx_nFreeCallStructs=0A=0A=09=20=20=20=20=20=20=20=2018=20=
host_NumHostEntries=0A=09=20=20=20=20=20=20=20=20=201=20host_HostBlocks=0A=
=09=20=20=20=20=20=20=20=2018=20host_NonDeletedHosts=0A=09=20=20=20=20=20=
=20=20=20=207=20host_HostsInSameNetOrSubnet=0A=09=20=20=20=20=20=20=20=20=
=200=20host_HostsInDiffSubnet=0A=09=20=20=20=20=20=20=20=2011=20=
host_HostsInDiffNetwork=0A=09=20=20=20=20=20=20=20=2079=20=
host_NumClients=0A=09=20=20=20=20=20=20=20=20=202=20host_ClientBlocks=0A=0A=
=091119267660=20epoch=0A=20=20=20=20=20=20FetchData:=2056105=20ops=20=
(56099=20OK);=20sum=3D23572.234239,=20min=3D0.000008,=20max=3D4.409999=0A=
=20=20=20=20=20=20=20FetchACL:=203=20ops=20(3=20OK);=20sum=3D0.043934,=20=
min=3D0.000319,=20max=3D0.025784=0A=20=20=20=20FetchStatus:=201760=20ops=20=
(1760=20OK);=20sum=3D2.699738,=20min=3D0.000003,=20max=3D0.181252=0A=20=20=
=20=20=20=20StoreData:=209892=20ops=20(9887=20OK);=20sum=3D26042.284335,=20=
min=3D0.000059,=20max=3D92.664678=0A=20=20=20=20=20=20=20StoreACL:=201=20=
ops=20(1=20OK);=20sum=3D0.000397,=20min=3D0.000397,=20max=3D0.000397=0A=20=
=20=20=20StoreStatus:=209645=20ops=20(9645=20OK);=20sum=3D0.331826,=20=
min=3D0.000010,=20max=3D0.020294=0A=20=20=20=20=20RemoveFile:=208=20ops=20=
(8=20OK);=20sum=3D0.541631,=20min=3D0.000392,=20max=3D0.499645=0A=20=20=20=
=20=20CreateFile:=208832=20ops=20(8832=20OK);=20sum=3D29.165590,=20=
min=3D0.000426,=20max=3D0.193746=0A=20=20=20=20=20=20=20=20=20Rename:=20=
1=20ops=20(1=20OK);=20sum=3D0.000537,=20min=3D0.000537,=20max=3D0.000537=0A=
=20=20=20=20=20=20=20=20Symlink:=2039=20ops=20(39=20OK);=20sum=3D0.117115,=
=20min=3D0.000382,=20max=3D0.020918=0A=20=20=20=20=20=20=20=20=20=20=20=
Link:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20=
max=3D0.000000=0A=20=20=20=20=20=20=20=20MakeDir:=20893=20ops=20(893=20=
OK);=20sum=3D2.326453,=20min=3D0.000369,=20max=3D0.068759=0A=20=20=20=20=20=
=20RemoveDir:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20=
min=3D999999.000000,=20max=3D0.000000=0A=20=20=20=20=20=20=20=20SetLock:=20=
0=20ops=20(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20=
max=3D0.000000=0A=20=20=20=20=20ExtendLock:=200=20ops=20(0=20OK);=20=
sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=20=20=20=20=
ReleaseLock:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20=
min=3D999999.000000,=20max=3D0.000000=0A=20=20GetStatistics:=200=20ops=20=
(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=
GiveUpCallbacks:=20558=20ops=20(558=20OK);=20sum=3D0.006573,=20=
min=3D0.000001,=20max=3D0.000053=0A=20=20GetVolumeInfo:=200=20ops=20(0=20=
OK);=20sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=
GetVolumeStatus:=2014=20ops=20(14=20OK);=20sum=3D0.069256,=20=
min=3D0.000006,=20max=3D0.069048=0ASetVolumeStatus:=200=20ops=20(0=20=
OK);=20sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=20=20=
GetRootVolume:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20=
min=3D999999.000000,=20max=3D0.000000=0A=20=20=20=20=20CheckToken:=200=20=
ops=20(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20max=3D0.000000=0A=
=20=20=20=20=20=20=20=20GetTime:=20148=20ops=20(148=20OK);=20=
sum=3D0.900492,=20min=3D0.000001,=20max=3D0.110030=0A=20NGetVolumeInfo:=20=
0=20ops=20(0=20OK);=20sum=3D0.000000,=20min=3D999999.000000,=20=
max=3D0.000000=0A=20=20=20=20=20BulkStatus:=203=20ops=20(3=20OK);=20=
sum=3D0.056428,=20min=3D0.001006,=20max=3D0.036645=0A=20=20=
XStatsVersion:=200=20ops=20(0=20OK);=20sum=3D0.000000,=20=
min=3D999999.000000,=20max=3D0.000000=0A=20=20=20=20=20=20GetXStats:=204=20=
ops=20(3=20OK);=20sum=3D0.000039,=20min=3D0.000011,=20max=3D0.000014=0A=
FetchData:=2056099=20xfers=20(56099=20OK),=20time=20sum=3D23569.524060,=20=
min=3D0.000003,=20max=3D4.409978=0A=09[bytes:=20sum=3D36309899,=20min=3D0,=
=20max=3D1048576]=0A=09[buckets:=200:=20709,=201:=200,=202:=201123,=203:=20=
8,=204:=2045,=205:=2019964=206:=2038,=207:=2034212,=208:=200]=0A=
StoreData:=209887=20xfers=20(9887=20OK),=20time=20sum=3D26041.247507,=20=
min=3D0.000046,=20max=3D92.664642=0A=09[bytes:=20sum=3D18033863,=20=
min=3D1,=20max=3D67108864]=0A=09[buckets:=200:=201375,=201:=201569,=202:=20=
3099,=203:=20651,=204:=20593,=205:=20700=206:=20633,=207:=20226,=208:=20=
1041]=0A=
--Apple-Mail-12--767038201
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=US-ASCII;
delsp=yes;
format=flowed
--
TU Muenchen, Physik-Department E18, James-Franck-Str. 85747 Garching
Telefon 089/289-12592; Telefax 089/289-12570
--
A mouse is a device used to point at
the xterm you want to type in.
Kim Alm on a.s.r.
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GS/CS/M/MU d-(++) s:+ a-> C+++ UL++++ P-(+) L+++ E(+) W+ !N K- w--- M
+ !V Y+
PGP++ t+(++) 5 R+ tv-- b+ DI++ e+++>++++ h---- y+++
------END GEEK CODE BLOCK------
--Apple-Mail-12--767038201--
--Apple-Mail-13--767037878
content-type: application/pgp-signature; x-mac-type=70674453;
name=PGP.sig
content-description: This is a digitally signed message part
content-disposition: inline; filename=PGP.sig
content-transfer-encoding: 7bit
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)
iD8DBQFCtsRnI4MWO8QIRP0RAqNoAKCqOnfmks1YVbyV8fCPfq8q80wRAQCeKSvP
laFX/iBsvxnBIV004I86zMo=
=0XlP
-----END PGP SIGNATURE-----
--Apple-Mail-13--767037878--