[OpenAFS] Advice regarding OpenAFS performance?

Collin Gros Collin.Gros@ricoh-usa.com
Mon, 14 Aug 2023 22:29:57 +0000


--_004_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_
Content-Type: multipart/alternative;
	boundary="_000_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_"

--_000_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Dear OpenAFS community,


We are administrators for an OpenAFS environment of (what will be) about 40=
0 users and are running into some performance issues, for which we hope you=
 might have some advice...

1. Do you have any sources we can look at that might help us in adjusting c=
onfiguration to improve performance? We read the man page for `dafileserver=
` and messed around a lot with our arguments to `dafileserver` (increasing =
them past the values set for -L, or Large)... though we haven't noticed muc=
h of an improvement in performance through our testing. See below for the c=
onfiguration we currently have set for `dafileserver` on all of our OpenAFS=
 file servers.

2. Do you know what kind of read/write speed we should expect for an enviro=
ment/configuration of this size? It would be helpful for us to know what we=
 should be expecting in our environment as far as performance is concerned.

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Our performance test
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Here are results from our testing with a binary file (7103053824 bytes in s=
ize, or 6.7GB), copying it from one client to AFS:
  client1: openSUSE 15.1
  server: AFS file server that hosts the AFS volumes used for our testing

  `scp`: client1 (local) -> server (local): 102.2MB/s (66s)
  `cp`: client1 (local) -> client1 (AFS file space): 19.2MB/s (352s)
  `cp`: client1 (AFS file space) -> client1 (AFS file space): 19.46MB/s (34=
8s)

Here are results from our testing with the same binary file (7103053824 byt=
es in size, or 6.7GB), copying it in parallel from two clients to the same =
AFS volume:
  client1 (local) -> server (AFS file space): 10.22MB/s (663s)
  client2 (local) -> server (AFS file space): 9.69MB/s (699s)

  client1 (AFS file space) -> client1 (AFS file space): 5.38MB/s (1258s)
  client2 (AFS file space) -> client2 (AFS file space): 7MB/s (965s)

  client1 (AFS file space) -> client1 (local): 13.15MB/s (515s)
  client2 (AFS file space) -> client2 (local): 15.57MB/s (435s)

  client1 total time taken: 2436s
  client2 total time taken: 2099s



Here is a snapshot of what `top` looks like from the AFS file server while =
the copy is taking place:

  top - 16:14:14 up 5 days,  7:29,  2 users,  load average: 1.06, 0.37, 0.2=
6
  Tasks: 297 total,   2 running, 294 sleeping,   1 stopped,   0 zombie
  %Cpu0  : 17.3 us,  6.5 sy,  0.0 ni, 69.4 id,  1.7 wa,  1.0 hi,  4.1 si,  =
0.0 st
  %Cpu1  : 16.2 us,  4.1 sy,  0.0 ni, 65.5 id, 13.2 wa,  0.7 hi,  0.3 si,  =
0.0 st
  %Cpu2  :  5.0 us,  6.7 sy,  0.3 ni, 12.4 id, 63.2 wa,  1.0 hi, 11.4 si,  =
0.0 st
  %Cpu3  :  7.5 us,  5.1 sy,  9.2 ni, 44.2 id, 31.5 wa,  1.4 hi,  1.0 si,  =
0.0 st
  %Cpu4  : 13.3 us,  6.5 sy,  2.0 ni, 67.6 id,  9.9 wa,  0.7 hi,  0.0 si,  =
0.0 st
  %Cpu5  : 37.4 us, 14.6 sy,  0.0 ni, 41.1 id,  6.0 wa,  0.7 hi,  0.3 si,  =
0.0 st
  MiB Mem :  24080.5 total,  14283.7 free,    526.5 used,   9270.3 buff/cac=
he
  MiB Swap:   4060.0 total,   4060.0 free,      0.0 used.  23105.9 avail Me=
m

      PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ CO=
MMAND
    22409 root      15  -5 4282356  65240   2808 S 118.3   0.3  75:55.61 da=
fileserver

Here is the output of `fs getcacheparms` while both clients were copying th=
e file to AFS:

  client1: AFS using 781060 of the cache's available 891289 1K byte blocks.
  client2: AFS using 0 of the cache's available 891289 1K byte blocks.


***************************
Our environment
***************************
We have our environment configuration documented below, and are hoping you =
might give us some pointers as to what might be a performance bottleneck.

  Our testing environment:
    - OpenAFS Servers
      - OpenAFS 1.8.9
      - DB servers (total of 3)
        - 1 master
          - Rocky Linux 8.8
          - 2 CPU
          - 4GB RAM
        - 2 replicas, with each having:
          - Rocky Linux 8.8
          - 2 CPU
          - 4GB RAM
      - FS servers (total of 3)
        - 3 fileservers, with each having:
          - Rocky Linux 8.8
          - 6 CPU
          - 24GB RAM
          - /usr/afs/local/BosConfig:
              restrictmode 0
              restarttime 16 0 0 0 0
              checkbintime 3 0 5 0 0
              bnode dafs dafs 1
              parm /usr/afs/bin/dafileserver -L -cb 640000 -abortthreshold =
0 -vc 1000
              parm /usr/afs/bin/davolserver -p 64 -log
              parm /usr/afs/bin/salvageserver
              parm /usr/afs/bin/dasalvager -parallel all32
              end
              bnode simple upclientetc 1
              parm /usr/afs/bin/upclient db1 /usr/afs/etc
              end
              bnode simple upclientbin 1
              parm /usr/afs/bin/upclient db1 /usr/afs/bin
              end
    - OpenAFS Clients
      - client1
        - openSUSE 15.1
        - OpenAFS 1.8.7
        - 6 CPUs
        - 16GB RAM
        - `fs getcacheparms`
            AFS using 12 of the cache's available 891289 1K byte blocks.
        - /etc/sysconfig/openafs-client:
            AFSD_ARGS=3D"-fakestat -stat 6000 -dcache 6000 -daemons 6 -volu=
mes 256 -files 50000 -chunksize 17"
      - client2
        - openSUSE 13.2
        - OpenAFS 1.8.7
        - 2 CPUs
        - 2GB RAM
        - `fs getcacheparms`
            AFS using 0 of the cache's available 891289 1K byte blocks.
        - /etc/sysconfig/afs
            OPTIONS=3D$XXLARGE
              (and XXLARGE=3D"-fakestat -stat 4000 -dcache 4000 -daemons 6 =
-volumes 256 -afsdb")



Thanks for the help!!


Regards,

Collin

Collin Gros
Staff Software Engineer
RICOH Graphic Communications - DSBC

Ricoh USA, Inc
Phone: +1 720-663-3225
Email: collin.gros@ricoh-usa.com
[cid:image001.png@01D9CEC4.5B423DC0]


--_000_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:Verdana;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
	{font-family:"Lucida Console";
	panose-1:2 11 6 9 4 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;
	mso-ligatures:standardcontextual;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri",sans-serif;}
@page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"#0563C1" vlink=3D"#954F72" style=3D"word-wrap:=
break-word">
<div class=3D"WordSection1">
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">Dear OpenAFS community,<o=
:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">We are administrators for=
 an OpenAFS environment of (what will be) about 400 users and are running i=
nto some performance issues, for which we hope you
 might have some advice...<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">1. Do you have any source=
s we can look at that might help us in adjusting configuration to improve p=
erformance? We read the man page for `dafileserver`
 and messed around a lot with our arguments to `dafileserver` (increasing t=
hem past the values set for -L, or Large)... though we haven't noticed much=
 of an improvement in performance through our testing. See below for the co=
nfiguration we currently have set
 for `dafileserver` on all of our OpenAFS file servers.<o:p></o:p></span></=
p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">2. Do you know what kind =
of read/write speed we should expect for an enviroment/configuration of thi=
s size? It would be helpful for us to know what
 we should be expecting in our environment as far as performance is concern=
ed.<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">Our performance test<o:p>=
</o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">Here are results from our=
 testing with a binary file (7103053824 bytes in size, or 6.7GB), copying i=
t from one client to AFS:<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; client1: openSUSE =
15.1<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; server: AFS file s=
erver that hosts the AFS volumes used for our testing<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; `scp`: client1 (lo=
cal) -&gt; server (local): 102.2MB/s (66s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; `cp`: client1 (loc=
al) -&gt; client1 (AFS file space): 19.2MB/s (352s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; `cp`: client1 (AFS=
 file space) -&gt; client1 (AFS file space): 19.46MB/s (348s)<o:p></o:p></s=
pan></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">Here are results from our=
 testing with the same binary file (7103053824 bytes in size, or 6.7GB), co=
pying it in parallel from two clients to the same
 AFS volume:<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; client1 (local) -&=
gt; server (AFS file space): 10.22MB/s (663s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; client2 (local) -&=
gt; server (AFS file space): 9.69MB/s (699s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; client1 (AFS file =
space) -&gt; client1 (AFS file space): 5.38MB/s (1258s)<o:p></o:p></span></=
p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; client2 (AFS file =
space) -&gt; client2 (AFS file space): 7MB/s (965s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; client1 (AFS file =
space) -&gt; client1 (local): 13.15MB/s (515s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; client2 (AFS file =
space) -&gt; client2 (local): 15.57MB/s (435s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; client1 total time=
 taken: 2436s<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; client2 total time=
 taken: 2099s<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">Here is a snapshot of wha=
t `top` looks like from the AFS file server while the copy is taking place:=
<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; top - 16:14:14 up =
5 days,&nbsp; 7:29,&nbsp; 2 users,&nbsp; load average: 1.06, 0.37, 0.26<o:p=
></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; Tasks: 297 total,&=
nbsp;&nbsp; 2 running, 294 sleeping,&nbsp;&nbsp; 1 stopped,&nbsp;&nbsp; 0 z=
ombie<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; %Cpu0&nbsp; : 17.3=
 us,&nbsp; 6.5 sy,&nbsp; 0.0 ni, 69.4 id,&nbsp; 1.7 wa,&nbsp; 1.0 hi,&nbsp;=
 4.1 si,&nbsp; 0.0 st<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; %Cpu1&nbsp; : 16.2=
 us,&nbsp; 4.1 sy,&nbsp; 0.0 ni, 65.5 id, 13.2 wa,&nbsp; 0.7 hi,&nbsp; 0.3 =
si,&nbsp; 0.0 st<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; %Cpu2&nbsp; :&nbsp=
; 5.0 us,&nbsp; 6.7 sy,&nbsp; 0.3 ni, 12.4 id, 63.2 wa,&nbsp; 1.0 hi, 11.4 =
si,&nbsp; 0.0 st<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; %Cpu3&nbsp; :&nbsp=
; 7.5 us,&nbsp; 5.1 sy,&nbsp; 9.2 ni, 44.2 id, 31.5 wa,&nbsp; 1.4 hi,&nbsp;=
 1.0 si,&nbsp; 0.0 st<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; %Cpu4&nbsp; : 13.3=
 us,&nbsp; 6.5 sy,&nbsp; 2.0 ni, 67.6 id,&nbsp; 9.9 wa,&nbsp; 0.7 hi,&nbsp;=
 0.0 si,&nbsp; 0.0 st<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; %Cpu5&nbsp; : 37.4=
 us, 14.6 sy,&nbsp; 0.0 ni, 41.1 id,&nbsp; 6.0 wa,&nbsp; 0.7 hi,&nbsp; 0.3 =
si,&nbsp; 0.0 st<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; MiB Mem :&nbsp; 24=
080.5 total,&nbsp; 14283.7 free,&nbsp;&nbsp;&nbsp; 526.5 used,&nbsp;&nbsp; =
9270.3 buff/cache<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; MiB Swap:&nbsp;&nb=
sp; 4060.0 total,&nbsp;&nbsp; 4060.0 free,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0.=
0 used.&nbsp; 23105.9 avail Mem<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; PID USER&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PR&nbsp; NI&nbsp;&nbsp;&nbsp; =
VIRT&nbsp;&nbsp;&nbsp; RES&nbsp;&nbsp;&nbsp; SHR S&nbsp; %CPU&nbsp; %MEM&nb=
sp;&nbsp;&nbsp;&nbsp; TIME+ COMMAND<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp; 22409 =
root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 15&nbsp; -5 4282356&nbsp; 65240&nbsp;&nb=
sp; 2808 S 118.3&nbsp;&nbsp; 0.3&nbsp; 75:55.61 dafileserver<o:p></o:p></sp=
an></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">Here is the output of `fs=
 getcacheparms` while both clients were copying the file to AFS:<o:p></o:p>=
</span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; client1: AFS using=
 781060 of the cache's available 891289 1K byte blocks.<o:p></o:p></span></=
p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; client2: AFS using=
 0 of the cache's available 891289 1K byte blocks.<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">*************************=
**<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">Our environment<o:p></o:p=
></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">*************************=
**<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">We have our environment c=
onfiguration documented below, and are hoping you might give us some pointe=
rs as to what might be a performance bottleneck.<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp; Our testing enviro=
nment:<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp; - Open=
AFS Servers<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; - OpenAFS 1.8.9<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; - DB servers (total of 3)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - 1 master<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Rocky Linux 8.8<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - 2 CPU<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - 4GB RAM<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - 2 replicas, with each having:<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Rocky Linux 8.8<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - 2 CPU<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - 4GB RAM<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; - FS servers (total of 3)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - 3 fileservers, with each having:<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Rocky Linux 8.8<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - 6 CPU<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp; &nbsp;- 24GB RAM<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - /usr/afs/local/BosConfig:<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; restrictmode 0<o:p></=
o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; restarttime 16 0 0 0 =
0<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; checkbintime 3 0 5 0 =
0<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; bnode dafs dafs 1<o:p=
></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parm /usr/afs/bin/daf=
ileserver -L -cb 640000 -abortthreshold 0 -vc 1000<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parm /usr/afs/bin/dav=
olserver -p 64 -log<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parm /usr/afs/bin/sal=
vageserver<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parm /usr/afs/bin/das=
alvager -parallel all32<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; end<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; bnode simple upclient=
etc 1<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parm /usr/afs/bin/upc=
lient db1 /usr/afs/etc<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; end<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; bnode simple upclient=
bin 1<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parm /usr/afs/bin/upc=
lient db1 /usr/afs/bin<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; end<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp; - Open=
AFS Clients<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; - client1<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - openSUSE 15.1<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - OpenAFS 1.8.7<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - 6 CPUs<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - 16GB RAM<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - `fs getcacheparms`<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; AFS using 12 of the cache's avail=
able 891289 1K byte blocks.<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - /etc/sysconfig/openafs-client:<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; AFSD_ARGS=3D&quot;-fakestat -stat=
 6000 -dcache 6000 -daemons 6 -volumes 256 -files 50000 -chunksize 17&quot;=
<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp; - client2<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - openSUSE 13.2<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - OpenAFS 1.8.7<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - 2 CPUs<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - 2GB RAM<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - `fs getcacheparms`<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; AFS using 0 of the cache's availa=
ble 891289 1K byte blocks.<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp; - /etc/sysconfig/afs<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; OPTIONS=3D$XXLARGE<o:p></o:p></sp=
an></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (and XXLARGE=3D&quot;=
-fakestat -stat 4000 -dcache 4000 -daemons 6 -volumes 256 -afsdb&quot;)<o:p=
></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">Thanks for the help!!<o:p=
></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;">Regards,<o:p></o:p></span=
></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:&quot;Lucida Console&quot;"><o:p>&nbsp;</o:p></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:&quot;Lu=
cida Console&quot;">Collin</span><o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal"><b><span style=3D"font-size:7.5pt;font-family:&quot;=
Verdana&quot;,sans-serif;mso-ligatures:none">Collin Gros</span></b><span st=
yle=3D"mso-ligatures:none"><o:p></o:p></span></p>
<p class=3D"MsoNormal"><b><span style=3D"font-size:7.5pt;font-family:&quot;=
Verdana&quot;,sans-serif;color:gray;mso-ligatures:none">Staff Software Engi=
neer<o:p></o:p></span></b></p>
<p class=3D"MsoNormal"><b><span style=3D"font-size:7.5pt;font-family:&quot;=
Verdana&quot;,sans-serif;color:gray;mso-ligatures:none">RICOH Graphic Commu=
nications - DSBC</span></b><span style=3D"mso-ligatures:none"><o:p></o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"mso-ligatures:none"><o:p>&nbsp;</o:p>=
</span></p>
<p class=3D"MsoNormal"><b><span style=3D"font-size:7.5pt;font-family:&quot;=
Verdana&quot;,sans-serif;color:gray;mso-ligatures:none">Ricoh USA, Inc</spa=
n></b><span style=3D"font-size:7.5pt;font-family:&quot;Verdana&quot;,sans-s=
erif;color:gray;mso-ligatures:none"><br>
Phone: +1 720-663-3225<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:7.5pt;font-family:&quot;Ver=
dana&quot;,sans-serif;color:gray;mso-ligatures:none">Email: collin.gros@ric=
oh-usa.com</span><span style=3D"mso-ligatures:none"><o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.0pt;font-family:&quot;Ar=
ial&quot;,sans-serif;color:black"><img width=3D"100" height=3D"44" style=3D=
"width:1.0416in;height:.4583in" id=3D"Picture_x0020_1" src=3D"cid:image001.=
png@01D9CEC4.5B423DC0"></span><span style=3D"mso-ligatures:none"><o:p></o:p=
></span></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
</body>
</html>

--_000_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_--

--_004_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_
Content-Type: image/png; name="image001.png"
Content-Description: image001.png
Content-Disposition: inline; filename="image001.png"; size=2215;
	creation-date="Mon, 14 Aug 2023 22:29:56 GMT";
	modification-date="Mon, 14 Aug 2023 22:29:56 GMT"
Content-ID: <image001.png@01D9CEC4.5B423DC0>
Content-Transfer-Encoding: base64

iVBORw0KGgoAAAANSUhEUgAAAGQAAAAsCAMAAACkN+1nAAADAFBMVEW/wcPl5ueIio3gWmGUlpna
IknDxcfiZ2nkdnTtq6WRk5X10syoqq333tngXGLjbm6HiYu0trh4enzyxL6MjpHywbr66eXlfnuF
h4meoKLmgn6wsrTeTlvsoJpxcnXjcXCqrK7vtK7bKkyZm56Ymp3nhYHdQlWusLL10MrpmJLeUFzL
zc+OkJKipKff4OHoioXnhoKBg4Xzx8HbM099foHfWWHcOVHBwsS8vsCho6XaJUrdRVb329X32NPk
enjU1tfzycPniIOcnqH////rnZfplpHpko3dSVjkeHZ0dnjiaWrcNlDfVV7cPFLaH0n/+vjrnpj9
/f3ojYjvsKrhYmb8/Pz98++5u73FxsjcPlO3ubzbMU74+PmWmJvW19nX2Nr9+Pf98u+2uLrJy8zg
X2TbL056fH/r7Oz39/i3ubuAgYT1z8l7fX/hYWaytLft7e6Ji4799vTso5309fXx8fLi4+T//Pzj
5OV/gYOmqKvY2dv19vbgYGXfV1//+/qlp6nMztC8vcD6+/v//f3Q0tPOz9HxvrfNz9B+gILu7u+j
pajKy83T1Nbg4eLGyMrS09Xz9PTc3d744dzu7/DyxL3cP1SSlJeXmpzo6Ol3eXztrKakpqnqmpT9
9PL98e55e33sopvP0NLeU12bnaD76+jwt7Da3N3hZGf29veTlZiDhYjZ29z0zcfvtq/rm5X66ub4
4NvhY2b99/WDhIfq6+zoj4qPkZTurqfcNE/wuLLn6OnplY/bKEvv8PDs7e3vs63z8/OsrrGnqaz4
49/rn5n77+z0zcjp6erHycvh4uO4uryKjI7vsKnFx8n0zsj77erbLEz0zMX55eHV19j98/H76+f9
9fN0dXjR09T55N/rm5b219HtqKL98O27vb/p6uve3+DzysTspqDia2zvsav11M7wurT5+fryx7/w
8fHmg3/44t2eoaPokIt2d3pvcHPqmZTuraflgHzExsj77OjspZ765+P00Mj55uLql5L77uvgXWPo
jIfplZDur6j///8bOZTjAAAFYklEQVRYhe2VaVRTRxTHQwygAUIoIihIKJSCGKAtIIoKprdqKxWI
moUYEkgEQsq+C0UWQXbZEaQYFHCnat33fde6r3RfraV2X6xtjzPvkRAhydEvPa3H+TDzn3v/M783
705eKPAvNMozyDPI/xxy/68xgazPTU1NLW3vf3WNiE3pSE0N+7TfYHFuDivhwZrhM3pu9QeWe1s+
YMxy/dUslAyYhaV2fDaKkIt7xqaG9WQOgYRoNWOP53FsOdYckmcbrsk2LUGB+rtN6rkPy53w3Eba
+RIhI/2QTnhPJySP4719TuCL2Iw3motjc3C2ewKx+yxbW0uPEIejAI3YFLCF423pg0T0B9gUhtT8
t4jdQhcgPS9UJ+Q2Ib90QJKlDVkSjcS+GRaLkR71RSOAOz7X7B68y7RV+OgHHx/yLqm9kXRIGYAc
98CPvXzAvMgSZ3aTk97Z2H75EcgiA5B3yKrhLdZkDkCI2gRqmd+fjwKz1HXdjtMjSEi0dfumTZu6
rjEMnaTrj6nWw/sXaSB38dijZR6BA+PUMxs8cyUhIROawnHbpx8y1szDoSkAiV3HMrUgLPy2TmiZ
x6kfY+BcrH6IVtMDObYZFzTa9Z9viZgago9m3KhlJrZLVc+m4cd6nYwa5215GzcfQ6/LHdXYOJD8
RWkgf+PxBy3zEhy4O/jlEYW/BCkpKZHXHxgs/NEENE5ufARidhGNM+sHzL/h6xa+mZyk4BXRG5/o
do2ah0RwtzYEOFgsmEI6r/8J0IEDrBV4uvg79bEe83dCXOHLw/CTndOGfEhcuOCXDlpbd4/LS+gC
cMUBv50n3HcTPxnTC48PmU7I3sm4gujy/IhjxLerfe8urWvTgd5RWDBWzs64D+Acx6bXkLy4kYTg
B02IHALxYzDydpLahsFgvLlmKtgMQ2MHGdu813RmsHFIwOxVw3uI11Q/fctMdHedJzM45PcR5now
/NAqoiY/5TFWjVk0BMLn81eo0Vej+PwLURCKhijNpzTyltkr1tN6f9Esiaz//c6djRZXNXNk5/d/
Btqj+FHtgxlP0z/jIYWeTG6a3kUGUrohfSZ6MnHmehcZSOmGeG3Tk9mTpXfRwognhHwkKRLJO0e3
CNhCEMtotBqAquaKUwJYKQQToRuKwulX141X+6v7duTnHqZZsf0B/COaTaDcqnKZixHwrHbEiWog
qYKuo8oUpoDHZFayY2RWrdVf++cU+rYYtTqdz2ZCWzO0vUGxv5eRvl5WqpSQ9rrn+krPp7k5ikQ3
csCkSq6SxKro49nrIFEltP8+DthHpHbCoZBkAZd5BgpVRQpVKaSVUG5IKL5FkGgH5hHguQwUqhp5
dlbLOifSXsHmot78LPDKKMCrESonlmcXgDy+NtkJ1lJF0mxJgwtdF4R3Vg4CO1D4VhkxRyfek8io
XPh5PYZMWghi32KKKt+zs/96NLfh3twFuFtNDrFpsuzCJGUOjC87YCcCoMcVqDw9O0U6IclLQbCe
uza+yo0GsY45BY4fgwsTV3dSPojji2/G1CIjL12MentVEuqPLAPey/JTSl5RGYJIoDK+iE4Do3iB
V0w1SnPTa/VCYoIKYg57Oq7k0pSdVKoGUsmNoLr1ZcX6liB7A13ZtnC/E4YIvVT5Tq2FB5RSkMdw
pb7sztY4qNjq1lcCzBcGQVa3QE4SnNwAPOk3UCzKqIsFcVCWjApGXlCdAVxpOdQuTRSmKYrLsT+3
IHGpOOsT4G6IhdWy1Rl7xFfSoUHKhdibJ9lnQFGa6J8EV+oGQYZeuPSgbXVlVkPjBtvpDfsnqjL0
JHVAGjpHJtvnPiFk5Ug2u0Rf8in6Cj+D/OcgDwGTViHhhXUGpgAAAABJRU5ErkJggg==

--_004_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_--