[OpenAFS] Advice regarding OpenAFS performance?
Collin Gros
Collin.Gros@ricoh-usa.com
Mon, 14 Aug 2023 22:29:57 +0000
--_004_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_
Content-Type: multipart/alternative;
boundary="_000_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_"
--_000_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Dear OpenAFS community,
We are administrators for an OpenAFS environment of (what will be) about 40=
0 users and are running into some performance issues, for which we hope you=
might have some advice...
1. Do you have any sources we can look at that might help us in adjusting c=
onfiguration to improve performance? We read the man page for `dafileserver=
` and messed around a lot with our arguments to `dafileserver` (increasing =
them past the values set for -L, or Large)... though we haven't noticed muc=
h of an improvement in performance through our testing. See below for the c=
onfiguration we currently have set for `dafileserver` on all of our OpenAFS=
file servers.
2. Do you know what kind of read/write speed we should expect for an enviro=
ment/configuration of this size? It would be helpful for us to know what we=
should be expecting in our environment as far as performance is concerned.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Our performance test
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Here are results from our testing with a binary file (7103053824 bytes in s=
ize, or 6.7GB), copying it from one client to AFS:
client1: openSUSE 15.1
server: AFS file server that hosts the AFS volumes used for our testing
`scp`: client1 (local) -> server (local): 102.2MB/s (66s)
`cp`: client1 (local) -> client1 (AFS file space): 19.2MB/s (352s)
`cp`: client1 (AFS file space) -> client1 (AFS file space): 19.46MB/s (34=
8s)
Here are results from our testing with the same binary file (7103053824 byt=
es in size, or 6.7GB), copying it in parallel from two clients to the same =
AFS volume:
client1 (local) -> server (AFS file space): 10.22MB/s (663s)
client2 (local) -> server (AFS file space): 9.69MB/s (699s)
client1 (AFS file space) -> client1 (AFS file space): 5.38MB/s (1258s)
client2 (AFS file space) -> client2 (AFS file space): 7MB/s (965s)
client1 (AFS file space) -> client1 (local): 13.15MB/s (515s)
client2 (AFS file space) -> client2 (local): 15.57MB/s (435s)
client1 total time taken: 2436s
client2 total time taken: 2099s
Here is a snapshot of what `top` looks like from the AFS file server while =
the copy is taking place:
top - 16:14:14 up 5 days, 7:29, 2 users, load average: 1.06, 0.37, 0.2=
6
Tasks: 297 total, 2 running, 294 sleeping, 1 stopped, 0 zombie
%Cpu0 : 17.3 us, 6.5 sy, 0.0 ni, 69.4 id, 1.7 wa, 1.0 hi, 4.1 si, =
0.0 st
%Cpu1 : 16.2 us, 4.1 sy, 0.0 ni, 65.5 id, 13.2 wa, 0.7 hi, 0.3 si, =
0.0 st
%Cpu2 : 5.0 us, 6.7 sy, 0.3 ni, 12.4 id, 63.2 wa, 1.0 hi, 11.4 si, =
0.0 st
%Cpu3 : 7.5 us, 5.1 sy, 9.2 ni, 44.2 id, 31.5 wa, 1.4 hi, 1.0 si, =
0.0 st
%Cpu4 : 13.3 us, 6.5 sy, 2.0 ni, 67.6 id, 9.9 wa, 0.7 hi, 0.0 si, =
0.0 st
%Cpu5 : 37.4 us, 14.6 sy, 0.0 ni, 41.1 id, 6.0 wa, 0.7 hi, 0.3 si, =
0.0 st
MiB Mem : 24080.5 total, 14283.7 free, 526.5 used, 9270.3 buff/cac=
he
MiB Swap: 4060.0 total, 4060.0 free, 0.0 used. 23105.9 avail Me=
m
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ CO=
MMAND
22409 root 15 -5 4282356 65240 2808 S 118.3 0.3 75:55.61 da=
fileserver
Here is the output of `fs getcacheparms` while both clients were copying th=
e file to AFS:
client1: AFS using 781060 of the cache's available 891289 1K byte blocks.
client2: AFS using 0 of the cache's available 891289 1K byte blocks.
***************************
Our environment
***************************
We have our environment configuration documented below, and are hoping you =
might give us some pointers as to what might be a performance bottleneck.
Our testing environment:
- OpenAFS Servers
- OpenAFS 1.8.9
- DB servers (total of 3)
- 1 master
- Rocky Linux 8.8
- 2 CPU
- 4GB RAM
- 2 replicas, with each having:
- Rocky Linux 8.8
- 2 CPU
- 4GB RAM
- FS servers (total of 3)
- 3 fileservers, with each having:
- Rocky Linux 8.8
- 6 CPU
- 24GB RAM
- /usr/afs/local/BosConfig:
restrictmode 0
restarttime 16 0 0 0 0
checkbintime 3 0 5 0 0
bnode dafs dafs 1
parm /usr/afs/bin/dafileserver -L -cb 640000 -abortthreshold =
0 -vc 1000
parm /usr/afs/bin/davolserver -p 64 -log
parm /usr/afs/bin/salvageserver
parm /usr/afs/bin/dasalvager -parallel all32
end
bnode simple upclientetc 1
parm /usr/afs/bin/upclient db1 /usr/afs/etc
end
bnode simple upclientbin 1
parm /usr/afs/bin/upclient db1 /usr/afs/bin
end
- OpenAFS Clients
- client1
- openSUSE 15.1
- OpenAFS 1.8.7
- 6 CPUs
- 16GB RAM
- `fs getcacheparms`
AFS using 12 of the cache's available 891289 1K byte blocks.
- /etc/sysconfig/openafs-client:
AFSD_ARGS=3D"-fakestat -stat 6000 -dcache 6000 -daemons 6 -volu=
mes 256 -files 50000 -chunksize 17"
- client2
- openSUSE 13.2
- OpenAFS 1.8.7
- 2 CPUs
- 2GB RAM
- `fs getcacheparms`
AFS using 0 of the cache's available 891289 1K byte blocks.
- /etc/sysconfig/afs
OPTIONS=3D$XXLARGE
(and XXLARGE=3D"-fakestat -stat 4000 -dcache 4000 -daemons 6 =
-volumes 256 -afsdb")
Thanks for the help!!
Regards,
Collin
Collin Gros
Staff Software Engineer
RICOH Graphic Communications - DSBC
Ricoh USA, Inc
Phone: +1 720-663-3225
Email: collin.gros@ricoh-usa.com
[cid:image001.png@01D9CEC4.5B423DC0]
--_000_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Lucida Console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"#0563C1" vlink=3D"#954F72" style=3D"word-wrap:=
break-word">
<div class=3D"WordSection1">
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">Dear OpenAFS community,<o=
:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">We are administrators for=
an OpenAFS environment of (what will be) about 400 users and are running i=
nto some performance issues, for which we hope you
might have some advice...<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">1. Do you have any source=
s we can look at that might help us in adjusting configuration to improve p=
erformance? We read the man page for `dafileserver`
and messed around a lot with our arguments to `dafileserver` (increasing t=
hem past the values set for -L, or Large)... though we haven't noticed much=
of an improvement in performance through our testing. See below for the co=
nfiguration we currently have set
for `dafileserver` on all of our OpenAFS file servers.<o:p></o:p></span></=
p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">2. Do you know what kind =
of read/write speed we should expect for an enviroment/configuration of thi=
s size? It would be helpful for us to know what
we should be expecting in our environment as far as performance is concern=
ed.<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">Our performance test<o:p>=
</o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">Here are results from our=
testing with a binary file (7103053824 bytes in size, or 6.7GB), copying i=
t from one client to AFS:<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> client1: openSUSE =
15.1<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> server: AFS file s=
erver that hosts the AFS volumes used for our testing<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> `scp`: client1 (lo=
cal) -> server (local): 102.2MB/s (66s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> `cp`: client1 (loc=
al) -> client1 (AFS file space): 19.2MB/s (352s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> `cp`: client1 (AFS=
file space) -> client1 (AFS file space): 19.46MB/s (348s)<o:p></o:p></s=
pan></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">Here are results from our=
testing with the same binary file (7103053824 bytes in size, or 6.7GB), co=
pying it in parallel from two clients to the same
AFS volume:<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> client1 (local) -&=
gt; server (AFS file space): 10.22MB/s (663s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> client2 (local) -&=
gt; server (AFS file space): 9.69MB/s (699s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> client1 (AFS file =
space) -> client1 (AFS file space): 5.38MB/s (1258s)<o:p></o:p></span></=
p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> client2 (AFS file =
space) -> client2 (AFS file space): 7MB/s (965s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> client1 (AFS file =
space) -> client1 (local): 13.15MB/s (515s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> client2 (AFS file =
space) -> client2 (local): 15.57MB/s (435s)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> client1 total time=
taken: 2436s<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> client2 total time=
taken: 2099s<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">Here is a snapshot of wha=
t `top` looks like from the AFS file server while the copy is taking place:=
<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> top - 16:14:14 up =
5 days, 7:29, 2 users, load average: 1.06, 0.37, 0.26<o:p=
></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> Tasks: 297 total,&=
nbsp; 2 running, 294 sleeping, 1 stopped, 0 z=
ombie<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> %Cpu0 : 17.3=
us, 6.5 sy, 0.0 ni, 69.4 id, 1.7 wa, 1.0 hi, =
4.1 si, 0.0 st<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> %Cpu1 : 16.2=
us, 4.1 sy, 0.0 ni, 65.5 id, 13.2 wa, 0.7 hi, 0.3 =
si, 0.0 st<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> %Cpu2 : =
; 5.0 us, 6.7 sy, 0.3 ni, 12.4 id, 63.2 wa, 1.0 hi, 11.4 =
si, 0.0 st<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> %Cpu3 : =
; 7.5 us, 5.1 sy, 9.2 ni, 44.2 id, 31.5 wa, 1.4 hi, =
1.0 si, 0.0 st<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> %Cpu4 : 13.3=
us, 6.5 sy, 2.0 ni, 67.6 id, 9.9 wa, 0.7 hi, =
0.0 si, 0.0 st<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> %Cpu5 : 37.4=
us, 14.6 sy, 0.0 ni, 41.1 id, 6.0 wa, 0.7 hi, 0.3 =
si, 0.0 st<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> MiB Mem : 24=
080.5 total, 14283.7 free, 526.5 used, =
9270.3 buff/cache<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> MiB Swap: &nb=
sp; 4060.0 total, 4060.0 free, 0.=
0 used. 23105.9 avail Mem<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; PID USER PR NI =
VIRT RES SHR S %CPU %MEM&nb=
sp; TIME+ COMMAND<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> 22409 =
root 15 -5 4282356 65240 &nb=
sp; 2808 S 118.3 0.3 75:55.61 dafileserver<o:p></o:p></sp=
an></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">Here is the output of `fs=
getcacheparms` while both clients were copying the file to AFS:<o:p></o:p>=
</span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> client1: AFS using=
781060 of the cache's available 891289 1K byte blocks.<o:p></o:p></span></=
p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> client2: AFS using=
0 of the cache's available 891289 1K byte blocks.<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">*************************=
**<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">Our environment<o:p></o:p=
></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">*************************=
**<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">We have our environment c=
onfiguration documented below, and are hoping you might give us some pointe=
rs as to what might be a performance bottleneck.<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> Our testing enviro=
nment:<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> - Open=
AFS Servers<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - OpenAFS 1.8.9<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - DB servers (total of 3)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - 1 master<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - Rocky Linux 8.8<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - 2 CPU<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - 4GB RAM<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - 2 replicas, with each having:<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - Rocky Linux 8.8<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - 2 CPU<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - 4GB RAM<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - FS servers (total of 3)<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - 3 fileservers, with each having:<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - Rocky Linux 8.8<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - 6 CPU<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - 24GB RAM<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - /usr/afs/local/BosConfig:<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; restrictmode 0<o:p></=
o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; restarttime 16 0 0 0 =
0<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; checkbintime 3 0 5 0 =
0<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; bnode dafs dafs 1<o:p=
></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; parm /usr/afs/bin/daf=
ileserver -L -cb 640000 -abortthreshold 0 -vc 1000<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; parm /usr/afs/bin/dav=
olserver -p 64 -log<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; parm /usr/afs/bin/sal=
vageserver<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; parm /usr/afs/bin/das=
alvager -parallel all32<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; end<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; bnode simple upclient=
etc 1<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; parm /usr/afs/bin/upc=
lient db1 /usr/afs/etc<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; end<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; bnode simple upclient=
bin 1<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; parm /usr/afs/bin/upc=
lient db1 /usr/afs/bin<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; end<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> - Open=
AFS Clients<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - client1<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - openSUSE 15.1<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - OpenAFS 1.8.7<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - 6 CPUs<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - 16GB RAM<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - `fs getcacheparms`<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; AFS using 12 of the cache's avail=
able 891289 1K byte blocks.<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - /etc/sysconfig/openafs-client:<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; AFSD_ARGS=3D"-fakestat -stat=
6000 -dcache 6000 -daemons 6 -volumes 256 -files 50000 -chunksize 17"=
<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - client2<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - openSUSE 13.2<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - OpenAFS 1.8.7<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - 2 CPUs<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - 2GB RAM<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - `fs getcacheparms`<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; AFS using 0 of the cache's availa=
ble 891289 1K byte blocks.<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; - /etc/sysconfig/afs<o:p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; OPTIONS=3D$XXLARGE<o:p></o:p></sp=
an></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""> &=
nbsp; (and XXLARGE=3D"=
-fakestat -stat 4000 -dcache 4000 -daemons 6 -volumes 256 -afsdb")<o:p=
></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">Thanks for the help!!<o:p=
></o:p></span></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console"">Regards,<o:p></o:p></span=
></p>
<p class=3D"MsoNormal" style=3D"text-autospace:none"><span style=3D"font-si=
ze:12.0pt;font-family:"Lucida Console""><o:p> </o:p></span><=
/p>
<p class=3D"MsoNormal"><span style=3D"font-size:12.0pt;font-family:"Lu=
cida Console"">Collin</span><o:p></o:p></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
<p class=3D"MsoNormal"><b><span style=3D"font-size:7.5pt;font-family:"=
Verdana",sans-serif;mso-ligatures:none">Collin Gros</span></b><span st=
yle=3D"mso-ligatures:none"><o:p></o:p></span></p>
<p class=3D"MsoNormal"><b><span style=3D"font-size:7.5pt;font-family:"=
Verdana",sans-serif;color:gray;mso-ligatures:none">Staff Software Engi=
neer<o:p></o:p></span></b></p>
<p class=3D"MsoNormal"><b><span style=3D"font-size:7.5pt;font-family:"=
Verdana",sans-serif;color:gray;mso-ligatures:none">RICOH Graphic Commu=
nications - DSBC</span></b><span style=3D"mso-ligatures:none"><o:p></o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"mso-ligatures:none"><o:p> </o:p>=
</span></p>
<p class=3D"MsoNormal"><b><span style=3D"font-size:7.5pt;font-family:"=
Verdana",sans-serif;color:gray;mso-ligatures:none">Ricoh USA, Inc</spa=
n></b><span style=3D"font-size:7.5pt;font-family:"Verdana",sans-s=
erif;color:gray;mso-ligatures:none"><br>
Phone: +1 720-663-3225<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:7.5pt;font-family:"Ver=
dana",sans-serif;color:gray;mso-ligatures:none">Email: collin.gros@ric=
oh-usa.com</span><span style=3D"mso-ligatures:none"><o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:10.0pt;font-family:"Ar=
ial",sans-serif;color:black"><img width=3D"100" height=3D"44" style=3D=
"width:1.0416in;height:.4583in" id=3D"Picture_x0020_1" src=3D"cid:image001.=
png@01D9CEC4.5B423DC0"></span><span style=3D"mso-ligatures:none"><o:p></o:p=
></span></p>
<p class=3D"MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>
--_000_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_--
--_004_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_
Content-Type: image/png; name="image001.png"
Content-Description: image001.png
Content-Disposition: inline; filename="image001.png"; size=2215;
creation-date="Mon, 14 Aug 2023 22:29:56 GMT";
modification-date="Mon, 14 Aug 2023 22:29:56 GMT"
Content-ID: <image001.png@01D9CEC4.5B423DC0>
Content-Transfer-Encoding: base64
iVBORw0KGgoAAAANSUhEUgAAAGQAAAAsCAMAAACkN+1nAAADAFBMVEW/wcPl5ueIio3gWmGUlpna
IknDxcfiZ2nkdnTtq6WRk5X10syoqq333tngXGLjbm6HiYu0trh4enzyxL6MjpHywbr66eXlfnuF
h4meoKLmgn6wsrTeTlvsoJpxcnXjcXCqrK7vtK7bKkyZm56Ymp3nhYHdQlWusLL10MrpmJLeUFzL
zc+OkJKipKff4OHoioXnhoKBg4Xzx8HbM099foHfWWHcOVHBwsS8vsCho6XaJUrdRVb329X32NPk
enjU1tfzycPniIOcnqH////rnZfplpHpko3dSVjkeHZ0dnjiaWrcNlDfVV7cPFLaH0n/+vjrnpj9
/f3ojYjvsKrhYmb8/Pz98++5u73FxsjcPlO3ubzbMU74+PmWmJvW19nX2Nr9+Pf98u+2uLrJy8zg
X2TbL056fH/r7Oz39/i3ubuAgYT1z8l7fX/hYWaytLft7e6Ji4799vTso5309fXx8fLi4+T//Pzj
5OV/gYOmqKvY2dv19vbgYGXfV1//+/qlp6nMztC8vcD6+/v//f3Q0tPOz9HxvrfNz9B+gILu7u+j
pajKy83T1Nbg4eLGyMrS09Xz9PTc3d744dzu7/DyxL3cP1SSlJeXmpzo6Ol3eXztrKakpqnqmpT9
9PL98e55e33sopvP0NLeU12bnaD76+jwt7Da3N3hZGf29veTlZiDhYjZ29z0zcfvtq/rm5X66ub4
4NvhY2b99/WDhIfq6+zoj4qPkZTurqfcNE/wuLLn6OnplY/bKEvv8PDs7e3vs63z8/OsrrGnqaz4
49/rn5n77+z0zcjp6erHycvh4uO4uryKjI7vsKnFx8n0zsj77erbLEz0zMX55eHV19j98/H76+f9
9fN0dXjR09T55N/rm5b219HtqKL98O27vb/p6uve3+DzysTspqDia2zvsav11M7wurT5+fryx7/w
8fHmg3/44t2eoaPokIt2d3pvcHPqmZTuraflgHzExsj77OjspZ765+P00Mj55uLql5L77uvgXWPo
jIfplZDur6j///8bOZTjAAAFYklEQVRYhe2VaVRTRxTHQwygAUIoIihIKJSCGKAtIIoKprdqKxWI
moUYEkgEQsq+C0UWQXbZEaQYFHCnat33fde6r3RfraV2X6xtjzPvkRAhydEvPa3H+TDzn3v/M783
705eKPAvNMozyDPI/xxy/68xgazPTU1NLW3vf3WNiE3pSE0N+7TfYHFuDivhwZrhM3pu9QeWe1s+
YMxy/dUslAyYhaV2fDaKkIt7xqaG9WQOgYRoNWOP53FsOdYckmcbrsk2LUGB+rtN6rkPy53w3Eba
+RIhI/2QTnhPJySP4719TuCL2Iw3motjc3C2ewKx+yxbW0uPEIejAI3YFLCF423pg0T0B9gUhtT8
t4jdQhcgPS9UJ+Q2Ib90QJKlDVkSjcS+GRaLkR71RSOAOz7X7B68y7RV+OgHHx/yLqm9kXRIGYAc
98CPvXzAvMgSZ3aTk97Z2H75EcgiA5B3yKrhLdZkDkCI2gRqmd+fjwKz1HXdjtMjSEi0dfumTZu6
rjEMnaTrj6nWw/sXaSB38dijZR6BA+PUMxs8cyUhIROawnHbpx8y1szDoSkAiV3HMrUgLPy2TmiZ
x6kfY+BcrH6IVtMDObYZFzTa9Z9viZgago9m3KhlJrZLVc+m4cd6nYwa5215GzcfQ6/LHdXYOJD8
RWkgf+PxBy3zEhy4O/jlEYW/BCkpKZHXHxgs/NEENE5ufARidhGNM+sHzL/h6xa+mZyk4BXRG5/o
do2ah0RwtzYEOFgsmEI6r/8J0IEDrBV4uvg79bEe83dCXOHLw/CTndOGfEhcuOCXDlpbd4/LS+gC
cMUBv50n3HcTPxnTC48PmU7I3sm4gujy/IhjxLerfe8urWvTgd5RWDBWzs64D+Acx6bXkLy4kYTg
B02IHALxYzDydpLahsFgvLlmKtgMQ2MHGdu813RmsHFIwOxVw3uI11Q/fctMdHedJzM45PcR5now
/NAqoiY/5TFWjVk0BMLn81eo0Vej+PwLURCKhijNpzTyltkr1tN6f9Esiaz//c6djRZXNXNk5/d/
Btqj+FHtgxlP0z/jIYWeTG6a3kUGUrohfSZ6MnHmehcZSOmGeG3Tk9mTpXfRwognhHwkKRLJO0e3
CNhCEMtotBqAquaKUwJYKQQToRuKwulX141X+6v7duTnHqZZsf0B/COaTaDcqnKZixHwrHbEiWog
qYKuo8oUpoDHZFayY2RWrdVf++cU+rYYtTqdz2ZCWzO0vUGxv5eRvl5WqpSQ9rrn+krPp7k5ikQ3
csCkSq6SxKro49nrIFEltP8+DthHpHbCoZBkAZd5BgpVRQpVKaSVUG5IKL5FkGgH5hHguQwUqhp5
dlbLOifSXsHmot78LPDKKMCrESonlmcXgDy+NtkJ1lJF0mxJgwtdF4R3Vg4CO1D4VhkxRyfek8io
XPh5PYZMWghi32KKKt+zs/96NLfh3twFuFtNDrFpsuzCJGUOjC87YCcCoMcVqDw9O0U6IclLQbCe
uza+yo0GsY45BY4fgwsTV3dSPojji2/G1CIjL12MentVEuqPLAPey/JTSl5RGYJIoDK+iE4Do3iB
V0w1SnPTa/VCYoIKYg57Oq7k0pSdVKoGUsmNoLr1ZcX6liB7A13ZtnC/E4YIvVT5Tq2FB5RSkMdw
pb7sztY4qNjq1lcCzBcGQVa3QE4SnNwAPOk3UCzKqIsFcVCWjApGXlCdAVxpOdQuTRSmKYrLsT+3
IHGpOOsT4G6IhdWy1Rl7xFfSoUHKhdibJ9lnQFGa6J8EV+oGQYZeuPSgbXVlVkPjBtvpDfsnqjL0
JHVAGjpHJtvnPiFk5Ug2u0Rf8in6Cj+D/OcgDwGTViHhhXUGpgAAAABJRU5ErkJggg==
--_004_DM6PR17MB4181BC29946B9CDF07706998AC17ADM6PR17MB4181namp_--