Hi Collin,
in case network traffic encryption is enabled for your AFS client (check
with "fs getcrypt") a huge performance improvement can be achieved by
switching it off immediately after the client daemon starts, for example
in init script with "/usr/afsws/bin/fs setcrypt off" (specify full path
to "fs" command in your environment).
Thanks,
Andreas
On 15.08.23 00:29, Collin Gros wrote:
Dear OpenAFS community,
We are administrators for an OpenAFS environment of (what will be) about
400 users and are running into some performance issues, for which we
hope you might have some advice...
1. Do you have any sources we can look at that might help us in
adjusting configuration to improve performance? We read the man page for
`dafileserver` and messed around a lot with our arguments to
`dafileserver` (increasing them past the values set for -L, or Large)...
though we haven't noticed much of an improvement in performance through
our testing. See below for the configuration we currently have set for
`dafileserver` on all of our OpenAFS file servers.
2. Do you know what kind of read/write speed we should expect for an
enviroment/configuration of this size? It would be helpful for us to
know what we should be expecting in our environment as far as
performance is concerned.
===========================
Our performance test
===========================
Here are results from our testing with a binary file (7103053824 bytes
in size, or 6.7GB), copying it from one client to AFS:
client1: openSUSE 15.1
server: AFS file server that hosts the AFS volumes used for our testing
`scp`: client1 (local) -> server (local): 102.2MB/s (66s)
`cp`: client1 (local) -> client1 (AFS file space): 19.2MB/s (352s)
`cp`: client1 (AFS file space) -> client1 (AFS file space): 19.46MB/s
(348s)
Here are results from our testing with the same binary file (7103053824
bytes in size, or 6.7GB), copying it in parallel from two clients to the
same AFS volume:
client1 (local) -> server (AFS file space): 10.22MB/s (663s)
client2 (local) -> server (AFS file space): 9.69MB/s (699s)
client1 (AFS file space) -> client1 (AFS file space): 5.38MB/s (1258s)
client2 (AFS file space) -> client2 (AFS file space): 7MB/s (965s)
client1 (AFS file space) -> client1 (local): 13.15MB/s (515s)
client2 (AFS file space) -> client2 (local): 15.57MB/s (435s)
client1 total time taken: 2436s
client2 total time taken: 2099s
Here is a snapshot of what `top` looks like from the AFS file server
while the copy is taking place:
top - 16:14:14 up 5 days, 7:29, 2 users, load average: 1.06, 0.37,
0.26
Tasks: 297 total, 2 running, 294 sleeping, 1 stopped, 0 zombie
%Cpu0 : 17.3 us, 6.5 sy, 0.0 ni, 69.4 id, 1.7 wa, 1.0 hi, 4.1
si, 0.0 st
%Cpu1 : 16.2 us, 4.1 sy, 0.0 ni, 65.5 id, 13.2 wa, 0.7 hi, 0.3
si, 0.0 st
%Cpu2 : 5.0 us, 6.7 sy, 0.3 ni, 12.4 id, 63.2 wa, 1.0 hi, 11.4
si, 0.0 st
%Cpu3 : 7.5 us, 5.1 sy, 9.2 ni, 44.2 id, 31.5 wa, 1.4 hi, 1.0
si, 0.0 st
%Cpu4 : 13.3 us, 6.5 sy, 2.0 ni, 67.6 id, 9.9 wa, 0.7 hi, 0.0
si, 0.0 st
%Cpu5 : 37.4 us, 14.6 sy, 0.0 ni, 41.1 id, 6.0 wa, 0.7 hi, 0.3
si, 0.0 st
MiB Mem : 24080.5 total, 14283.7 free, 526.5 used, 9270.3
buff/cache
MiB Swap: 4060.0 total, 4060.0 free, 0.0 used. 23105.9
avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM
TIME+ COMMAND
22409 root 15 -5 4282356 65240 2808 S 118.3 0.3
75:55.61 dafileserver
Here is the output of `fs getcacheparms` while both clients were copying
the file to AFS:
client1: AFS using 781060 of the cache's available 891289 1K byte blocks.
client2: AFS using 0 of the cache's available 891289 1K byte blocks.
***************************
Our environment
***************************
We have our environment configuration documented below, and are hoping
you might give us some pointers as to what might be a performance
bottleneck.
Our testing environment:
- OpenAFS Servers
- OpenAFS 1.8.9
- DB servers (total of 3)
- 1 master
- Rocky Linux 8.8
- 2 CPU
- 4GB RAM
- 2 replicas, with each having:
- Rocky Linux 8.8
- 2 CPU
- 4GB RAM
- FS servers (total of 3)
- 3 fileservers, with each having:
- Rocky Linux 8.8
- 6 CPU
- 24GB RAM
- /usr/afs/local/BosConfig:
restrictmode 0
restarttime 16 0 0 0 0
checkbintime 3 0 5 0 0
bnode dafs dafs 1
parm /usr/afs/bin/dafileserver -L -cb 640000
-abortthreshold 0 -vc 1000
parm /usr/afs/bin/davolserver -p 64 -log
parm /usr/afs/bin/salvageserver
parm /usr/afs/bin/dasalvager -parallel all32
end
bnode simple upclientetc 1
parm /usr/afs/bin/upclient db1 /usr/afs/etc
end
bnode simple upclientbin 1
parm /usr/afs/bin/upclient db1 /usr/afs/bin
end
- OpenAFS Clients
- client1
- openSUSE 15.1
- OpenAFS 1.8.7
- 6 CPUs
- 16GB RAM
- `fs getcacheparms`
AFS using 12 of the cache's available 891289 1K byte blocks.
- /etc/sysconfig/openafs-client:
AFSD_ARGS="-fakestat -stat 6000 -dcache 6000 -daemons 6
-volumes 256 -files 50000 -chunksize 17"
- client2
- openSUSE 13.2
- OpenAFS 1.8.7
- 2 CPUs
- 2GB RAM
- `fs getcacheparms`
AFS using 0 of the cache's available 891289 1K byte blocks.
- /etc/sysconfig/afs
OPTIONS=$XXLARGE
(and XXLARGE="-fakestat -stat 4000 -dcache 4000 -daemons
6 -volumes 256 -afsdb")
Thanks for the help!!
Regards,
Collin
*Collin Gros*
*Staff Software Engineer*
*RICOH Graphic Communications - DSBC*
*Ricoh USA, Inc*
Phone: +1 720-663-3225
Email: [email protected]
_______________________________________________
OpenAFS-info mailing list
[email protected]
https://lists.openafs.org/mailman/listinfo/openafs-info
