Below is the top output of running "top -bHd d" on one of the nodes, maybe that
can help to see what that glusterfsd process is doing?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4375 root 20 0 2856784 120492 8360 D 61.1 0.4 117:09.29 glfs_iotwr001
4385 root 20 0 2856784 120492 8360 R 61.1 0.4 117:12.92 glfs_iotwr003
4387 root 20 0 2856784 120492 8360 R 61.1 0.4 117:32.19 glfs_iotwr005
4388 root 20 0 2856784 120492 8360 R 61.1 0.4 117:28.87 glfs_iotwr006
4391 root 20 0 2856784 120492 8360 D 61.1 0.4 117:20.71 glfs_iotwr008
4395 root 20 0 2856784 120492 8360 D 61.1 0.4 117:17.22 glfs_iotwr009
4405 root 20 0 2856784 120492 8360 R 61.1 0.4 117:19.52 glfs_iotwr00d
4406 root 20 0 2856784 120492 8360 R 61.1 0.4 117:29.51 glfs_iotwr00e
4366 root 20 0 2856784 120492 8360 D 55.6 0.4 117:27.58 glfs_iotwr000
4386 root 20 0 2856784 120492 8360 D 55.6 0.4 117:22.77 glfs_iotwr004
4390 root 20 0 2856784 120492 8360 D 55.6 0.4 117:26.49 glfs_iotwr007
4396 root 20 0 2856784 120492 8360 R 55.6 0.4 117:23.68 glfs_iotwr00a
4376 root 20 0 2856784 120492 8360 D 50.0 0.4 117:36.17 glfs_iotwr002
4397 root 20 0 2856784 120492 8360 D 50.0 0.4 117:11.09 glfs_iotwr00b
4403 root 20 0 2856784 120492 8360 R 50.0 0.4 117:26.34 glfs_iotwr00c
4408 root 20 0 2856784 120492 8360 D 50.0 0.4 117:27.47 glfs_iotwr00f
9814 root 20 0 2043684 75208 8424 D 22.2 0.2 50:15.20 glfs_iotwr003
28131 root 20 0 2043684 75208 8424 R 22.2 0.2 50:07.46 glfs_iotwr004
2208 root 20 0 2043684 75208 8424 R 22.2 0.2 49:32.70 glfs_iotwr008
2372 root 20 0 2043684 75208 8424 R 22.2 0.2 49:52.60 glfs_iotwr009
2375 root 20 0 2043684 75208 8424 D 22.2 0.2 49:54.08 glfs_iotwr00c
767 root 39 19 0 0 0 R 16.7 0.0 67:50.83 dbuf_evict
4132 onadmin 20 0 45292 4184 3176 R 16.7 0.0 0:00.04 top
28484 root 20 0 2043684 75208 8424 R 11.1 0.2 49:41.34 glfs_iotwr005
2376 root 20 0 2043684 75208 8424 R 11.1 0.2 49:49.49 glfs_iotwr00d
2719 root 20 0 2043684 75208 8424 R 11.1 0.2 49:58.61 glfs_iotwr00e
4384 root 20 0 2856784 120492 8360 S 5.6 0.4 4:01.27 glfs_rpcrqhnd
3842 root 20 0 2043684 75208 8424 S 5.6 0.2 0:30.12 glfs_epoll001
1 root 20 0 57696 7340 5248 S 0.0 0.0 0:03.59 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:09.57 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.16 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root 20 0 0 0 0 S 0.0 0.0 0:07.36 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root rt 0 0 0 0 S 0.0 0.0 0:00.03 migration/0
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
11 root rt 0 0 0 0 S 0.0 0.0 0:00.01 watchdog/0
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/1
Any clues anyone?
The load is really high around 20 now on the two nodes...
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, November 5, 2020 11:50 AM, mabi <[email protected]> wrote:
> Hello,
>
> I have a 3 node replica including arbiter GlusterFS 7.8 server with 3 volumes
> and the two nodes (not arbiter) seem to have a high load due to the
> glusterfsd brick process taking all CPU resources (12 cores).
>
> Checking these two servers with iostat command shows that the disks are not
> so busy and that they are mostly doing writes activity. On the FUSE clients
> there is not so much activity so I was wondering how to find out or explain
> why GlusterFS is currently generating such a high load on these two servers
> (the arbiter does not show any high load). There are no files currently
> healing either. This volume is the only volume which has the quota enabled if
> this might be a hint. So does anyone know how to see why GlusterFS is so busy
> on a specific volume?
>
> Here is a sample "vmstat 60" of one of the nodes:
>
> onadmin@gfs1b:~$ vmstat 60
> procs -----------memory---------- ---swap-- -----io---- -system--
> ------cpu-----
> r b swpd free buff cache si so bi bo in cs us sy id wa st
> 9 2 0 22296776 32004 260284 0 0 33 301 153 39 2 60 36 2 0
> 13 0 0 22244540 32048 260456 0 0 343 2798 10898 367652 2 80 16 1 0
> 18 0 0 22215740 32056 260672 0 0 308 2524 9892 334537 2 83 14 1 0
> 18 0 0 22179348 32084 260828 0 0 169 2038 8703 250351 1 88 10 0 0
>
> I already tried rebooting but that did not help and there is nothing special
> in the log files either.
>
> Best regards,
> Mabi
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
[email protected]
https://lists.gluster.org/mailman/listinfo/gluster-users
