Default shard size was 64MB. I've tested to create another volume with a
512MB shard size and settings you provided.
I observe a performance improvement with `dd`, which does a sequential
write (from 1MBps to 50MBps). However, I find VM still slow : as an
example, `apt upgrade` is very slow to unpack packages, more than 1 hour
for around 400MB (90 packages).
Le 08/09/2021 à 11:34, Alex K a écrit :
On Wed, Sep 8, 2021 at 12:33 PM Alex K <[email protected]
<mailto:[email protected]>> wrote:
On Wed, Sep 8, 2021 at 12:15 PM Mathieu Valois <[email protected]
<mailto:[email protected]>> wrote:
Sorry for double post but I don't know if this mail has been
received.
Hello everyone,
I know this issue was already treated on this mailing list.
However none of the proposed solutions is satisfying me.
Here is my situation : I've got 3 hyperconverged gluster ovirt
nodes, with 6 network interfaces, bounded in bunches of 2
(management, VMs and gluster). The gluster network is on a
dedicated bound where the 2 interfaces are directly connected
to the 2 other ovirt nodes. Gluster is apparently using it :
# gluster volume status vmstore
Status of volume: vmstore
Gluster process TCP Port RDMA
Port Online Pid
------------------------------------------------------------------------------
Brick gluster-ov1:/gluster_bricks
/vmstore/vmstore 49152 0 Y 3019
Brick gluster-ov2:/gluster_bricks
/vmstore/vmstore 49152 0 Y 3009
Brick gluster-ov3:/gluster_bricks
/vmstore/vmstore
where 'gluster-ov{1,2,3}' are domain names referencing nodes
in the gluster network. This networks has 10Gbps capabilities :
# iperf3 -c gluster-ov3
Connecting to host gluster-ov3, port 5201
[ 5] local 10.20.0.50 port 46220 connected to 10.20.0.51
port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.16 GBytes 9.92 Gbits/sec
17 900 KBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec
0 900 KBytes
[ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec
4 996 KBytes
[ 5] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec
1 996 KBytes
[ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec
0 996 KBytes
[ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec
0 996 KBytes
[ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec
0 996 KBytes
[ 5] 7.00-8.00 sec 1.15 GBytes 9.91 Gbits/sec
0 996 KBytes
[ 5] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec
0 996 KBytes
[ 5] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec
0 996 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec
22 sender
[ 5] 0.00-10.04 sec 11.5 GBytes 9.86
Gbits/sec receiver
iperf Done.
However, VMs stored on the vmstore gluster volume has poor
write performances, oscillating between 100KBps and 30MBps. I
almost always observe a write spike (180Mbps) at the beginning
until around 500MB written, then it drastically falls at
10MBps, sometimes even less (100KBps). Hypervisors have 32
threads (2 sockets, 8 cores per socket, 2 threads per core).
Here is the volume settings :
Volume Name: vmstore
Type: Replicate
Volume ID: XXX
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: gluster-ov1:/gluster_bricks/vmstore/vmstore
Brick2: gluster-ov2:/gluster_bricks/vmstore/vmstore
Brick3: gluster-ov3:/gluster_bricks/vmstore/vmstore
Options Reconfigured:
performance.io-thread-count: 32 # was 16 by default.
cluster.granular-entry-heal: enable
storage.owner-gid: 36
storage.owner-uid: 36
cluster.lookup-optimize: off
server.keepalive-count: 5
server.keepalive-interval: 2
server.keepalive-time: 10
server.tcp-user-timeout: 20
network.ping-timeout: 30
server.event-threads: 4
client.event-threads: 8 # was 4 by default
cluster.choose-local: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
performance.strict-o-direct: on
network.remote-dio: off
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
auth.allow: *
user.cifs: off
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: on
I see you have not enabled sharding for the gluster volume that is
hosting VMs. I usually set a shard size of 512MB. Note though that
changing shard size in a live system will corrupt your data.
You need to wipe and set again.
I mean you have not changed the default sharde size which is I think 4MB.
I've seen also good perf with the following tweaks:
gluster volume set vms remote-dio enable
gluster volume set vms performance.write-behind-window-size 8MB
gluster volume set vms performance.cache-size 1GB
gluster volume set vms performance.io-thread-count 16
gluster volume set vms performance.readdir-ahead on
gluster volume set vms client.event-threads 8
gluster volume set vms server.event-threads 8
gluster volume set engine features.shard-block-size 512MB
When I naively write directly on the logical volume, which is
mounted on a material RAID5 3-disks array, I have interesting
performances:
# dd if=/dev/zero of=a bs=4M count=2048
2048+0 records in
2048+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 17.2485 s, 498
MB/s #urandom gives around 200MBps
Moreover, hypervisors have SSD which have been configured as
lvcache, but I'm unsure how to test it efficiently.
I can't find where is the problem, as every piece of the chain
is apparently doing well ...
Thanks anyone for helping me :)
--
téïcée <https://www.teicee.com/?pk_campaign=Email> *Mathieu
Valois*
Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760
Bretteville-sur-Odon
Bureau Vitré: Zone de la baratière - 12, route de Domalain -
35500 Vitré
02 72 34 13 20 | www.teicee.com
<https://www.teicee.com/?pk_campaign=Email>
téïcée sur facebook <https://www.facebook.com/teicee> téïcée
sur twitter <https://twitter.com/Teicee_fr> téïcée sur
linkedin <https://www.linkedin.com/company/t-c-e> téïcée sur
viadeo <https://fr.viadeo.com/fr/company/teicee> Datadocké
_______________________________________________
Users mailing list -- [email protected] <mailto:[email protected]>
To unsubscribe send an email to [email protected]
<mailto:[email protected]>
Privacy Statement: https://www.ovirt.org/privacy-policy.html
<https://www.ovirt.org/privacy-policy.html>
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
<https://www.ovirt.org/community/about/community-guidelines/>
List Archives:
https://lists.ovirt.org/archives/list/[email protected]/message/BTY7TJLMQUJ3QRTFEYMHAA2P437DXW6I/
<https://lists.ovirt.org/archives/list/[email protected]/message/BTY7TJLMQUJ3QRTFEYMHAA2P437DXW6I/>
--
téïcée <https://www.teicee.com/?pk_campaign=Email> *Mathieu Valois*
Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760
Bretteville-sur-Odon
Bureau Vitré: Zone de la baratière - 12, route de Domalain - 35500 Vitré
02 72 34 13 20 | www.teicee.com <https://www.teicee.com/?pk_campaign=Email>
téïcée sur facebook <https://www.facebook.com/teicee> téïcée sur twitter
<https://twitter.com/Teicee_fr> téïcée sur linkedin
<https://www.linkedin.com/company/t-c-e> téïcée sur viadeo
<https://fr.viadeo.com/fr/company/teicee> Datadocké
_______________________________________________
Users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/[email protected]/message/6CGAEH5J5M6FYNFNUJQNQB3TMJYXGMCL/
