I’ve tried various methods to improve gluster performance on similar hardware and never had much luck. Small file workloads were particularly troublesome. I ended up switching high performance vms to nfs storage and performance with nfs improved greatly in my use case.
On Sun, Jun 28, 2020 at 6:42 PM shadow emy <[email protected]> wrote: > > Hello , > > Hello and thank you for the reply.Bellow are the answers to your questions. > > > > Let me ask some questions: > > 1. What is the scheduler for your PV ? > > > On the Raid Controller device where the SSD disks are in Raid 0 (device > sda) it is set to "deadline". But on the lvm volume logical volume dm-7, > where the logical block is set for "data" volunr it is set to none.(i think > this is ok ) > > > [root@host1 ~]# ls -al /dev/mapper/gluster_vg_sd > v_data ter_l > lrwxrwxrwx. 1 root root 7 Jun 28 14:14 /dev/mapper/gluster_v > g_sda3-gluster_lv_data -> ../dm-7 > [root@host1 ~]# cat /sys/block/dm-7/queue/scheduler > none > root@host1:~[root@host1 ~]# cat /sys/block/dm-7/queue/schedu > [root@host1 ~]# cat /sys/block/sda/queue/scheduler > noop [deadline] cfq > > > > > 2. Have you aligned your PV during the setup 'pvcreate > --dataalignment alignment_value > > device' > > > I did not made other alignment then the default.Bellow are the partitions > on /dev/sda. > Can i enable partition alignment now, if yes how ? > > sfdisk -d /dev/sda > # partition table of /dev/sda > unit: sectors > > /dev/sda1 : start= 2048, size= 487424, Id=83, bootable > /dev/sda2 : start= 489472, size= 95731712, Id=8e > /dev/sda3 : start= 96221184, size=3808675840, Id=83 > /dev/sda4 : start= 0, size= 0, Id= 0 > > > > > 3. What is your tuned profile ? Do you use rhgs-random-io from > > the > ftp://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHS/SRPMS/red... > > ? > > My tuned active profile is virtual-host > > Current active profile: virtual-host > > No i dont use any of the rhgs-random-io profiles > > > 4. What is the output of "xfs_info /path/to/your/gluster/brick" ? > > xfs_info /gluster_bricks/data > meta-data=/dev/mapper/gluster_vg_sda3-gluster_lv_data isize= > 512 agcount=32, agsize=6553600 blks > = sectsz=512 attr=2, projid > 32bit=1 > = crc=1 finobt=0 spino > des=0 > data = bsize=4096 blocks=2097152 > 00, imaxpct=25 > = sunit=64 swidth=64 blks > naming =version 2 bsize=8192 ascii-ci=0 fty > pe=1 > log =internal bsize=4096 blocks=102400, > version=2 > = sectsz=512 sunit=64 blks, > lazy-count=1 > realtime =none extsz=4096 blocks=0, rtex > tents=0 > > > 5. Are you using Jumbo Frames ? Does your infra support them? > > Usually MTU of 9k is standard, but some switches and NICs support up to > 16k. > > > > Unfortunately I can not enable MTU to 9000 and Jumbo Frames on these > Cisco SG350X switches to specific ports.The switches dont suport Jumbo > Frames enable to a single port, only on all ports . > I have others devices connected to the switches on the remaining 48 ports > that have 1Gb/s. > > > All the options for "optimize for virt...." are located > > at /var/lib/glusterd/groups/virt on each gluster node. > > I have already looked previously at that file, but not all the volume > settings that are set by "Optime for Virt Store" are stored there. > For example "Optimize for Virt Store " sets network.remote.dio to > disable and in the glusterd/groups/virt is set to enabled.Or > cluster.granular-entry-heal: enable is not present there, bit it is set by > "Optimize for Virt Store" > > > > > Best Regards, > > Strahil Nikolov > > > > > > > > > > В неделя, 28 юни 2020 г., 22:13:09 Гринуич+3, jury cat <shadow.emy1(a) > gmail.com> > > написа: > > > > > > > > > > > > Hello all, > > > > I am using Ovirt 4.3.10 on Centos 7.8 with glusterfs 6.9 . > > My Gluster setup is of 3 hosts in replica 3 (2 hosts + 1 arbiter). > > All the 3 hosts are Dell R720 with Perc Raid Controller H710 mini(that > has maximim > > throughtout 6Gbs) and with 2×1TB samsumg SSD in RAID 0. The volume is > partitioned using > > LVM thin provision and formated XFS. > > The hosts have separate 10GE network cards for storage traffic. > > The Gluster Network is connected to this 10GE network cards and is > mounted using Fuse > > Glusterfs(NFS is disabled).Also Migration Network is activated on the > same storage > > network. > > > > > > The problem is that the 10GE network is not used at full potential by > the Gluster. > > If i do live Migration of Vms i can see speeds of 7GB/s ~ 9GB/s. > > The same network tests using iperf3 reported 9.9GB/s , these exluding > the network setup > > as a bottleneck(i will not paste all the iperf3 tests here for now). > > I did not enable all the Volume options from "Optimize for Virt Store", > because > > of the bug that cant set volume cluster.granural-heal to enable(this > was fixed in vdsm-4 > > 40, but that is working only on Centos 8 with ovirt 4.4 ) . > > i whould be happy to know what are all these "Optimize for Virt Store" > options, > > so i can set them manually. > > > > > > The speed on the disk inside the host using dd is b etween 1GB/s to > 700Mbs. > > > > > > [root@host1 ~]# dd if=/dev/zero of=test bs=100M count=40 cou nt=80 > status=progress > > 8074035200 bytes (8.1 GB) copied, 11.059372 s, 730 MB/s 80+0 records in > 80+0 records out > > 8388608000 bytes (8.4 GB) copied, 11.9928 s, 699 MB/s > > > > > > The dd write test on the gluster volme inside the host is poor only ~ > 120MB/s . > > During the dd test, if i look at Networks->Gluster network ->Hosts at Tx > and Rx the > > network speed barerly reaches over 1Gbs (~1073 Mbs) out of maximum of > 10000 Mbs. > > > > > > dd if=/dev/zero of=/rhev/data-center/mnt/glu > sterSD/gluster1.domain.local\: _data/test > > bs=100M count=80 status=progress 8283750400 bytes (8.3 GB) copied, > 71.297942 s, 116 MB/s > > 80+0 records in 80+0 records out 8388608000 bytes (8.4 GB) copied, > 71.9545 s, 117 MB/s > > > > > > I have attached my Gluster volume settings and mount options. > > > > Thanks, > > Emy > > > > > > _______________________________________________ > > Users mailing list -- users(a)ovirt.org > > To unsubscribe send an email to users-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/privacy-policy.html > > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > > https://lists.ovirt.org/archives/list/[email protected]/message/7BR6TZQ4EXS. > .. > _______________________________________________ > Users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/[email protected]/message/R5XPLZ7QARJG2TYRTOK6BADJDNONPOPQ/ >
_______________________________________________ Users mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/XSBM4D4PS6R6FNYP3MZDNAI7X4PKIMQT/
