You also might want to set the allocation mode to something else then shared. This causes the qcow2 metadata to be pre-allocated and that will improve performance.
Wido On 5/17/19 3:04 PM, Ivan Kudryavtsev wrote: > Well, just FYI, I changed cache_mode from NULL (none), to writethrough > directly in DB and the performance boosted greatly. It may be an important > feature for NVME drives. > > Currently, on 4.11, the user can set cache-mode for disk offerings, but > cannot for service offerings, which are translated to cache=none > corresponding disk offerings. > > The only way is to use SQL to change that for root disk disk offerings. > CreateServiceOffering API doesn't support cache mode. It can be a serious > limitation for NVME users, because by default they could meet poor > read/write performance. > > пт, 17 мая 2019 г., 19:30 Ivan Kudryavtsev <kudryavtsev...@bw-sw.com>: > >> Darius, thanks for your participation, >> >> first, I used 4.14 kernel which is the default one for my cluster. Next, >> switched to 4.15 with dist-upgrade. >> >> Do you have an idea how to turn on amount of queues for virtio-scsi with >> Cloudstack? >> >> пт, 17 мая 2019 г., 19:26 Darius Kasparavičius <daz...@gmail.com>: >> >>> Hi, >>> >>> I can see a few issues with your xml file. You can try using "queues" >>> inside your disk definitions. This should help a little, not sure by >>> how much for your case, but for my specific it went up by almost the >>> number of queues. Also try cache directsync or writethrough. You >>> should switch kernel if bugs are still there with 4.15 kernel. >>> >>> On Fri, May 17, 2019 at 12:14 PM Ivan Kudryavtsev >>> <kudryavtsev...@bw-sw.com> wrote: >>>> >>>> Hello, colleagues. >>>> >>>> Hope, someone could help me. I just deployed a new VM host with Intel >>> P4500 >>>> local storage NVMe drive. >>>> >>>> From Hypervisor host I can get expected performance, 200K RIOPS, 3GBs >>> with >>>> FIO, write performance is also high as expected. >>>> >>>> I've created a new KVM VM Service offering with virtio-scsi controller >>>> (tried virtio as well) and VM is deployed. Now I try to benchmark it >>> with >>>> FIO. Results are very strange: >>>> >>>> 1. Read/Write with large blocks (1M) shows expected performance (my >>> limits >>>> are R=1000/W=500 MBs). >>>> >>>> 2. Write with direct=0 leads to expected 50K IOPS, while write with >>>> direct=1 leads to very moderate 2-3K IOPS. >>>> >>>> 3. Read with direct=0, direct=1 both lead to 3000 IOPS. >>>> >>>> During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0% which >>> is >>>> strange. >>>> >>>> So, basically, from inside VM my NVMe works very slow when small IOPS >>> are >>>> executed. From the host, it works great. >>>> >>>> I tried to mount the volume with NBD to /dev/nbd0 and benchmark. Read >>>> performance is nice. Maybe someone managed to use NVME with KVM with >>> small >>>> IOPS? >>>> >>>> The filesystem is XFS, previously tried with EXT4 - results are the >>> same. >>>> >>>> This is the part of VM XML definition generated by CloudStack: >>>> >>>> <devices> >>>> <emulator>/usr/bin/kvm-spice</emulator> >>>> <disk type='file' device='disk'> >>>> <driver name='qemu' type='qcow2' cache='none' discard='unmap'/> >>>> <source >>>> file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/> >>>> <backingStore type='file' index='1'> >>>> <format type='raw'/> >>>> <source >>>> file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/> >>>> <backingStore/> >>>> </backingStore> >>>> <target dev='sda' bus='scsi'/> >>>> <iotune> >>>> <read_bytes_sec>1048576000</read_bytes_sec> >>>> <write_bytes_sec>524288000</write_bytes_sec> >>>> <read_iops_sec>100000</read_iops_sec> >>>> <write_iops_sec>50000</write_iops_sec> >>>> </iotune> >>>> <serial>6809dbd04a1540149322</serial> >>>> <alias name='scsi0-0-0-0'/> >>>> <address type='drive' controller='0' bus='0' target='0' unit='0'/> >>>> </disk> >>>> <disk type='file' device='cdrom'> >>>> <driver name='qemu' type='raw'/> >>>> <backingStore/> >>>> <target dev='hdc' bus='ide'/> >>>> <readonly/> >>>> <alias name='ide0-1-0'/> >>>> <address type='drive' controller='0' bus='1' target='0' unit='0'/> >>>> </disk> >>>> <controller type='scsi' index='0' model='virtio-scsi'> >>>> <alias name='scsi0'/> >>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' >>>> function='0x0'/> >>>> </controller> >>>> >>>> So, what I see now, is that it works slower than couple of two Samsung >>> 960 >>>> PRO which is extremely strange. >>>> >>>> Thanks in advance. >>>> >>>> >>>> -- >>>> With best regards, Ivan Kudryavtsev >>>> Bitworks LLC >>>> Cell RU: +7-923-414-1515 >>>> Cell USA: +1-201-257-1512 >>>> WWW: http://bitworks.software/ <http://bw-sw.com/> >>> >> >
