Hi Team,
We are observing abnormal variation in VM launch time w.r.t guest config.
A simple VM(2gb ram no passthrough device) creation takes usually 6sec
(Time from execution of 'virsh create guest.xml to get print "Welcome to
FreeBSD"' ),but when we add a USB passthrough device launch time increased
to 18-19 sec and further increased to 39-44sec when we have increased guest
ram to 48Gb.
The increase in time is too high than excepted. Hence this is marked as
performance blocker in our test.
Here is the table summarises our experiments .
--------------------------------------------------------------------------------------------|---------------------|--------------------
Sl | Case
| time taken to launch vm
--------------------------------------------------------------------------------------------|---------------------|--------------------
1. | Freebsd 11 with 2gb guest ram and no pci pass through
| 6-7 sec
2. | Freebsd 11 with 48gb guest ram and no pci pass through
| 6-7 sec
3. | Freebsd 11 with 48gb guest ram and usb pci pass-through
| 39-44 sec
4. | Freebsd 11 with 2gb guest ram and usb pci pass-through
| 18-19 sec
Attaching guest config:
1. vm_2gb_usbpassthrough.xml --- Feebsd with 2gb RAM and
USB as pci pass through device
2. vm_48gb_usbpassthrough.xml --- Feebsd with 48gb RAM and
USB as pci pass through device
3. vm_48gb_nopci.xml --- Feebsd with 48gb RAM and
no pci device passed
Note : 1) This we have seen with legacy pci passthrough device not with
vfio. And we have not noticed any perfoance impact other than qemu-init/vm
laucn time .
2). We are using "FreeBSD-11.2-RELEASE-amd64.qcow2"(changed to raw
format) image downloaded from
https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/amd64/Latest/
-> Then modified the settings to set console to serial refer to the link ->
https://www.freebsd.org/doc/handbook/serialconsole-setup.html
-> Changed the /boot/default/loader.conf
add the console="comconsole"
and then change the /etc/ttys
USB hub device details :
------------------------------
root@root-node:/vm/vm# lspci -v -s 00:14.0
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI
Host Controller (rev 05) (prog-if 30 [XHCI])
Subsystem: Intel Corporation C610/X99 series chipset USB xHCI
Host Controller
Flags: medium devsel, IRQ 16, NUMA node 0
Memory at 183fc0000000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [70] Power Management version 2
Kernel driver in use: pci-stub
root@root-node:/vm/vm#
root@root-node:/vm/vm# virsh version
Compiled against library: libvirt 1.3.5
Using library: libvirt 1.3.5
Using API: QEMU 1.3.5
Running hypervisor: QEMU 2.7.0
root@root-node:/vm/vm# uname -a
Linux root-node 4.8.28-rt10-WR9.0.0.24_ovp #1 SMP Wed Feb 19 12:47:32 PST
2020 x86_64 x86_64 x86_64 GNU/Linux
root@root-node:/vm/vm#
Host CPU details :
CPU model : Haswell E
microcode : 0x3d
cpu cores : 6
Have you ever faced similar issues ? is it a qemu bug ?
Any suggestions to improve launch time with legacy passthrough ?
Regards,
Gokul
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>Freebsd</name>
<title>Generic bsd guest</title>
<description>Generated by libvirt-xml-examples.py.</description>
<memory unit='GiB'>2</memory>
<vcpu placement='static'>1</vcpu>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
</os>
<features>
<acpi/>
</features>
<cpu mode='custom'>
<topology sockets='1' cores='1' threads='1'/>
</cpu>
<clock offset='variable' adjustment='0' basis='utc'>
<timer name='rtc' tickpolicy='catchup'/>
</clock>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='directsync' io='threads'/>
<source file='/vm/vm/FreeBSD-11.2-RELEASE-amd64-2.raw'/>
<target dev='vda' bus='virtio'/>
<serial>VDA</serial>
<boot order='1'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='fa:da:83:fc:ec:db'/>
<source bridge='virbr0'/>
<target dev='vnet0'/>
<model type='virtio'/>
<link state='up'/>
<boot order='3'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<channel type='unix'>
<target type='virtio' name='default'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<alias name='channel1'/>
<address type='virtio-serial' controller='0' bus='0' port='2'/>
</channel>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</memballoon>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
</devices>
<qemu:commandline>
<qemu:arg value='-device'/>
<qemu:arg value='kvm-pci-assign,host=0000:00:14.0'/>
</qemu:commandline>
</domain><domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>Freebsd</name>
<title>Generic bsd guest</title>
<description>Generated by libvirt-xml-examples.py.</description>
<memory unit='GiB'>48</memory>
<vcpu placement='static'>1</vcpu>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
</os>
<features>
<acpi/>
</features>
<cpu mode='custom'>
<topology sockets='1' cores='1' threads='1'/>
</cpu>
<clock offset='variable' adjustment='0' basis='utc'>
<timer name='rtc' tickpolicy='catchup'/>
</clock>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='directsync' io='threads'/>
<source file='/vm/vm/FreeBSD-11.2-RELEASE-amd64-2.raw'/>
<target dev='vda' bus='virtio'/>
<serial>VDA</serial>
<boot order='1'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='fa:da:83:fc:ec:db'/>
<source bridge='virbr0'/>
<target dev='vnet0'/>
<model type='virtio'/>
<link state='up'/>
<boot order='3'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<channel type='unix'>
<target type='virtio' name='default'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<alias name='channel1'/>
<address type='virtio-serial' controller='0' bus='0' port='2'/>
</channel>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</memballoon>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
</devices>
</domain>
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>Freebsd</name>
<title>Generic bsd guest</title>
<description>Generated by libvirt-xml-examples.py.</description>
<memory unit='GiB'>48</memory>
<vcpu placement='static'>1</vcpu>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
</os>
<features>
<acpi/>
</features>
<cpu mode='custom'>
<topology sockets='1' cores='1' threads='1'/>
</cpu>
<clock offset='variable' adjustment='0' basis='utc'>
<timer name='rtc' tickpolicy='catchup'/>
</clock>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='directsync' io='threads'/>
<source file='/vm/vm/FreeBSD-11.2-RELEASE-amd64-2.raw'/>
<target dev='vda' bus='virtio'/>
<serial>VDA</serial>
<boot order='1'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='fa:da:83:fc:ec:db'/>
<source bridge='virbr0'/>
<target dev='vnet0'/>
<model type='virtio'/>
<link state='up'/>
<boot order='3'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<channel type='unix'>
<target type='virtio' name='default'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<alias name='channel1'/>
<address type='virtio-serial' controller='0' bus='0' port='2'/>
</channel>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</memballoon>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
</devices>
<qemu:commandline>
<qemu:arg value='-device'/>
<qemu:arg value='kvm-pci-assign,host=0000:00:14.0'/>
</qemu:commandline>
</domain>
