On 09/16/2016 07:29 AM, Andrea Bolognani wrote:
> On Thu, 2016-09-15 at 01:04 -0400, Wei Huang wrote:
>> This patchset adds a pmu=[on/off] option to enable/disable vPMU support
>> for guest VM. There are several reasons to justify this option. First,
>> vPMU can be problematic for cross-migration between different SoC as perf
>> counters are architecture-dependent. It is more flexible to have an option
>> to turn it on/off. Secondly Secondly this option matches the "pmu" option
>> as supported in libvirt. To make sure backward compatible, a PMU property
>> is added to mach-virt machine types.
>
> [...]
>
>> Wei Huang (2):
>> arm64: Add an option to turn on/off vPMU support
>> arm: virt: add PMU property to mach-virt machine type
>>
>> hw/arm/virt-acpi-build.c | 2 +-
>> hw/arm/virt.c | 15 ++++++++++++++-
>> target-arm/cpu.c | 22 ++++++++++++++++++++++
>> target-arm/cpu.h | 1 +
>> target-arm/cpu64.c | 2 ++
>> target-arm/kvm64.c | 19 +++++++++++++++----
>> 6 files changed, 55 insertions(+), 6 deletions(-)
>
> This doesn't seem to be working :(
>
> My guest configuration looks like
>
> <domain type='kvm'>
> <name>abologna-rhel73</name>
> <uuid>78f74104-65f2-4faf-9faa-26107c1071ef</uuid>
> <memory unit='KiB'>2097152</memory>
> <currentMemory unit='KiB'>2097152</currentMemory>
> <vcpu placement='static'>4</vcpu>
> <os>
> <type arch='aarch64' machine='virt-2.7'>hvm</type>
> <loader readonly='yes'
> type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
> <nvram>/var/lib/libvirt/qemu/nvram/abologna-rhel73_VARS.fd</nvram>
> <boot dev='hd'/>
> </os>
> <features>
> <pmu state='off'/>
> <gic version='2'/>
> </features>
> <cpu mode='host-passthrough'/>
> <clock offset='utc'/>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>restart</on_crash>
> <devices>
> <emulator>/usr/libexec/abologna-qemu-kvm</emulator>
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2'/>
> <source file='/var/lib/libvirt/images/abologna-rhel73.qcow2'/>
> <target dev='vda' bus='virtio'/>
> <address type='virtio-mmio'/>
> </disk>
> <controller type='pci' index='0' model='pcie-root'/>
> <controller type='virtio-serial' index='0'>
> <address type='virtio-mmio'/>
> </controller>
> <interface type='network'>
> <mac address='52:54:00:57:76:ad'/>
> <source network='default'/>
> <model type='virtio'/>
> <address type='virtio-mmio'/>
> </interface>
> <serial type='pty'>
> <target port='0'/>
> </serial>
> <console type='pty'>
> <target type='serial' port='0'/>
> </console>
> <channel type='unix'>
> <target type='virtio' name='org.qemu.guest_agent.0'/>
> <address type='virtio-serial' controller='0' bus='0' port='1'/>
> </channel>
> </devices>
> </domain>
>
> and based on that configuration, libvirt creates a QEMU
> command line that looks like
>
> LC_ALL=C \
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
> QEMU_AUDIO_DRV=none \
> /usr/libexec/abologna-qemu-kvm \
> -name guest=abologna-rhel73,debug-threads=on \
> -S \
> -object
> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-abologna-rhel73/master-key.aes
> \
> -machine virt-2.7,accel=kvm,usb=off \
> -cpu host,pmu=off \
> -drive
> file=/usr/share/AAVMF/AAVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on \
> -drive
> file=/var/lib/libvirt/qemu/nvram/abologna-rhel73_VARS.fd,if=pflash,format=raw,unit=1
> \
> -m 2048 \
> -realtime mlock=off \
> -smp 4,sockets=4,cores=1,threads=1 \
> -uuid 78f74104-65f2-4faf-9faa-26107c1071ef \
> -nographic \
> -no-user-config \
> -nodefaults \
> -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-5-abologna-rhel73/monitor.sock,server,nowait
> \
> -mon chardev=charmonitor,id=monitor,mode=control \
> -rtc base=utc \
> -no-shutdown \
> -boot strict=on \
> -device virtio-serial-device,id=virtio-serial0 \
> -drive
> file=/var/lib/libvirt/images/abologna-rhel73.qcow2,format=qcow2,if=none,id=drive-virtio-disk0
> \
> -device
> virtio-blk-device,scsi=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> \
> -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=27 \
> -device virtio-net-device,netdev=hostnet0,id=net0,mac=52:54:00:57:76:ad \
> -serial pty \
> -chardev
> socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-5-abologna-rhel73/org.qemu.guest_agent.0,server,nowait
> \
> -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
> \
> -msg timestamp=on
>
> Inside the guest, however, even with pmu=off, I can still see
> the PMU being initialized during boot
>
> # dmesg | grep -i pmu
> [ 0.000000] ACPI-PMU: Assign CPU 0 girq 23 level 0
> [ 0.000000] ACPI-PMU: Assign CPU 1 girq 23 level 0
> [ 0.000000] ACPI-PMU: Assign CPU 2 girq 23 level 0
> [ 0.000000] ACPI-PMU: Assign CPU 3 girq 23 level 0
> [ 0.087277] ACPI-PMU: Setting up 4 PMUs for CPU type 0
> [ 0.576255] hw perfevents: enabled with armv8_pmuv3 PMU driver,
> 5 counters available
>
> and perf can list the counters
>
> # perf list | grep Hardware
> branch-misses [Hardware event]
> cache-misses [Hardware event]
> cache-references [Hardware event]
> cpu-cycles OR cycles [Hardware event]
> instructions [Hardware event]
> L1-dcache-load-misses [Hardware cache event]
> L1-dcache-loads [Hardware cache event]
> L1-dcache-store-misses [Hardware cache event]
> L1-dcache-stores [Hardware cache event]
> branch-load-misses [Hardware cache event]
> branch-loads [Hardware cache event]
> mem:<addr>[/len][:access] [Hardware breakpoint]
>
> and use them
>
> # perf stat true
>
> Performance counter stats for 'true':
>
> 0.610960 task-clock (msec) # 0.102 CPUs utilized
> 0 context-switches # 0.000 K/sec
> 0 cpu-migrations # 0.000 K/sec
> 24 page-faults # 0.039 M/sec
> 1,383,676 cycles # 2.265 GHz
> <not supported> stalled-cycles-frontend
> <not supported> stalled-cycles-backend
> 396,335 instructions # 0.29 insns per cycle
> <not supported> branches
> 5,536 branch-misses # 0.00% of all branches
>
> 0.006013060 seconds time elapsed
>
> same way it can when using pmu=on or not passing any pmu=
> value to QEMU.
>
> I've tested this both on Moonshot and on ThunderX, the
> behavior is the same.
>
> Am I doing something wrong?
I can replicate the problem. To answer your question: No, I think it is
a bug caused in this patch. Following code path, it turns out that
machvirt_init() is called after ARMCPU->has_pmu property has been
configured. So in your example, machvirt->pmu_default_on will overwrite
the ARMCPU->has_pmu value to be TRUE (always).
Even worse, our desired PMU setting can't be fully covered by a Boolean
ARMCPU->has_pmu field because machvirt_init() can't tell ARMCPU->has_pmu
is default-OFF or forced-OFF. Here is the matrix of one example:
ARMCPU->has_pmu VMC->pmu_default_on vPMU
-M virt-2.7 -cpu host default-OFF TRUE ON
-M virt-2.7 -cpu host,pmu=off forced-OFF TRUE OFF*
-M virt-2.7 -cpu host,pmu=on forced-ON TRUE ON
-M virt-2.6 -cpu host default-OFF FALSE OFF
-M virt-2.6 -cpu host,pmu=off forced-OFF FALSE OFF
-M virt-2.6 -cpu host,pmu=on forced-ON FALSE ON
-M virt-2.8+ -cpu host default-OFF FALSE OFF
-M virt-2.8+ -cpu host,pmu=off forced-OFF FALSE OFF
-M virt-2.8+ -cpu host,pmu=on forced-ON FALSE ON
The problematic one is the second line (*). To solve the problem, we
might want to change ARMCPU->has_pmu to an int (or enum).
>
> --
> Andrea Bolognani / Red Hat / Virtualization
>
