Public bug reported: We have test setup with 3 identical compute nodes. Recently test setup upgraded from Zed to 2023.1 Openstack Version, with nova 27.0.0, libvirt 8.0.0, qemu 6.2
During live migration we always getting error: 2023-06-05 13:33:06.158 1458525 ERROR nova.virt.libvirt.driver [None req-6295f150-f0cf-41d1-8dd4-60c0ac32223f 59de9e2e2a8a413384be5ee27e027fc1 185764021e19409dae135a967f032fa4 - - b589410bd7e14872bf3ac74c45057691 b589410bd7e14872bf3ac74c45057691] CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server [None req-6295f150-f0cf-41d1-8dd4-60c0ac32223f 59de9e2e2a8a413384be5ee27e027fc1 185764021e19409dae135a967f032fa4 - - b589410bd7e14872bf3ac74c45057691 b589410bd7e14872bf3ac74c45057691] Exception during message handling: nova.exception.MigrationPreCheckEr ror: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9637, in check_can_live_migrate_destination 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server self._compare_cpu(None, source_cpu_info, instance) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 10014, in _compare_cpu 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise exception.InvalidCPUInfo(reason=m % {'ret': ret, 'u': u}) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility. 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 0 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 65, in wrapped 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise self.value 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 63, in wrapped 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1439, in decorated_function 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 213, in decorated_function 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise self.value 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 203, in decorated_function 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 8408, in check_can_live_migrate_destination 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server dest_check_data = self.driver.check_can_live_migrate_destination(ctxt, 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9641, in check_can_live_migrate_destination 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise exception.MigrationPreCheckError(reason=e) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server nova.exception.MigrationPreCheckError: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. Zed setup works ok with this nodes. More info: 1. https://github.com/openstack/nova/commit/468b03e0ee4a917ae26106f6e57081bcd9e7a65b If revert libvirt to old cpu_compare instead cpu_hypervisor_compare in libvirt driver ret = self._host.compare_cpu(cpu_xml) #ret = self._host.compare_hypervisor_cpu(cpu_xml) live migration works ok. ret = self._host.compare_cpu(cpu_xml) - always returns 2 (VIR_CPU_COMPARE_SUPERSET) ret = self._host.compare_hypervisor_cpu(cpu_xml) - always returns 0 (VIR_CPU_COMPARE_INCOMPATIBLE) 2. Disabling cpu Comparation with changing nova.conf: [workarounds] skip_cpu_compare_on_dest = True makes live migration work ok. 3. Entry xml to cpu_compare looks like: <cpu> <arch>x86_64</arch> <model>Skylake-Client-noTSX-IBRS</model> <vendor>Intel</vendor> <topology sockets="1" cores="4" threads="2"/> <feature name="3dnowprefetch"/> <feature name="abm"/> <feature name="acpi"/> <feature name="adx"/> <feature name="aes"/> <feature name="apic"/> <feature name="arat"/> <feature name="arch-capabilities"/> <feature name="avx"/> <feature name="avx2"/> <feature name="bmi1"/> <feature name="bmi2"/> <feature name="clflush"/> <feature name="clflushopt"/> <feature name="cmov"/> <feature name="cx16"/> <feature name="cx8"/> <feature name="de"/> <feature name="ds"/> <feature name="ds_cpl"/> <feature name="dtes64"/> <feature name="erms"/> <feature name="est"/> <feature name="f16c"/> <feature name="fma"/> <feature name="fpu"/> <feature name="fsgsbase"/> <feature name="fxsr"/> <feature name="ht"/> <feature name="intel-pt"/> <feature name="invpcid"/> <feature name="invtsc"/> <feature name="lahf_lm"/> <feature name="lm"/> <feature name="mca"/> <feature name="mce"/> <feature name="md-clear"/> <feature name="mmx"/> <feature name="monitor"/> <feature name="movbe"/> <feature name="mpx"/> <feature name="msr"/> <feature name="mtrr"/> <feature name="nx"/> <feature name="pae"/> <feature name="pat"/> <feature name="pbe"/> <feature name="pcid"/> <feature name="pclmuldq"/> <feature name="pdcm"/> <feature name="pdpe1gb"/> <feature name="pge"/> <feature name="pni"/> <feature name="popcnt"/> <feature name="pse"/> <feature name="pse36"/> <feature name="rdrand"/> <feature name="rdseed"/> <feature name="rdtscp"/> <feature name="rsba"/> <feature name="sep"/> <feature name="smap"/> <feature name="smep"/> <feature name="smx"/> <feature name="spec-ctrl"/> <feature name="ss"/> <feature name="ssbd"/> <feature name="sse"/> <feature name="sse2"/> <feature name="sse4.1"/> <feature name="sse4.2"/> <feature name="ssse3"/> <feature name="stibp"/> <feature name="syscall"/> <feature name="tm"/> <feature name="tm2"/> <feature name="tsc"/> <feature name="tsc-deadline"/> <feature name="tsc_adjust"/> <feature name="vme"/> <feature name="vmx"/> <feature name="x2apic"/> <feature name="xgetbv1"/> <feature name="xsave"/> <feature name="xsavec"/> <feature name="xsaveopt"/> <feature name="xsaves"/> <feature name="xtpr"/> </cpu> nova.conf [libvirt] cpu_mode = host-model ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2023035 Title: 2023.1: Live migration fails with new compare_hypervisor_cpu method Status in OpenStack Compute (nova): New Bug description: We have test setup with 3 identical compute nodes. Recently test setup upgraded from Zed to 2023.1 Openstack Version, with nova 27.0.0, libvirt 8.0.0, qemu 6.2 During live migration we always getting error: 2023-06-05 13:33:06.158 1458525 ERROR nova.virt.libvirt.driver [None req-6295f150-f0cf-41d1-8dd4-60c0ac32223f 59de9e2e2a8a413384be5ee27e027fc1 185764021e19409dae135a967f032fa4 - - b589410bd7e14872bf3ac74c45057691 b589410bd7e14872bf3ac74c45057691] CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server [None req-6295f150-f0cf-41d1-8dd4-60c0ac32223f 59de9e2e2a8a413384be5ee27e027fc1 185764021e19409dae135a967f032fa4 - - b589410bd7e14872bf3ac74c45057691 b589410bd7e14872bf3ac74c45057691] Exception during message handling: nova.exception.MigrationPreCheckEr ror: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. 0 Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9637, in check_can_live_migrate_destination 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server self._compare_cpu(None, source_cpu_info, instance) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 10014, in _compare_cpu 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise exception.InvalidCPUInfo(reason=m % {'ret': ret, 'u': u}) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility. 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 0 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 65, in wrapped 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise self.value 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 63, in wrapped 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1439, in decorated_function 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 213, in decorated_function 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server self.force_reraise() 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise self.value 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 203, in decorated_function 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 8408, in check_can_live_migrate_destination 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server dest_check_data = self.driver.check_can_live_migrate_destination(ctxt, 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9641, in check_can_live_migrate_destination 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise exception.MigrationPreCheckError(reason=e) 2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server nova.exception.MigrationPreCheckError: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. Zed setup works ok with this nodes. More info: 1. https://github.com/openstack/nova/commit/468b03e0ee4a917ae26106f6e57081bcd9e7a65b If revert libvirt to old cpu_compare instead cpu_hypervisor_compare in libvirt driver ret = self._host.compare_cpu(cpu_xml) #ret = self._host.compare_hypervisor_cpu(cpu_xml) live migration works ok. ret = self._host.compare_cpu(cpu_xml) - always returns 2 (VIR_CPU_COMPARE_SUPERSET) ret = self._host.compare_hypervisor_cpu(cpu_xml) - always returns 0 (VIR_CPU_COMPARE_INCOMPATIBLE) 2. Disabling cpu Comparation with changing nova.conf: [workarounds] skip_cpu_compare_on_dest = True makes live migration work ok. 3. Entry xml to cpu_compare looks like: <cpu> <arch>x86_64</arch> <model>Skylake-Client-noTSX-IBRS</model> <vendor>Intel</vendor> <topology sockets="1" cores="4" threads="2"/> <feature name="3dnowprefetch"/> <feature name="abm"/> <feature name="acpi"/> <feature name="adx"/> <feature name="aes"/> <feature name="apic"/> <feature name="arat"/> <feature name="arch-capabilities"/> <feature name="avx"/> <feature name="avx2"/> <feature name="bmi1"/> <feature name="bmi2"/> <feature name="clflush"/> <feature name="clflushopt"/> <feature name="cmov"/> <feature name="cx16"/> <feature name="cx8"/> <feature name="de"/> <feature name="ds"/> <feature name="ds_cpl"/> <feature name="dtes64"/> <feature name="erms"/> <feature name="est"/> <feature name="f16c"/> <feature name="fma"/> <feature name="fpu"/> <feature name="fsgsbase"/> <feature name="fxsr"/> <feature name="ht"/> <feature name="intel-pt"/> <feature name="invpcid"/> <feature name="invtsc"/> <feature name="lahf_lm"/> <feature name="lm"/> <feature name="mca"/> <feature name="mce"/> <feature name="md-clear"/> <feature name="mmx"/> <feature name="monitor"/> <feature name="movbe"/> <feature name="mpx"/> <feature name="msr"/> <feature name="mtrr"/> <feature name="nx"/> <feature name="pae"/> <feature name="pat"/> <feature name="pbe"/> <feature name="pcid"/> <feature name="pclmuldq"/> <feature name="pdcm"/> <feature name="pdpe1gb"/> <feature name="pge"/> <feature name="pni"/> <feature name="popcnt"/> <feature name="pse"/> <feature name="pse36"/> <feature name="rdrand"/> <feature name="rdseed"/> <feature name="rdtscp"/> <feature name="rsba"/> <feature name="sep"/> <feature name="smap"/> <feature name="smep"/> <feature name="smx"/> <feature name="spec-ctrl"/> <feature name="ss"/> <feature name="ssbd"/> <feature name="sse"/> <feature name="sse2"/> <feature name="sse4.1"/> <feature name="sse4.2"/> <feature name="ssse3"/> <feature name="stibp"/> <feature name="syscall"/> <feature name="tm"/> <feature name="tm2"/> <feature name="tsc"/> <feature name="tsc-deadline"/> <feature name="tsc_adjust"/> <feature name="vme"/> <feature name="vmx"/> <feature name="x2apic"/> <feature name="xgetbv1"/> <feature name="xsave"/> <feature name="xsavec"/> <feature name="xsaveopt"/> <feature name="xsaves"/> <feature name="xtpr"/> </cpu> nova.conf [libvirt] cpu_mode = host-model To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2023035/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
