Thanks for the advice and feedback!
>> I'm encountering an issue with live migrating between 2 Compute nodes with
>> different CPU models. Host A has a Westmere CPU and host B has a Broadwell.
>That’s expected unless you specify the least common denominator with the cpu
>model, in this case Westmere, when you boot the instances.
>
>> I can migrate an instance from the Westmere host A to the Broadwell host B
>> Yes, you can move an instance from a host with less cpu features to more
>> cpu features as long as you do not configure cpu-passthrough AFAIK.
>
>> but when I attempt to live migrate a VM from the Broadwell host B to
>> the older Westmere host I get the following error; Live Migration failure:
>> unsupported configuration: guest and host CPU are not compatible:
>> Host CPU does not provide required features: fma, x2apic, movbe,
>> tsc-deadline, xsave, osxsave, avx, ….
>That’s because the instances booted with those extra features enabled.
>(you can also see feature when grepping for the qemu process in the
>processlist, you will see the configured cpu in the qemu commandline) The
>Westmere CPU does not support avx/avx2 etc.
Therein lies the problem, the VM is running using only CPU features that are
compatible with both CPUs. The CPU features libvirt says are missing are not in
use by the VM. It was first booted on a host with a Westmere CPU, successfully
live migrated to host with Broadwell CPU, but then refuses to migrate BACK to
the Westmere host. The qemu-kvm process was launched with the following CPU
parameters;
"-cpu
Westmere,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+pclmuldq,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+dca,+arat,+pdpe1gb,+rdtscp"
And from "virsh dumpxml <instance-id>" the VM running with the following XML
CPU config;
<cpu mode='host-model'>
<model fallback='allow'>Westmere</model>
<vendor>Intel</vendor>
<topology sockets='1' cores='1' threads='1'/>
<feature policy='require' name='vme'/>
<feature policy='require' name='ds'/>
<feature policy='require' name='acpi'/>
<feature policy='require' name='ss'/>
<feature policy='require' name='ht'/>
<feature policy='require' name='tm'/>
<feature policy='require' name='pbe'/>
<feature policy='require' name='pclmuldq'/>
<feature policy='require' name='dtes64'/>
<feature policy='require' name='monitor'/>
<feature policy='require' name='ds_cpl'/>
<feature policy='require' name='vmx'/>
<feature policy='require' name='smx'/>
<feature policy='require' name='est'/>
<feature policy='require' name='tm2'/>
<feature policy='require' name='xtpr'/>
<feature policy='require' name='pdcm'/>
<feature policy='require' name='pcid'/>
<feature policy='require' name='dca'/>
<feature policy='require' name='arat'/>
<feature policy='require' name='pdpe1gb'/>
<feature policy='require' name='rdtscp'/>
</cpu>
Which is compatible with the Westmere host CPU. The Westmere hosts capabilities
from "virsh capabilities";
<host>
<uuid>35333036-3936-4e43-3731-343030485837</uuid>
<cpu>
<arch>x86_64</arch>
<model>Westmere</model>
<vendor>Intel</vendor>
<topology sockets='1' cores='4' threads='2'/>
<feature name='vme'/>
<feature name='ds'/>
<feature name='acpi'/>
<feature name='ss'/>
<feature name='ht'/>
<feature name='tm'/>
<feature name='pbe'/>
<feature name='pclmuldq'/>
<feature name='dtes64'/>
<feature name='monitor'/>
<feature name='ds_cpl'/>
<feature name='vmx'/>
<feature name='smx'/>
<feature name='est'/>
<feature name='tm2'/>
<feature name='xtpr'/>
<feature name='pdcm'/>
<feature name='pcid'/>
<feature name='dca'/>
<feature name='arat'/>
<feature name='pdpe1gb'/>
<feature name='rdtscp'/>
<feature name='invtsc'/>
<pages unit='KiB' size='4'/>
<pages unit='KiB' size='2048'/>
<pages unit='KiB' size='1048576'/>
</cpu>
The instance can also be migrated between 2 Westmere hosts. As I outline in
another email, I suspect (perhaps incorrectly) as the VM has "cpu
mode='host-model'" in its running config, libvirt may be comparing source and
destination host CPUs and not guest and destination CPUs.
>>How can I avoid this error, without rebooting any VMs?
>You cannot do that for any instance which is already running with the extra
>cpu flags.
>Basically, you would need to remove cpu-features on the fly when
>live-migrating.
>That’s not possible.
>> Host B with the Broadwell CPU has the libvirt CPU masking enabled in its
>> nova.conf;
>> [libvirt]
>> cpu_mode = custom
>> cpu_model = Westmere
>>
>> Host A with the Westmere CPU, never had the cpu_mode set in nova.conf but
>> as its KVM it should default to the following;
>> [libvirt]
>> cpu_mode = host-model
>>
>According the docs :"host-model" - this causes libvirt to identify the named
>CPU model which most closely matches the host from the above list, and then
>request additional CPU flags to complete the match.
>If you want full live migration compatibility across different cpu generations
>I would recommend to set the cpu mode and model everywhere to the same values.
>> Interestingly once a host is booted on Host B and has the above CPU config,
>> it can be migrated back to Host A without any error.
>I would expect that, as long as the destination CPU has all the cpu features
>where the qemu process was started with (in this case Westmere profile) it
>should work.
>
>There have been a couple of bugs in the past with the comparison of cpu
>features but I think what you are seeing is expected behavior.
>
>Note that there is quite a bit of change between Westmere and Broadwell and
>you might really want those extra cpu features.
>e.g. avx2 can seriously improve SSL offloading (200% improvement with the
>correct cypher suites)
This is defiantly in the longer term plan, but for the moment backwards
compatibility is a requirement for us.
>Cheers,
>Robert van Leeuwen
Oisin O'Malley
Systems Engineer
Iocane Pty Ltd
763 South Road
Black Forest SA 5035
Office:+61 (8) 8413 1010
Fax:+61 (8) 8231 2050
Email:oisin.omal...@iocane.com.au
Web:www.iocane.com.au
Better for business
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
