Matt, This was a bandwidth issue. Nova kept being connected to the broker, but it started to timeout on RPC messages, e.g., we lost some RPC messages triggering post live migration steps which are there to update nova DB to reflect new host of an instance.
The good workaround for this issue is to slightly limit bandwidth used for live migrations through nova.conf [1], live_migration_bandwidth config option in [libvirt] section. By default it is set to 0, so, basically, it is unlimited. Also please be aware that we changed default live migration configuration in OpenStack Newton and tunneling is now off by default (live_migration_tunnelled in [libvirt] section is set to False) due to huge performance impact. [1] http://docs.openstack.org/newton/config-reference/compute/config-options.html Kind Regards, Pawel Koniszewski From: tadow...@gmail.com [mailto:tadow...@gmail.com] On Behalf Of Matt Fischer Sent: Friday, December 30, 2016 5:31 AM To: Koniszewski, Pawel <pawel.koniszew...@intel.com> Cc: openstack-operators@lists.openstack.org Subject: Re: [Openstack-operators] [nova] Live migration performance tests on 100 compute nodes On Wed, Dec 28, 2016 at 6:11 AM, Koniszewski, Pawel <pawel.koniszew...@intel.com<mailto:pawel.koniszew...@intel.com>> wrote: Hello everyone, We made a research to see how live migration performance varies between different configurations, especially we aimed to test tunneled vs non-tunneled live migrations. To test live migration we simulated a case of 0-day patching of 100 compute nodes (including reboot) with workloads that are close to the real world workloads. All the results were published [1] along with environment configuration and how we built test framework. Hope you find this useful. [1] https://01.org/openstack/blogs/pkoniszewski/2016/ossc-zero-day-patching Kind Regards, Pawel Koniszewski Thanks for the write-up. I'm curious about your RabbitMQ connection failures. Was it nova-compute failing to connect? Was it a bandwidth or heartbeat issue?
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
