Public bug reported: The gate-grenade-dsvm-neutron-multinode-live-migration-nv job has been failing at about 100% since August 18:
http://graphite.openstack.org/render/?from=-2160hours&height=500&until=now&width=800&bgcolor=ffffff&fgcolor=000000&yMax=100&yMin=0&vtitle=Failure%20Rate%20in%20Percent&title=Test%20failure%20rates%20over%20last%202160%20hours%20%2812%20hour%20rolling%20average%29&drawNullAsZero=true&&target=lineWidth(color(alias(movingAverage(asPercent(transformNull(stats_counts.zuul.pipeline.check.job .gate-grenade-dsvm-neutron-multinode-live-migration- nv.FAILURE),transformNull(sum(stats_counts.zuul.pipeline.check.job.gate- grenade-dsvm-neutron-multinode-live-migration- nv.{SUCCESS,FAILURE}))),%2712hours%27),%20%27gate-grenade-dsvm-neutron- multinode-live-migration-nv%20%28check%29%27),%27ff0000%27),1) With this failure: http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron- multinode-live-migration- nv/ae8875f/logs/subnode-2/screen-n-cpu.txt.gz?level=TRACE#_Sep_26_14_28_11_637958 Sep 26 14:28:11.637958 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server [None req-b8c949a1-606a-48dc-88df-f848ac421d75 tempest-LiveMigrationRemoteConsolesV26Test-1985366765 tempest-LiveMigrationRemoteConsolesV26Test-1985366765] Exception during message handling: InvalidSharedStorage: ubuntu-xenial-2-node-rax-ord-11140716-924370 is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks. Sep 26 14:28:11.638227 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Sep 26 14:28:11.638476 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming Sep 26 14:28:11.638699 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Sep 26 14:28:11.638980 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch Sep 26 14:28:11.639207 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Sep 26 14:28:11.639413 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch Sep 26 14:28:11.639617 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Sep 26 14:28:11.639823 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/exception_wrapper.py", line 76, in wrapped Sep 26 14:28:11.639981 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server function_name, call_dict, binary) Sep 26 14:28:11.640148 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Sep 26 14:28:11.640311 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server self.force_reraise() Sep 26 14:28:11.640449 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Sep 26 14:28:11.640609 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) Sep 26 14:28:11.640863 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/exception_wrapper.py", line 67, in wrapped Sep 26 14:28:11.641005 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) Sep 26 14:28:11.641157 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/compute/utils.py", line 874, in decorated_function Sep 26 14:28:11.641301 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) Sep 26 14:28:11.641460 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/compute/manager.py", line 217, in decorated_function Sep 26 14:28:11.641601 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) Sep 26 14:28:11.641731 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Sep 26 14:28:11.641848 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server self.force_reraise() Sep 26 14:28:11.641982 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Sep 26 14:28:11.642132 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) Sep 26 14:28:11.642291 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/compute/manager.py", line 205, in decorated_function Sep 26 14:28:11.642474 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) Sep 26 14:28:11.642659 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/compute/manager.py", line 5380, in check_can_live_migrate_source Sep 26 14:28:11.642827 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server block_device_info) Sep 26 14:28:11.643055 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 6001, in check_can_live_migrate_source Sep 26 14:28:11.643637 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server raise exception.InvalidSharedStorage(reason=reason, path=source) Looking at the tests that fail and pass: http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron- multinode-live-migration- nv/ae8875f/console.html#_2017-09-26_14_28_03_894125 Anything using the block_migration=auto flag with the 2.25 microversion is successful, so all microversions before that are failing when we have a mixed compute (one pike, one queens) environment. In this failure case, it's the n-1 Pike compute: http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm-neutron- multinode-live-migration-nv/ae8875f/logs/subnode-2/screen-n-cpu.txt.gz Sep 26 14:09:42.382331 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[23418]: INFO nova.service [-] Starting compute node (version 16.0.1) So we might be sending something from the queens compute that the pike compute doesn't understand. ** Affects: nova Importance: Medium Status: Confirmed ** Tags: grenade live-migration -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1720191 Title: test_live_block_migration fails in gate-grenade-dsvm-neutron- multinode-live-migration-nv with "Shared storage live-migration requires either shared storage or boot-from-volume with no local disks." since ~8/18/2017 Status in OpenStack Compute (nova): Confirmed Bug description: The gate-grenade-dsvm-neutron-multinode-live-migration-nv job has been failing at about 100% since August 18: http://graphite.openstack.org/render/?from=-2160hours&height=500&until=now&width=800&bgcolor=ffffff&fgcolor=000000&yMax=100&yMin=0&vtitle=Failure%20Rate%20in%20Percent&title=Test%20failure%20rates%20over%20last%202160%20hours%20%2812%20hour%20rolling%20average%29&drawNullAsZero=true&&target=lineWidth(color(alias(movingAverage(asPercent(transformNull(stats_counts.zuul.pipeline.check.job .gate-grenade-dsvm-neutron-multinode-live-migration- nv.FAILURE),transformNull(sum(stats_counts.zuul.pipeline.check.job .gate-grenade-dsvm-neutron-multinode-live-migration- nv.{SUCCESS,FAILURE}))),%2712hours%27),%20%27gate-grenade-dsvm- neutron-multinode-live-migration-nv%20%28check%29%27),%27ff0000%27),1) With this failure: http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm- neutron-multinode-live-migration- nv/ae8875f/logs/subnode-2/screen-n-cpu.txt.gz?level=TRACE#_Sep_26_14_28_11_637958 Sep 26 14:28:11.637958 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server [None req-b8c949a1-606a-48dc-88df-f848ac421d75 tempest-LiveMigrationRemoteConsolesV26Test-1985366765 tempest-LiveMigrationRemoteConsolesV26Test-1985366765] Exception during message handling: InvalidSharedStorage: ubuntu-xenial-2-node-rax-ord-11140716-924370 is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks. Sep 26 14:28:11.638227 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Sep 26 14:28:11.638476 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming Sep 26 14:28:11.638699 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Sep 26 14:28:11.638980 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch Sep 26 14:28:11.639207 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Sep 26 14:28:11.639413 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch Sep 26 14:28:11.639617 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Sep 26 14:28:11.639823 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/exception_wrapper.py", line 76, in wrapped Sep 26 14:28:11.639981 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server function_name, call_dict, binary) Sep 26 14:28:11.640148 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Sep 26 14:28:11.640311 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server self.force_reraise() Sep 26 14:28:11.640449 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Sep 26 14:28:11.640609 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) Sep 26 14:28:11.640863 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/exception_wrapper.py", line 67, in wrapped Sep 26 14:28:11.641005 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) Sep 26 14:28:11.641157 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/compute/utils.py", line 874, in decorated_function Sep 26 14:28:11.641301 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) Sep 26 14:28:11.641460 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/compute/manager.py", line 217, in decorated_function Sep 26 14:28:11.641601 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) Sep 26 14:28:11.641731 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Sep 26 14:28:11.641848 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server self.force_reraise() Sep 26 14:28:11.641982 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Sep 26 14:28:11.642132 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) Sep 26 14:28:11.642291 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/compute/manager.py", line 205, in decorated_function Sep 26 14:28:11.642474 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) Sep 26 14:28:11.642659 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/compute/manager.py", line 5380, in check_can_live_migrate_source Sep 26 14:28:11.642827 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server block_device_info) Sep 26 14:28:11.643055 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 6001, in check_can_live_migrate_source Sep 26 14:28:11.643637 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[32137]: ERROR oslo_messaging.rpc.server raise exception.InvalidSharedStorage(reason=reason, path=source) Looking at the tests that fail and pass: http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm- neutron-multinode-live-migration- nv/ae8875f/console.html#_2017-09-26_14_28_03_894125 Anything using the block_migration=auto flag with the 2.25 microversion is successful, so all microversions before that are failing when we have a mixed compute (one pike, one queens) environment. In this failure case, it's the n-1 Pike compute: http://logs.openstack.org/87/463987/20/check/gate-grenade-dsvm- neutron-multinode-live-migration- nv/ae8875f/logs/subnode-2/screen-n-cpu.txt.gz Sep 26 14:09:42.382331 ubuntu-xenial-2-node-rax-ord-11140716-924370 nova-compute[23418]: INFO nova.service [-] Starting compute node (version 16.0.1) So we might be sending something from the queens compute that the pike compute doesn't understand. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1720191/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
