Public bug reported: After this Tempest change landed to test volume-backed resize:
https://review.openstack.org/#/c/314816/ The ceph plugin job has been intermittently failing the test: http://logs.openstack.org/00/314600/3/check/gate-tempest-dsvm-full- devstack-plugin- ceph/9dad224/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-05-11_12_14_26_874 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server [req-1eef0755-362f-4136-ab7c-695b0bb4f0bb tempest-TestServerAdvancedOps-1194878159 tempest-TestServerAdvancedOps-1912438683] Exception during handling message 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 153, in dispatch 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 122, in _do_dispatch 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception.py", line 110, in wrapped 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server payload) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception.py", line 89, in wrapped 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 232, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 210, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 198, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 3338, in confirm_resize 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server do_confirm_resize(context, instance, migration.id) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 3336, in do_confirm_resize 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server migration=migration) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 3362, in _confirm_resize 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server network_info) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7288, in confirm_migration 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self._cleanup_resize(instance, network_info) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1061, in _cleanup_resize 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server ignore_errors=True) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 903, in remove_snap 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return self.driver.remove_snap(self.rbd_name, name, ignore_errors) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/virt/libvirt/storage/rbd_utils.py", line 396, in remove_snap 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server with RBDVolumeProxy(self, str(volume), pool=pool) as vol: 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/virt/libvirt/storage/rbd_utils.py", line 65, in __init__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server driver._disconnect_from_rados(client, ioctx) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/virt/libvirt/storage/rbd_utils.py", line 61, in __init__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server read_only=read_only) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/rbd.py", line 374, in __init__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server raise make_ex(ret, 'error opening image %s at snapshot %s' % (name, snapshot)) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server ImageNotFound: error opening image 14decc0d-ac5a-4c29-95cd-a8b1bd48e5e8_disk at snapshot None 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ImageNotFound%3A%20error%20opening%20image%5C%22%20AND%20message%3A%5C%22at%20snapshot%20None%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d So far there are 29 hits in 24 hours since the Tempest change merged, and since it's Tempest it's hitting on all branches for any projects that run the ceph job, which is at least nova/cinder/glance/glance-store /os-brick/gnocchi. ** Affects: nova Importance: High Status: Confirmed ** Tags: ceph libvirt resize volumes -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1580625 Title: volume-backed resize intermittently fails with rbd imagebackend Status in OpenStack Compute (nova): Confirmed Bug description: After this Tempest change landed to test volume-backed resize: https://review.openstack.org/#/c/314816/ The ceph plugin job has been intermittently failing the test: http://logs.openstack.org/00/314600/3/check/gate-tempest-dsvm-full- devstack-plugin- ceph/9dad224/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-05-11_12_14_26_874 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server [req-1eef0755-362f-4136-ab7c-695b0bb4f0bb tempest-TestServerAdvancedOps-1194878159 tempest-TestServerAdvancedOps-1912438683] Exception during handling message 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 153, in dispatch 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 122, in _do_dispatch 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception.py", line 110, in wrapped 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server payload) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception.py", line 89, in wrapped 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 232, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 210, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 198, in decorated_function 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 3338, in confirm_resize 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server do_confirm_resize(context, instance, migration.id) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 3336, in do_confirm_resize 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server migration=migration) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 3362, in _confirm_resize 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server network_info) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7288, in confirm_migration 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self._cleanup_resize(instance, network_info) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1061, in _cleanup_resize 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server ignore_errors=True) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 903, in remove_snap 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server return self.driver.remove_snap(self.rbd_name, name, ignore_errors) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/virt/libvirt/storage/rbd_utils.py", line 396, in remove_snap 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server with RBDVolumeProxy(self, str(volume), pool=pool) as vol: 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/virt/libvirt/storage/rbd_utils.py", line 65, in __init__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server driver._disconnect_from_rados(client, ioctx) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/virt/libvirt/storage/rbd_utils.py", line 61, in __init__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server read_only=read_only) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/rbd.py", line 374, in __init__ 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server raise make_ex(ret, 'error opening image %s at snapshot %s' % (name, snapshot)) 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server ImageNotFound: error opening image 14decc0d-ac5a-4c29-95cd-a8b1bd48e5e8_disk at snapshot None 2016-05-11 12:14:26.874 17925 ERROR oslo_messaging.rpc.server http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ImageNotFound%3A%20error%20opening%20image%5C%22%20AND%20message%3A%5C%22at%20snapshot%20None%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d So far there are 29 hits in 24 hours since the Tempest change merged, and since it's Tempest it's hitting on all branches for any projects that run the ceph job, which is at least nova/cinder/glance/glance- store/os-brick/gnocchi. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1580625/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
