Public bug reported:
With versioned notifications, the instance payload tries to attach a
flavor payload which it looks up from the instance. It uses the one
which is attached in instance_extras however there seems to be a
scenario where the disabled field is missing in the database, causing
all operations to fail in the notification stage.
The JSON string for the flavor in the database is attached below (note
this is a cloud with a long lifetime so it might be some weird
conversion at some point in the life time of the cloud).
The temporary workaround as suggested by Matt was to switch to
unversioned notification which did the trick.
== flavor ==
{"new": null, "old": null, "cur": {"nova_object.version": "1.1",
"nova_object.changes": ["root_gb", "name", "ephemeral_gb", "memory_mb",
"vcpus", "extra_specs", "swap", "rxtx_factor", "flavorid", "vcpu_weight",
"id"], "nova_object.name": "Flavor", "nova_object.data": {"root_gb": 80,
"name": "nb.2G", "ephemeral_gb": 0, "memory_mb": 2048, "vcpus": 4,
"extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "flavorid":
"8c6a8477-20cb-4db9-ad1d-be3bc05cdae9", "vcpu_weight": null, "id": 8},
"nova_object.namespace": "nova"}}
== flavor ==
== stack ==
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
[req-edc9fb83-63ff-4c4b-b6c6-704d331905a8 604d5fd332904975a26b6e89c60a9d51
d6ebcbe536f848b3af4403f922377f80 - default default] Exception during message
handling: ValueError: Field `disabled' cannot be None
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in
_process_incoming
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213,
in dispatch
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183,
in _do_dispatch
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in
wrapped
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
function_name, call_dict, binary)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in
wrapped
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
f(self, context, *args, **kw)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 189, in
decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server "Error: %s",
e, instance=instance)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 159, in
decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 874, in
decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 217, in
decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
kwargs['instance'], e, sys.exc_info())
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 205, in
decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4125, in
pause_instance
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
phase=fields.NotificationPhase.START)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 365, in
notify_about_instance_action
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server fault=fault)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/notifications/objects/instance.py", line
138, in __init__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
super(InstanceActionPayload, self).__init__(instance=instance)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/notifications/objects/instance.py", line
115, in __init__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server self.flavor =
flavor_payload.FlavorPayload(flavor=instance.flavor)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/notifications/objects/flavor.py", line
85, in __init__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
self.populate_schema(flavor=flavor)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/notifications/objects/base.py", line
130, in populate_schema
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server setattr(self,
key, None)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 72, in
setter
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server field_value =
field.coerce(self, name, value)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 193,
in coerce
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
self._null(obj, attr)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 171,
in _null
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server raise
ValueError(_("Field `%s' cannot be None") % attr)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server ValueError: Field
`disabled' cannot be None
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
== stack ==
** Affects: nova
Importance: High
Assignee: Matt Riedemann (mriedem)
Status: In Progress
** Affects: nova/pike
Importance: Undecided
Status: New
** Tags: notifications
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1739325
Title:
Server operations fail to complete with versioned notifications if
payload contains unset non-nullable fields
Status in OpenStack Compute (nova):
In Progress
Status in OpenStack Compute (nova) pike series:
New
Bug description:
With versioned notifications, the instance payload tries to attach a
flavor payload which it looks up from the instance. It uses the one
which is attached in instance_extras however there seems to be a
scenario where the disabled field is missing in the database, causing
all operations to fail in the notification stage.
The JSON string for the flavor in the database is attached below (note
this is a cloud with a long lifetime so it might be some weird
conversion at some point in the life time of the cloud).
The temporary workaround as suggested by Matt was to switch to
unversioned notification which did the trick.
== flavor ==
{"new": null, "old": null, "cur": {"nova_object.version": "1.1",
"nova_object.changes": ["root_gb", "name", "ephemeral_gb", "memory_mb",
"vcpus", "extra_specs", "swap", "rxtx_factor", "flavorid", "vcpu_weight",
"id"], "nova_object.name": "Flavor", "nova_object.data": {"root_gb": 80,
"name": "nb.2G", "ephemeral_gb": 0, "memory_mb": 2048, "vcpus": 4,
"extra_specs": {}, "swap": 0, "rxtx_factor": 1.0, "flavorid":
"8c6a8477-20cb-4db9-ad1d-be3bc05cdae9", "vcpu_weight": null, "id": 8},
"nova_object.namespace": "nova"}}
== flavor ==
== stack ==
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
[req-edc9fb83-63ff-4c4b-b6c6-704d331905a8 604d5fd332904975a26b6e89c60a9d51
d6ebcbe536f848b3af4403f922377f80 - default default] Exception during message
handling: ValueError: Field `disabled' cannot be None
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in
_process_incoming
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213,
in dispatch
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183,
in _do_dispatch
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in
wrapped
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
function_name, call_dict, binary)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in
wrapped
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
f(self, context, *args, **kw)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 189, in
decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server "Error:
%s", e, instance=instance)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 159, in
decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 874, in
decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 217, in
decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
kwargs['instance'], e, sys.exc_info())
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
self.force_reraise()
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 205, in
decorated_function
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4125, in
pause_instance
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
phase=fields.NotificationPhase.START)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 365, in
notify_about_instance_action
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server fault=fault)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/notifications/objects/instance.py", line
138, in __init__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
super(InstanceActionPayload, self).__init__(instance=instance)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/notifications/objects/instance.py", line
115, in __init__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server self.flavor
= flavor_payload.FlavorPayload(flavor=instance.flavor)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/notifications/objects/flavor.py", line
85, in __init__
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
self.populate_schema(flavor=flavor)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/rpc.py", line 225, in wrapped
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/nova/notifications/objects/base.py", line
130, in populate_schema
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
setattr(self, key, None)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 72, in
setter
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server field_value
= field.coerce(self, name, value)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 193,
in coerce
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server return
self._null(obj, attr)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/fields.py", line 171,
in _null
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server raise
ValueError(_("Field `%s' cannot be None") % attr)
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server ValueError:
Field `disabled' cannot be None
2017-10-23 14:49:21.117 40200 ERROR oslo_messaging.rpc.server
== stack ==
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1739325/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
