On 07/07/2017 12:30 PM, Peter Doherty wrote:
Hi,
If I'm interpreting this correctly, nova compute is calling
get_diagnostics on all instances, including ones currently in a shutdown
state. And then it throws an exception, and adds an entry into the
instance_faults table in the database.
nova-compute logs this message:
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher
Traceback (most recent call last):
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 142, in _dispatch_and_reply
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher
executor_callback))
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 186, in _dispatch
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher
executor_callback)
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 129, in _do_dispatch
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher
result = func(ctxt, **new_args)
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher
payload)
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in
__exit__
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher
return f(self, context, *args, **kw)
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in
decorated_function
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher
kwargs['instance'], e, sys.exc_info())
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in
__exit__
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in
decorated_function
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher
return function(self, context, *args, **kwargs)
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4089,
in get_diagnostics
2017-07-07 16:29:46.184 23077 ERROR oslo_messaging.rpc.dispatcher
method='get_diagnostics')
2017-07-07 16:30:10.017 23077 ERROR oslo_messaging.rpc.dispatcher
InstanceInvalidState: Instance 6ab60005-ccbf-4bc2-95ac-7daf31716754 in
power_state 4. Cannot get_diagnostics while the instance is in this state.
I don't think it should be trying to gather diags on shutdown instances,
and if it did, it shouldn't just create a never-ending stream of errors.
If anyone has any info on if this might be a bug that is fixed in the
latest release, or if I can turn off this behavior, it would be appreciated.
get_diagnostics() doesn't run automatically. Something is triggering a
call to get_diagnostics() for each instance on the box (the internal
compute manager only has a get_diagnostics(instance) call that takes one
instance at a time). Not sure what is triggering that...
I agree with you that ERRORs shouldn't be spewed into the nova-compute
logs like the above, though. That should be fixed. Would you mind
submitting a bug for that on Launchpad, Peter?
Thank you!
-jay
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
