Public bug reported:
when compute node is down, nova api can accept delete action and delete the
instance
then during compute start up, periodic task will check whether the instance is
deleted in api layer
then performance actions
saw those logs because actually the info case is already deleted so it's
a valid case and should not report this exception ,just a log info
should be fine
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] Traceback (most recent call last):
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 50, in
update_instance_cache_with_nw_info
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] ic.save(update_cells=update_cells)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 197, in
wrapper
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] ctxt, self, fn.__name__, args, kwargs)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 242, in
object_action
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] objmethod=objmethod, args=args,
kwargs=kwargs)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in
call
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] retry=self.retry)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in
_send
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] timeout=timeout, retry=retry)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
431, in send
ages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] retry=retry)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
422, in _send
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] raise result
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] InstanceInfoCacheNotFound_Remote: Info
cache for instance d6a78566-0f7d-4173-b35a-b45d2054ba71 could not be found.
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] Traceback (most recent call last):
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 447, in
_object_dispatch
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] return getattr(target, method)(*args,
**kwargs)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 213, in
wrapper
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] return fn(self, *args, **kwargs)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/objects/instance_info_cache.py", line
98, in save
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] {'network_info': nw_info_json})
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/db/api.py", line 857, in
instance_info_cache_update
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] return
IMPL.instance_info_cache_update(context, instance_uuid, values)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 216, in
wrapper
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] return f(*args, **kwargs)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2684, in
instance_info_cache_update
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] instance_uuid=instance_uuid)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] InstanceInfoCacheNotFound: Info cache for
instance d6a78566-0f7d-4173-b35a-b45d2054ba71 could not be found.
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:57.101 25191 WARNING nova.compute.manager
[req-7801f16c-8845-4825-a18e-9806d142882c - - - - -] [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] Periodic cleanup failed to delete
instance: Info cache for instance d6a78566-0f7d-4173-b35a-b45d2054ba71 could
not be found.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 447,
in _object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line
213, in wrapper
return fn(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/objects/instance_info_cache.py",
line 98, in save
{'network_info': nw_info_json})
File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 857, in
instance_info_cache_update
return IMPL.instance_info_cache_update(context, instance_uuid, values)
File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 216,
in wrapper
return f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2684,
in instance_info_cache_update
instance_uuid=instance_uuid)
InstanceInfoCacheNotFound: Info cache for instance d6a78566-0f7d-4173
-b35a-b45d2054ba71 could not be found.
** Affects: nova
Importance: Undecided
Assignee: jichenjc (jichenjc)
Status: New
** Changed in: nova
Assignee: (unassigned) => jichenjc (jichenjc)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1618822
Title:
downgrade the exception log in update_instance_cache_with_nw_info
Status in OpenStack Compute (nova):
New
Bug description:
when compute node is down, nova api can accept delete action and delete the
instance
then during compute start up, periodic task will check whether the instance
is deleted in api layer
then performance actions
saw those logs because actually the info case is already deleted so
it's a valid case and should not report this exception ,just a log
info should be fine
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] Traceback (most recent call last):
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 50, in
update_instance_cache_with_nw_info
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] ic.save(update_cells=update_cells)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 197, in
wrapper
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] ctxt, self, fn.__name__, args, kwargs)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 242, in
object_action
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] objmethod=objmethod, args=args,
kwargs=kwargs)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in
call
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] retry=self.retry)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in
_send
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] timeout=timeout, retry=retry)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
431, in send
ages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] retry=retry)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line
422, in _send
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] raise result
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] InstanceInfoCacheNotFound_Remote: Info
cache for instance d6a78566-0f7d-4173-b35a-b45d2054ba71 could not be found.
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] Traceback (most recent call last):
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 447, in
_object_dispatch
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] return getattr(target, method)(*args,
**kwargs)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 213, in
wrapper
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] return fn(self, *args, **kwargs)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/objects/instance_info_cache.py", line
98, in save
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] {'network_info': nw_info_json})
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/db/api.py", line 857, in
instance_info_cache_update
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] return
IMPL.instance_info_cache_update(context, instance_uuid, values)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 216, in
wrapper
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] return f(*args, **kwargs)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] File
"/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2684, in
instance_info_cache_update
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] instance_uuid=instance_uuid)
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] InstanceInfoCacheNotFound: Info cache for
instance d6a78566-0f7d-4173-b35a-b45d2054ba71 could not be found.
2016-08-31 10:13:56.878 25191 ERROR nova.network.base_api [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71]
2016-08-31 10:13:57.101 25191 WARNING nova.compute.manager
[req-7801f16c-8845-4825-a18e-9806d142882c - - - - -] [instance:
d6a78566-0f7d-4173-b35a-b45d2054ba71] Periodic cleanup failed to delete
instance: Info cache for instance d6a78566-0f7d-4173-b35a-b45d2054ba71 could
not be found.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line
447, in _object_dispatch
return getattr(target, method)(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line
213, in wrapper
return fn(self, *args, **kwargs)
File
"/usr/lib/python2.7/site-packages/nova/objects/instance_info_cache.py", line
98, in save
{'network_info': nw_info_json})
File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 857, in
instance_info_cache_update
return IMPL.instance_info_cache_update(context, instance_uuid, values)
File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line
216, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line
2684, in instance_info_cache_update
instance_uuid=instance_uuid)
InstanceInfoCacheNotFound: Info cache for instance d6a78566-0f7d-4173
-b35a-b45d2054ba71 could not be found.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1618822/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
