[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]
** Changed in: nova
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1851917
Title:
Nova Schedular and Nova Conductor restart on repeat
Status in OpenStack Compute (nova):
Expired
Bug description:
--------------------------Enviornment------------------------------------------------------------
Controller - IP A 10.1.1.100
Compute - IP A 10.1.1.110
Release - Openstack Stein
--------------------------------Error
-----------------------------------------------------------
root@overcloud-controller etc]# systemctl status openstack-nova-api.service
openstack-nova-scheduler.service openstack-nova-conductor.service
openstack-nova-novncproxy.service
● openstack-nova-api.service - OpenStack Nova API Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service;
enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-11-09 15:44:21 +07; 58min ago
Main PID: 1213 (nova-api)
CGroup: /system.slice/openstack-nova-api.service
├─ 1213 /usr/bin/python2 /usr/bin/nova-api
├─17185 /usr/bin/python2 /usr/bin/nova-api
└─17186 /usr/bin/python2 /usr/bin/nova-api
Nov 09 15:42:51 overcloud-controller systemd[1]: Starting OpenStack Nova API
Server...
Nov 09 15:44:11 overcloud-controller nova-api[1213]:
/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22:
PkgResourcesDepreca...ately.
Nov 09 15:44:11 overcloud-controller nova-api[1213]: return
pkg_resources.EntryPoint.parse("x=" + s).load(False)
Nov 09 15:44:21 overcloud-controller systemd[1]: Started OpenStack Nova API
Server.
● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-scheduler.service;
enabled; vendor preset: disabled)
Active: activating (start) since Sat 2019-11-09 16:42:25 +07; 15s ago
Main PID: 17141 (nova-scheduler)
CGroup: /system.slice/openstack-nova-scheduler.service
└─17141 /usr/bin/python2 /usr/bin/nova-scheduler
Nov 09 16:42:25 overcloud-controller systemd[1]:
openstack-nova-scheduler.service holdoff time over, scheduling restart.
Nov 09 16:42:25 overcloud-controller systemd[1]: Stopped OpenStack Nova
Scheduler Server.
Nov 09 16:42:25 overcloud-controller systemd[1]: Starting OpenStack Nova
Scheduler Server...
● openstack-nova-conductor.service - OpenStack Nova Conductor Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-conductor.service;
enabled; vendor preset: disabled)
Active: activating (start) since Sat 2019-11-09 16:42:31 +07; 9s ago
Main PID: 17156 (nova-conductor)
CGroup: /system.slice/openstack-nova-conductor.service
└─17156 /usr/bin/python2 /usr/bin/nova-conductor
Nov 09 16:42:31 overcloud-controller systemd[1]:
openstack-nova-conductor.service holdoff time over, scheduling restart.
Nov 09 16:42:31 overcloud-controller systemd[1]: Stopped OpenStack Nova
Conductor Server.
Nov 09 16:42:31 overcloud-controller systemd[1]: Starting OpenStack Nova
Conductor Server...
● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-novncproxy.service;
enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-11-09 15:42:51 +07; 59min ago
Main PID: 1210 (nova-novncproxy)
CGroup: /system.slice/openstack-nova-novncproxy.service
└─1210 /usr/bin/python2 /usr/bin/nova-novncproxy --web
/usr/share/novnc/
Nov 09 15:42:51 overcloud-controller systemd[1]: Started OpenStack Nova NoVNC
Proxy Server.
Hint: Some lines were ellipsized, use -l to show in full.
-----------------------------------Expected Result
----------------------------------------
root@overcloud-controller etc]# systemctl status openstack-nova-api.service
openstack-nova-scheduler.service openstack-nova-conductor.service
openstack-nova-novncproxy.service
● openstack-nova-api.service - OpenStack Nova API Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service;
enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-11-09 15:44:21 +07; 58min ago
Main PID: 1213 (nova-api)
CGroup: /system.slice/openstack-nova-api.service
├─ 1213 /usr/bin/python2 /usr/bin/nova-api
├─17193 /usr/bin/python2 /usr/bin/nova-api
└─17194 /usr/bin/python2 /usr/bin/nova-api
Nov 09 15:42:51 overcloud-controller systemd[1]: Starting OpenStack Nova API
Server...
Nov 09 15:44:11 overcloud-controller nova-api[1213]:
/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22:
PkgResourcesDepreca...ately.
Nov 09 15:44:11 overcloud-controller nova-api[1213]: return
pkg_resources.EntryPoint.parse("x=" + s).load(False)
Nov 09 15:44:21 overcloud-controller systemd[1]: Started OpenStack Nova API
Server.
● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-scheduler.service;
enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-11-09 16:42:42 +07; 1s ago
Main PID: 17141 (nova-scheduler)
CGroup: /system.slice/openstack-nova-scheduler.service
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: first_packet =
self.connection._read_packet()
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, ...packet
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]:
packet.check_error()
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File
"/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in ..._error
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]:
err.raise_mysql_exception(self._data)
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File
"/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise...eption
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: raise
errorclass(errno, errval)
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: ProgrammingError:
(pymysql.err.ProgrammingError) (1146, u"Table 'nova.comp...local_
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: Removing
descriptor: 5
Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: Exception
AssertionError: AssertionError('do not call blocking functions f...gnored
● openstack-nova-conductor.service - OpenStack Nova Conductor Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-conductor.service;
enabled; vendor preset: disabled)
Active: activating (start) since Sat 2019-11-09 16:42:31 +07; 11s ago
Main PID: 17156 (nova-conductor)
CGroup: /system.slice/openstack-nova-conductor.service
└─17156 /usr/bin/python2 /usr/bin/nova-conductor
Nov 09 16:42:31 overcloud-controller systemd[1]:
openstack-nova-conductor.service holdoff time over, scheduling restart.
Nov 09 16:42:31 overcloud-controller systemd[1]: Stopped OpenStack Nova
Conductor Server.
Nov 09 16:42:31 overcloud-controller systemd[1]: Starting OpenStack Nova
Conductor Server...
● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-novncproxy.service;
enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-11-09 15:42:51 +07; 59min ago
Main PID: 1210 (nova-novncproxy)
CGroup: /system.slice/openstack-nova-novncproxy.service
└─1210 /usr/bin/python2 /usr/bin/nova-novncproxy --web
/usr/share/novnc/
Nov 09 15:42:51 overcloud-controller systemd[1]: Started OpenStack Nova NoVNC
Proxy Server.
Hint: Some lines were ellipsized, use -l to show in full.
-----------------------------------NOVA SCHEDULAR
LOGS-------------------------------------
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service Traceback (most
recent call last):
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/oslo_service/service.py", line 796, in
run_service
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service service.start()
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/service.py", line 166, in start
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ctxt, self.host,
self.binary)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in
wrapper
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result = fn(cls,
context, *args, **kwargs)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/objects/service.py", line 334, in
get_by_host_and_binary
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service host, binary)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/db/api.py", line 127, in
service_get_by_host_and_binary
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return
IMPL.service_get_by_host_and_binary(context, host, binary)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 251, in
wrapped
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return
f(context, *args, **kwargs)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 508, in
service_get_by_host_and_binary
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service
filter_by(binary=binary).\
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2979, in
first
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ret =
list(self[0:1])
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2771, in
__getitem__
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return list(res)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3081, in
__iter__
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return
self._execute_and_instances(context)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3106, in
_execute_and_instances
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result =
conn.execute(querycontext.statement, self._params)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 980, in
execute
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return
meth(self, multiparams, params)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 273, in
_execute_on_connection
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return
connection._execute_clauseelement(self, multiparams, params)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1099, in
_execute_clauseelement
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service distilled_params,
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1240, in
_execute_context
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service e, statement,
parameters, cursor, context
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in
_handle_dbapi_exception
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service
util.raise_from_cause(newraise, exc_info)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in
raise_from_cause
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service
reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1236, in
_execute_context
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service cursor,
statement, parameters, context
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 536, in
do_execute
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service
cursor.execute(statement, parameters)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 170, in execute
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result =
self._query(query)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 328, in _query
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service conn.query(q)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 516, in query
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 727, in
_read_query_result
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result.read()
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1066, in read
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service first_packet =
self.connection._read_packet()
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, in
_read_packet
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service
packet.check_error()
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service
err.raise_mysql_exception(self._data)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File
"/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in
raise_mysql_exception
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service raise
errorclass(errno, errval)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ProgrammingError:
(pymysql.err.ProgrammingError) (1146, u"Table 'nova.services' doesn't exist")
[SQL: u'SELECT services.created_at AS services_created_at, services.updated_at
AS services_updated_at, services.deleted_at AS services_deleted_at,
services.deleted AS services_deleted, services.id AS services_id, services.uuid
AS services_uuid, services.host AS services_host, services.`binary` AS
services_binary, services.topic AS services_topic, services.report_count AS
services_report_count, services.disabled AS services_disabled,
services.disabled_reason AS services_disabled_reason, services.last_seen_up AS
services_last_seen_up, services.forced_down AS services_forced_down,
services.version AS services_version \nFROM services \nWHERE services.deleted =
%(deleted_1)s AND services.host = %(host_1)s AND services.`binary` =
%(binary_1)s \n LIMIT %(param_1)s'] [parameters: {u'host_1':
'overcloud-controller', u'param_1': 1, u'deleted_1': 0, u'binary_1':
'nova-scheduler'}] (Background on this error at: http://sqlalche.me/e/f405)
2019-11-09 16:44:29.120 17464 ERROR oslo_service.service
--------------------------------------------Steps
Taken--------------------------------------
Restart Rabbit mq
Created NOVA DB Again, New privileges provided.
Restarted nova services
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1851917/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
