Public bug reported:
Description of problem:
This is a composable HA overcloud with tls-everywhere with 2020-05-28.2 compose.
We reboot the overcloud one node at the time, and from time to time (totally
not consistent) we see that the nova_api containers is in an unhealthy state
and returns the following in a loop:
```
[Thu Jun 04 07:00:29.332162 2020] [:error] [pid 19] [remote 172.17.1.36:180]
mod_wsgi (pid=19): Exception occurred processing WSGI script
'/var/www/cgi-bin/nova/nova-api'.
[Thu Jun 04 07:00:29.332185 2020] [:error] [pid 19] [remote 172.17.1.36:180]
Traceback (most recent call last):
[Thu Jun 04 07:00:29.332208 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/var/www/cgi-bin/nova/nova-api", line 54, in <module>
[Thu Jun 04 07:00:29.332271 2020] [:error] [pid 19] [remote 172.17.1.36:180]
application = init_application()
[Thu Jun 04 07:00:29.332286 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/wsgi.py",
line 20, in init_pplication
[Thu Jun 04 07:00:29.332317 2020] [:error] [pid 19] [remote 172.17.1.36:180]
return wsgi_app.init_application(NAME)
[Thu Jun 04 07:00:29.332327 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi_app.py", line
78, in init_applcation
[Thu Jun 04 07:00:29.332351 2020] [:error] [pid 19] [remote 172.17.1.36:180]
config.parse_args([], default_config_files=conf_files)
[Thu Jun 04 07:00:29.332367 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/nova/config.py", line 35, in parse_args
[Thu Jun 04 07:00:29.332385 2020] [:error] [pid 19] [remote 172.17.1.36:180]
log.register_options(CONF)
[Thu Jun 04 07:00:29.332401 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 250, in
register_options
[Thu Jun 04 07:00:29.332433 2020] [:error] [pid 19] [remote 172.17.1.36:180]
conf.register_cli_opts(_options.common_cli_opts)
[Thu Jun 04 07:00:29.332461 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2440, in
__inner
[Thu Jun 04 07:00:29.332490 2020] [:error] [pid 19] [remote 172.17.1.36:180]
result = f(self, *args, **kwargs)
[Thu Jun 04 07:00:29.332503 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2662, in
register_cli_opts
[Thu Jun 04 07:00:29.332523 2020] [:error] [pid 19] [remote 172.17.1.36:180]
self.register_cli_opt(opt, group, clear_cache=False)
[Thu Jun 04 07:00:29.332532 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2444, in
__inner
[Thu Jun 04 07:00:29.332550 2020] [:error] [pid 19] [remote 172.17.1.36:180]
return f(self, *args, **kwargs)
[Thu Jun 04 07:00:29.332559 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2654, in
register_cli_opt
[Thu Jun 04 07:00:29.332577 2020] [:error] [pid 19] [remote 172.17.1.36:180]
raise ArgsAlreadyParsedError("cannot register CLI option")
[Thu Jun 04 07:00:29.332603 2020] [:error] [pid 19] [remote 172.17.1.36:180]
ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option
```
It look like we should always reset CONF when starting the wsgi app [1].
Version-Release number of selected component (if applicable):
openstack-nova-api-17.0.13-7.el7ost.noarch
puppet-nova-12.5.0-8.el7ost.noarch
python-nova-17.0.13-7.el7ost.noarch
openstack-nova-common-17.0.13-7.el7ost.noarch
python2-novaclient-10.1.1-1.el7ost.noarch
[1]
https://github.com/openstack/nova/blob/stable/queens/nova/api/openstack/wsgi_app.py#L78
** Affects: nova
Importance: Undecided
Assignee: Herve Beraud (herveberaud)
Status: In Progress
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1882094
Title:
[queens][OSP13] An overcloud reboot will sometimes leave nova_api
broken
Status in OpenStack Compute (nova):
In Progress
Bug description:
Description of problem:
This is a composable HA overcloud with tls-everywhere with 2020-05-28.2
compose.
We reboot the overcloud one node at the time, and from time to time (totally
not consistent) we see that the nova_api containers is in an unhealthy state
and returns the following in a loop:
```
[Thu Jun 04 07:00:29.332162 2020] [:error] [pid 19] [remote 172.17.1.36:180]
mod_wsgi (pid=19): Exception occurred processing WSGI script
'/var/www/cgi-bin/nova/nova-api'.
[Thu Jun 04 07:00:29.332185 2020] [:error] [pid 19] [remote 172.17.1.36:180]
Traceback (most recent call last):
[Thu Jun 04 07:00:29.332208 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/var/www/cgi-bin/nova/nova-api", line 54, in <module>
[Thu Jun 04 07:00:29.332271 2020] [:error] [pid 19] [remote 172.17.1.36:180]
application = init_application()
[Thu Jun 04 07:00:29.332286 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/wsgi.py",
line 20, in init_pplication
[Thu Jun 04 07:00:29.332317 2020] [:error] [pid 19] [remote 172.17.1.36:180]
return wsgi_app.init_application(NAME)
[Thu Jun 04 07:00:29.332327 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi_app.py", line
78, in init_applcation
[Thu Jun 04 07:00:29.332351 2020] [:error] [pid 19] [remote 172.17.1.36:180]
config.parse_args([], default_config_files=conf_files)
[Thu Jun 04 07:00:29.332367 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/nova/config.py", line 35, in parse_args
[Thu Jun 04 07:00:29.332385 2020] [:error] [pid 19] [remote 172.17.1.36:180]
log.register_options(CONF)
[Thu Jun 04 07:00:29.332401 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 250, in
register_options
[Thu Jun 04 07:00:29.332433 2020] [:error] [pid 19] [remote 172.17.1.36:180]
conf.register_cli_opts(_options.common_cli_opts)
[Thu Jun 04 07:00:29.332461 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2440, in
__inner
[Thu Jun 04 07:00:29.332490 2020] [:error] [pid 19] [remote 172.17.1.36:180]
result = f(self, *args, **kwargs)
[Thu Jun 04 07:00:29.332503 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2662, in
register_cli_opts
[Thu Jun 04 07:00:29.332523 2020] [:error] [pid 19] [remote 172.17.1.36:180]
self.register_cli_opt(opt, group, clear_cache=False)
[Thu Jun 04 07:00:29.332532 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2444, in
__inner
[Thu Jun 04 07:00:29.332550 2020] [:error] [pid 19] [remote 172.17.1.36:180]
return f(self, *args, **kwargs)
[Thu Jun 04 07:00:29.332559 2020] [:error] [pid 19] [remote 172.17.1.36:180]
File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2654, in
register_cli_opt
[Thu Jun 04 07:00:29.332577 2020] [:error] [pid 19] [remote 172.17.1.36:180]
raise ArgsAlreadyParsedError("cannot register CLI option")
[Thu Jun 04 07:00:29.332603 2020] [:error] [pid 19] [remote 172.17.1.36:180]
ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option
```
It look like we should always reset CONF when starting the wsgi app
[1].
Version-Release number of selected component (if applicable):
openstack-nova-api-17.0.13-7.el7ost.noarch
puppet-nova-12.5.0-8.el7ost.noarch
python-nova-17.0.13-7.el7ost.noarch
openstack-nova-common-17.0.13-7.el7ost.noarch
python2-novaclient-10.1.1-1.el7ost.noarch
[1]
https://github.com/openstack/nova/blob/stable/queens/nova/api/openstack/wsgi_app.py#L78
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1882094/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
