Hello, SQA team found this bug during our testing.
>From running the tempest suite, we get the following error:
=================================================================================================================
tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_reassign_port_between_servers
fail 110.351
3b55a32e-4841-48b3-a3aa-050d261129c1
Traceback (most recent call last):
File
"/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-69ebebc9-78df-4ad7-9740-23b4f54ebb96/repo/tempest/api/compute/servers/test_attach_interfaces.py",
line 337, in test_reassign_port_between_servers
iface = self.interfaces_client.create_interface(
File
"/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-69ebebc9-78df-4ad7-9740-23b4f54ebb96/repo/tempest/lib/services/compute/interfaces_client.py",
line 47, in create_interface
resp, body = self.post('servers/%s/os-interface' % server_id,
File
"/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-69ebebc9-78df-4ad7-9740-23b4f54ebb96/repo/tempest/lib/common/rest_client.py",
line 300, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File
"/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-69ebebc9-78df-4ad7-9740-23b4f54ebb96/repo/tempest/lib/services/compute/base_compute_client.py",
line 47, in request
resp, resp_body = super(BaseComputeClient, self).request(
File
"/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-69ebebc9-78df-4ad7-9740-23b4f54ebb96/repo/tempest/lib/common/rest_client.py",
line 742, in request
self._error_checker(resp, resp_body)
File
"/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-69ebebc9-78df-4ad7-9740-23b4f54ebb96/repo/tempest/lib/common/rest_client.py",
line 922, in _error_checker
raise exceptions.ServerFault(resp_body, resp=resp,
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at
http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception_Remote.PortBindingFailed_Remote'>
=====================================================================================================================
Looking into the Nova API logs, we then get:
=====================================================================================================================
2023-05-30 22:27:37.882 395255 ERROR nova
2023-05-30 22:27:49.883 404005 INFO os_vif [-] Loaded VIF plugins:
linux_bridge, noop, ovs
2023-05-30 22:28:00.147 404005 WARNING nova.conductor.api
[req-01768358-b6bc-4503-82e6-2b74b5a8e7be - - - - -] Timed out waiting for
nova-conductor. Is it running? Or did this service start before nova-
conductor? Reattempting establishment of nova-conductor connection...:
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to
message ID 4f27a62457e74f2783f79b86415da0f7
=====================================================================================================================
And taking a look at the nova-conductor logs, it shows that around that same
time it was failing to initialize the placement client:
=====================================================================================================================
2023-05-30 22:28:32.618 237056 ERROR nova.scheduler.client.report [-] Failed to
initialize placement client (is keystone available?):
openstack.exceptions.NotSupported: The placement service for keystone
-internal.silo5.lab1.solutionsqa:RegionOne exists but does not have any
supported versions.
2023-05-30 22:28:32.618 237056 ERROR nova.conductor.manager [-] Fatal error
initializing placement client: The placement service for
keystone-internal.silo5.lab1.solutionsqa:RegionOne exists but does not
have any supported versions.: openstack.exceptions.NotSupported: The placement
service for keystone-internal.silo5.lab1.solutionsqa:RegionOne exists but does
not have any supported versions.
2023-05-30 22:28:32.620 237056 CRITICAL nova [-] Unhandled error:
openstack.exceptions.NotSupported: The placement service for
keystone-internal.silo5.lab1.solutionsqa:RegionOne exists but does not have any
supported versions.
2023-05-30 22:28:32.620 237056 ERROR nova Traceback (most recent call last):
2023-05-30 22:28:32.620 237056 ERROR nova File "/usr/bin/nova-conductor",
line 10, in <module>
2023-05-30 22:28:32.620 237056 ERROR nova sys.exit(main())
2023-05-30 22:28:32.620 237056 ERROR nova File
"/usr/lib/python3/dist-packages/nova/cmd/conductor.py", line 45, in main
2023-05-30 22:28:32.620 237056 ERROR nova server =
service.Service.create(binary='nova-conductor',
2023-05-30 22:28:32.620 237056 ERROR nova File
"/usr/lib/python3/dist-packages/nova/service.py", line 252, in create
2023-05-30 22:28:32.620 237056 ERROR nova service_obj = cls(host, binary,
topic, manager,
2023-05-30 22:28:32.620 237056 ERROR nova File
"/usr/lib/python3/dist-packages/nova/service.py", line 116, in __init__
2023-05-30 22:28:32.620 237056 ERROR nova self.manager =
manager_class(host=self.host, *args, **kwargs)
2023-05-30 22:28:32.620 237056 ERROR nova File
"/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 122, in
__init__
2023-05-30 22:28:32.620 237056 ERROR nova self.compute_task_mgr =
ComputeTaskManager()
2023-05-30 22:28:32.620 237056 ERROR nova File
"/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 253, in
__init__
2023-05-30 22:28:32.620 237056 ERROR nova self.report_client
2023-05-30 22:28:32.620 237056 ERROR nova File
"/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 281, in
report_client
2023-05-30 22:28:32.620 237056 ERROR nova return
report.report_client_singleton()
2023-05-30 22:28:32.620 237056 ERROR nova File
"/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 91, in
report_client_singleton
2023-05-30 22:28:32.620 237056 ERROR nova PLACEMENTCLIENT =
SchedulerReportClient()
2023-05-30 22:28:32.620 237056 ERROR nova File
"/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 234, in
__init__
2023-05-30 22:28:32.620 237056 ERROR nova self._client =
self._create_client()
2023-05-30 22:28:32.620 237056 ERROR nova File
"/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 277, in
_create_client
2023-05-30 22:28:32.620 237056 ERROR nova client = self._adapter or
utils.get_sdk_adapter('placement')
2023-05-30 22:28:32.620 237056 ERROR nova File
"/usr/lib/python3/dist-packages/nova/utils.py", line 985, in get_sdk_adapter
2023-05-30 22:28:32.620 237056 ERROR nova return getattr(conn, service_type)
2023-05-30 22:28:32.620 237056 ERROR nova File
"/usr/lib/python3/dist-packages/openstack/service_description.py", line 87, in
__get__
2023-05-30 22:28:32.620 237056 ERROR nova proxy = self._make_proxy(instance)
2023-05-30 22:28:32.620 237056 ERROR nova File
"/usr/lib/python3/dist-packages/openstack/service_description.py", line 266, in
_make_proxy
2023-05-30 22:28:32.620 237056 ERROR nova raise exceptions.NotSupported(
2023-05-30 22:28:32.620 237056 ERROR nova openstack.exceptions.NotSupported:
The placement service for keystone-internal.silo5.lab1.solutionsqa:RegionOne
exists but does not have any supported versions.
2023-05-30 22:28:32.620 237056 ERROR nova
2023-05-30 22:28:36.286 237411 WARNING keystoneauth.discover [-] Failed to
contact the endpoint at http://placement-internal.silo5.lab1.solutionsqa:8778
for discovery. Fallback to using that endpoint as
the base url.
2023-05-30 22:28:36.304 237411 WARNING keystoneauth.discover [-] Failed to
contact the endpoint at http://placement-internal.silo5.lab1.solutionsqa:8778
for discovery. Fallback to using that endpoint as
the base url.
=====================================================================================================================
Test Run can be found here:
https://solutions.qa.canonical.com/v2/testruns/9ea628e9-d51b-4327-82c6-663988079165/
Artifacts for the test run:
https://oil-jenkins.canonical.com/artifacts/9ea628e9-d51b-4327-82c6-663988079165/index.html
Further occurrences of that bug in SQA labs:
https://solutions.qa.canonical.com/v2/bugs/1949991
** Changed in: nova
Status: Invalid => New
** Tags added: cdo-qa cdo-temptest foundations-engine
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1949991
Title:
The placement service for 192.168.24.100:regionOne exists but does not
have any supported versions.
Status in OpenStack Compute (nova):
New
Bug description:
Nova Conductor container running under featureset 010 fails with:
"The placement service for 192.168.24.100:regionOne exists but does
not have any supported versions."
The nova-conductor fails a health check slightly before that, but I'm
not certain about connection between the two events.
As of now I don't know what is the cause of the error or how it is
related to the featureset and nodeset used.
Release: master
Trace:
------
2021-10-29 16:37:54.218 9 CRITICAL nova [-] Unhandled error:
openstack.exceptions.NotSupported: The placement service for
192.168.24.100:regionOne exists but does not have any supported versions.
2021-10-29 16:37:54.218 9 ERROR nova Traceback (most recent call last):
2021-10-29 16:37:54.218 9 ERROR nova File "/usr/bin/nova-conductor", line
10, in <module>
2021-10-29 16:37:54.218 9 ERROR nova sys.exit(main())
2021-10-29 16:37:54.218 9 ERROR nova File
"/usr/lib/python3.6/site-packages/nova/cmd/conductor.py", line 46, in main
2021-10-29 16:37:54.218 9 ERROR nova topic=rpcapi.RPC_TOPIC)
2021-10-29 16:37:54.218 9 ERROR nova File
"/usr/lib/python3.6/site-packages/nova/service.py", line 256, in create
2021-10-29 16:37:54.218 9 ERROR nova
periodic_interval_max=periodic_interval_max)
2021-10-29 16:37:54.218 9 ERROR nova File
"/usr/lib/python3.6/site-packages/nova/service.py", line 116, in __init__
2021-10-29 16:37:54.218 9 ERROR nova self.manager =
manager_class(host=self.host, *args, **kwargs)
2021-10-29 16:37:54.218 9 ERROR nova File
"/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 119, in
__init__
2021-10-29 16:37:54.218 9 ERROR nova self.compute_task_mgr =
ComputeTaskManager()
2021-10-29 16:37:54.218 9 ERROR nova File
"/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 244, in
__init__
2021-10-29 16:37:54.218 9 ERROR nova self.report_client =
report.SchedulerReportClient()
2021-10-29 16:37:54.218 9 ERROR nova File
"/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 187,
in __init__
2021-10-29 16:37:54.218 9 ERROR nova self._client = self._create_client()
2021-10-29 16:37:54.218 9 ERROR nova File
"/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 230,
in _create_client
2021-10-29 16:37:54.218 9 ERROR nova client = self._adapter or
utils.get_sdk_adapter('placement')
2021-10-29 16:37:54.218 9 ERROR nova File
"/usr/lib/python3.6/site-packages/nova/utils.py", line 983, in get_sdk_adapter
2021-10-29 16:37:54.218 9 ERROR nova return getattr(conn, service_type)
2021-10-29 16:37:54.218 9 ERROR nova File
"/usr/lib/python3.6/site-packages/openstack/service_description.py", line 87,
in __get__
2021-10-29 16:37:54.218 9 ERROR nova proxy = self._make_proxy(instance)
2021-10-29 16:37:54.218 9 ERROR nova File
"/usr/lib/python3.6/site-packages/openstack/service_description.py", line 271,
in _make_proxy
2021-10-29 16:37:54.218 9 ERROR nova region_name=region_name))
2021-10-29 16:37:54.218 9 ERROR nova openstack.exceptions.NotSupported: The
placement service for 192.168.24.100:regionOne exists but does not have any
supported versions.
2021-10-29 16:37:54.218 9 ERROR nova
Logs:
-----
https://logserver.rdoproject.org/99/36499/6/check/periodic-tripleo-ci-centos-8-containers-multinode-compute-master-validation/91262ed/logs/subnode-1/var/log/containers/nova/nova-conductor.log.txt.gz
https://logserver.rdoproject.org/99/36499/6/check/periodic-tripleo-ci-
centos-8-containers-multinode-compute-master-
validation/91262ed/logs/subnode-1/var/log/extra/failed_services.txt.gz
Full logs for the job:
https://logserver.rdoproject.org/99/36499/6/check/periodic-tripleo-ci-
centos-8-containers-multinode-compute-master-validation/91262ed/
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1949991/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
