Public bug reported:
We have a deployment of OpenStack in our cluster with a provider network
configured. We launched an Instance a while ago although since last week
we are not able to retrieve instance data from Horizon or the OpenStack
API.
Going through the logs on both Controller and Compute node, we collected
the following errors:
CONTROLLER:
nova-api.log:
- 2024-09-02 10:52:59.851 54640 INFO nova.api.openstack.requestlog [...]
10.0.0.1 "GET /v2.1/servers/detail
limit=21&project_id=332f0702092741ff8e12ee4658a33c09&sort_dir=desc&sort_dir=desc&sort_dir=desc&sort_key=created_at&sort_k>
- [pid: 54640|app: 0|req: 1/3] 10.0.0.1 () {34 vars in 988 bytes} [Mon
Sep 2 10:52:49 2024] GET
/v2.1/servers/detail/limit=21&project_id=332f0702092741ff8e12ee4658a33c09&sort_dir=desc&sort_dir=desc&sort_dir=desc&sort_key=created_at&sort_key=display_name&sort_key=uuid
=> generated 104 bytes in 10551 msecs (HTTP/1.1 400) 7 headers in 343
bytes (1 switches on core 0)
- 2024-09-02 10:52:59.842 54640 ERROR nova.network.neutron [None
req-2c374342-0aef-484f-8878-4bf57b34d7cb
7d929138651a4a5aa75517fecdf4b49c 332f0702092741ff8e12ee4658a33c09 - -
default default] Neutron client was not able to generate a valid admin
token, please verify Neutron admin credential located in nova.conf:
neutronclient.common.exceptions. Unauthorized: 401-{'error': {'code':
401, 'title': 'Unauthorized', 'message': 'The request you have made
requires authentication.'}}
- 2024-09-02 10:49:05.064 54636 WARNING oslo_config.cfg [-] Deprecated:
Option "auth_strategy" from group "api" is deprecated for removal
(2024-09-02 10:49:05.346 54636 WARNING oslo_config.cfg [-] Deprecated:
Option "api_servers" from group "glance" is deprecated for removal (
apache2/error.log:
- [Mon Sep 02 10:52:59.861119 2024] [wsgi:error] [pid 54433:tid
274466857248] [client 10.100.4.35:55124] WARNING horizon.exceptions
Recoverable error: Networking client is experiencing an unauthorized
exception. (HTTP 400) (Request-ID:
req-2c374342-0aef-484f-8878-4bf57b34d7cb), referer:
https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/
- [Mon Sep 02 14:00:48.396856 2024] [wsgi:error] [pid 54434:tid 274315788576]
[client 10.100.4.94:52630] DEBUG novaclient.v2.client RESP BODY:
{"computeFault": {"code": 500, "message": "Unexpected API Error.
Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API
log if possible.\\n<class
'nova.exception.NeutronAdminCredentialConfigurationInvalid'>"}}, referer:
https://licheenostrum.int.bsc.es/horizon/project/
- [Mon Sep 02 10:53:02.640401 2024] [wsgi:error] [pid 54433:tid
274466857248] [client 10.100.4.35:55124] /usr/lib/python3/dist-
packages/oslo_policy/policy.py:1588: DeprecationWarning:
identity:create_identity_provider deprecated without deprecated_reason
or deprecated_since. This will be an error in a future release, referer:
https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/
- [Mon Sep 02 10:53:11.440085 2024] [wsgi:error] [pid 54433:tid
274466857248] [client 10.100.4.35:55124] /usr/lib/python3/dist-
packages/oslo_policy/policy.py:1588: DeprecationWarning: update_trunk
deprecated without deprecated_reason or deprecated_since. This will be
an error in a future release, referer:
https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/
- [Mon Sep 02 10:53:11.401337 2024] [wsgi:error] [pid 54433:tid
274466857248] [client 10.100.4.35:55124] /usr/lib/python3/dist-
packages/oslo_policy/policy.py:1588: DeprecationWarning:
delete_security_group deprecated without deprecated_reason or
deprecated_since. This will be an error in a future release, referer:
https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/
neutron-api.log:
- 2024-09-02 10:46:45.400 54307 CRITICAL neutron [-] Unhandled error:
KeyError: 'formatters'
- 2024-09-02 10:46:45.513 54309 CRITICAL neutron [-] Unhandled error:
LookupError: No section 'main' (prefixed by 'app' or 'application' or
'composite' or 'composit' or 'pipeline' or 'filter-app') found in config
/etc/neutron/api-paste.ini
- [pid: 54539|app: 0|req: 53/277] 10.0.0.14 () {34 vars in 1038 bytes} [Mon Sep
2 13:06:29 2024] GET
/v2.0/ports?device_id=0ae813eb-8d8d-4eac-aada-efa6e8d89597&fields=
binding%3Ahost_id&fields=binding%3Avif_type => generated 114 bytes in 1976
msecs (HTTP/1.1 401) 4 headers in 204 bytes (1 switches on core 5)
- 2024-09-02 10:49:24.428 54540 WARNING keystonemiddleware.auth_token
[None req-f3d90c96-d832-463a-adf3-4506574eca38 - - - - - -] Configuring
www_authenticate_uri to point to the public identity endpoint is
required; clients may not be able to authenticate against an admin
endpoint
COMPUTE:
nova-compute.log:
- 2024-09-05 13:22:58.318 3592379 ERROR nova.compute.manager
nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img
failed to execute on /var/lib/nova/instances/0ae813eb-8d8d-4eac-aada-
efa6e8d89597/disk : Unexpected error while running command.
- 2024-09-05 13:22:58.318 3592379 ERROR nova.compute.manager Stderr:
"qemu-img: Could not open '/var/lib/nova/instances/0ae813eb-8d8d-4eac-
aada-efa6e8d89597/disk': Could not open
'/var/lib/nova/instances/0ae813eb-8d8d-4eac-aada-efa6e8d89597/disk':
Permission denied\n"
- 2024-09-05 13:24:27.741 3592379 ERROR nova.network.neutron [None
req-6b2ff8da-eb8f-4bfb-bf86-6617588c243a - - - - - -] Neutron client was
not able to generate a valid admin token, please verify Neutron admin
credential located in nova.conf:
neutronclient.common.exceptions.Unauthorized: 401-{'error': {'code':
401, 'title': 'Unauthorized', 'message': 'The request you have made
requires authentication.'}}
I tried:
- Making sure my configuration files (nova, neutron, keystone,
openvswitch, ml2_plugin,...) match OpenStack's documentation and fit our
specific setup.
- Different configuration drivers, types, credentials and firewall drivers
(openvswitch, iptables_hybrid)
- Checking file and directories permissions (in /var/lib/nova/instances) to
make sure qemu is able to access vms.
- Checked that endpoints match our configuration.
- Looked for duplicates or inconsistencies in MariaDB tables.
** Affects: nova
Importance: Undecided
Status: New
** Tags: neutron nova-network
** Tags added: neutron nova-network
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2079802
Title:
Unable to retrieve instance "..."
Status in OpenStack Compute (nova):
New
Bug description:
We have a deployment of OpenStack in our cluster with a provider
network configured. We launched an Instance a while ago although since
last week we are not able to retrieve instance data from Horizon or
the OpenStack API.
Going through the logs on both Controller and Compute node, we
collected the following errors:
CONTROLLER:
nova-api.log:
- 2024-09-02 10:52:59.851 54640 INFO nova.api.openstack.requestlog
[...] 10.0.0.1 "GET /v2.1/servers/detail
limit=21&project_id=332f0702092741ff8e12ee4658a33c09&sort_dir=desc&sort_dir=desc&sort_dir=desc&sort_key=created_at&sort_k>
- [pid: 54640|app: 0|req: 1/3] 10.0.0.1 () {34 vars in 988 bytes} [Mon
Sep 2 10:52:49 2024] GET
/v2.1/servers/detail/limit=21&project_id=332f0702092741ff8e12ee4658a33c09&sort_dir=desc&sort_dir=desc&sort_dir=desc&sort_key=created_at&sort_key=display_name&sort_key=uuid
=> generated 104 bytes in 10551 msecs (HTTP/1.1 400) 7 headers in 343
bytes (1 switches on core 0)
- 2024-09-02 10:52:59.842 54640 ERROR nova.network.neutron [None
req-2c374342-0aef-484f-8878-4bf57b34d7cb
7d929138651a4a5aa75517fecdf4b49c 332f0702092741ff8e12ee4658a33c09 - -
default default] Neutron client was not able to generate a valid admin
token, please verify Neutron admin credential located in nova.conf:
neutronclient.common.exceptions. Unauthorized: 401-{'error': {'code':
401, 'title': 'Unauthorized', 'message': 'The request you have made
requires authentication.'}}
- 2024-09-02 10:49:05.064 54636 WARNING oslo_config.cfg [-]
Deprecated: Option "auth_strategy" from group "api" is deprecated for
removal (2024-09-02 10:49:05.346 54636 WARNING oslo_config.cfg [-]
Deprecated: Option "api_servers" from group "glance" is deprecated for
removal (
apache2/error.log:
- [Mon Sep 02 10:52:59.861119 2024] [wsgi:error] [pid 54433:tid
274466857248] [client 10.100.4.35:55124] WARNING horizon.exceptions
Recoverable error: Networking client is experiencing an unauthorized
exception. (HTTP 400) (Request-ID:
req-2c374342-0aef-484f-8878-4bf57b34d7cb), referer:
https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/
- [Mon Sep 02 14:00:48.396856 2024] [wsgi:error] [pid 54434:tid 274315788576]
[client 10.100.4.94:52630] DEBUG novaclient.v2.client RESP BODY:
{"computeFault": {"code": 500, "message": "Unexpected API Error.
Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API
log if possible.\\n<class
'nova.exception.NeutronAdminCredentialConfigurationInvalid'>"}}, referer:
https://licheenostrum.int.bsc.es/horizon/project/
- [Mon Sep 02 10:53:02.640401 2024] [wsgi:error] [pid 54433:tid
274466857248] [client 10.100.4.35:55124] /usr/lib/python3/dist-
packages/oslo_policy/policy.py:1588: DeprecationWarning:
identity:create_identity_provider deprecated without deprecated_reason
or deprecated_since. This will be an error in a future release,
referer:
https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/
- [Mon Sep 02 10:53:11.440085 2024] [wsgi:error] [pid 54433:tid
274466857248] [client 10.100.4.35:55124] /usr/lib/python3/dist-
packages/oslo_policy/policy.py:1588: DeprecationWarning: update_trunk
deprecated without deprecated_reason or deprecated_since. This will be
an error in a future release, referer:
https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/
- [Mon Sep 02 10:53:11.401337 2024] [wsgi:error] [pid 54433:tid
274466857248] [client 10.100.4.35:55124] /usr/lib/python3/dist-
packages/oslo_policy/policy.py:1588: DeprecationWarning:
delete_security_group deprecated without deprecated_reason or
deprecated_since. This will be an error in a future release, referer:
https://licheenostrum.int.bsc.es/horizon/auth/login/?next=/horizon/project/instances/
neutron-api.log:
- 2024-09-02 10:46:45.400 54307 CRITICAL neutron [-] Unhandled error:
KeyError: 'formatters'
- 2024-09-02 10:46:45.513 54309 CRITICAL neutron [-] Unhandled error:
LookupError: No section 'main' (prefixed by 'app' or 'application' or
'composite' or 'composit' or 'pipeline' or 'filter-app') found in
config /etc/neutron/api-paste.ini
- [pid: 54539|app: 0|req: 53/277] 10.0.0.14 () {34 vars in 1038 bytes} [Mon
Sep 2 13:06:29 2024] GET
/v2.0/ports?device_id=0ae813eb-8d8d-4eac-aada-efa6e8d89597&fields=
binding%3Ahost_id&fields=binding%3Avif_type => generated 114 bytes in 1976
msecs (HTTP/1.1 401) 4 headers in 204 bytes (1 switches on core 5)
- 2024-09-02 10:49:24.428 54540 WARNING keystonemiddleware.auth_token
[None req-f3d90c96-d832-463a-adf3-4506574eca38 - - - - - -]
Configuring www_authenticate_uri to point to the public identity
endpoint is required; clients may not be able to authenticate against
an admin endpoint
COMPUTE:
nova-compute.log:
- 2024-09-05 13:22:58.318 3592379 ERROR nova.compute.manager
nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img
failed to execute on /var/lib/nova/instances/0ae813eb-8d8d-4eac-aada-
efa6e8d89597/disk : Unexpected error while running command.
- 2024-09-05 13:22:58.318 3592379 ERROR nova.compute.manager Stderr:
"qemu-img: Could not open '/var/lib/nova/instances/0ae813eb-8d8d-4eac-
aada-efa6e8d89597/disk': Could not open
'/var/lib/nova/instances/0ae813eb-8d8d-4eac-aada-efa6e8d89597/disk':
Permission denied\n"
- 2024-09-05 13:24:27.741 3592379 ERROR nova.network.neutron [None
req-6b2ff8da-eb8f-4bfb-bf86-6617588c243a - - - - - -] Neutron client
was not able to generate a valid admin token, please verify Neutron
admin credential located in nova.conf:
neutronclient.common.exceptions.Unauthorized: 401-{'error': {'code':
401, 'title': 'Unauthorized', 'message': 'The request you have made
requires authentication.'}}
I tried:
- Making sure my configuration files (nova, neutron, keystone,
openvswitch, ml2_plugin,...) match OpenStack's documentation and fit
our specific setup.
- Different configuration drivers, types, credentials and firewall drivers
(openvswitch, iptables_hybrid)
- Checking file and directories permissions (in /var/lib/nova/instances) to
make sure qemu is able to access vms.
- Checked that endpoints match our configuration.
- Looked for duplicates or inconsistencies in MariaDB tables.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2079802/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
