Are you sure you also upgraded Neutron ? Apparently, the Neutron
metadata agent tries to call the neutron server by RPC but the server
doesn't support the RPC version from the agent.
Doesn't look to me a problem from Nova, so I'll close this bug report
but please reopen it if you find some problem on Nova.
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2054409
Title:
<class 'neutronclient.common.exceptions.InternalServerError'> (HTTP
500) after upgrade
Status in OpenStack Compute (nova):
Invalid
Bug description:
Description
===========
After upgrading from Antelope to Bobcat, I'm unable to manage instances.
Steps to reproduce
==================
Following the upgrade guide
https://docs.openstack.org/nova/2023.2/admin/upgrades.html
* Pre-existing installation, Antelope 2023.1
* Upgraded the controller node to Bobcat 2023.2
* Ran 'nova-manage api_db sync' and 'nova-manage db sync'
* Upgraded other services, 'nova-status upgrade check' reports
everything as succesful.
* 'nova service-list' does not report orphaned records
* Ran 'nova-manage db online_data_migrations'
Expected result
===============
Upgrade successful, able to use the OpenStack instance
Actual result
=============
The last step does not complete succesfully. Even after running
it multiple times, there still is a pending operation:
# nova-manage db online_data_migrations
Modules with known eventlet monkey patching issues were imported prior to
eventlet monkey patching: urllib3. This warning can usually be ignored if the
caller is only importing and not executing nova code.
Running batches of 50 until complete
1 rows matched query populate_instance_compute_id, 0 migrated
+-------------------------------------+--------------+-----------+
| Migration | Total Needed | Completed |
+-------------------------------------+--------------+-----------+
| fill_virtual_interface_list | 0 | 0 |
| migrate_empty_ratio | 0 | 0 |
| migrate_quota_classes_to_api_db | 0 | 0 |
| migrate_quota_limits_to_api_db | 0 | 0 |
| migration_migrate_to_uuid | 0 | 0 |
| populate_dev_uuids | 0 | 0 |
| populate_instance_compute_id | 1 | 0 |
| populate_missing_availability_zones | 0 | 0 |
| populate_queued_for_delete | 0 | 0 |
| populate_user_id | 0 | 0 |
| populate_uuids | 0 | 0 |
+-------------------------------------+--------------+-----------+
Listing instances on the dashboard now fails with the error:
Error: Unable to retrieve instances. Details
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/
and attach the Nova API log if possible. <class
'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) (Request-ID:
req-2fa94a10-3168-41a5-8b0f-6d7499465ff1)
Environment
===========
1. Exact version of OpenStack you are running? Bobcat 2023.2
2. Which hypervisor did you use? Compute nodes are running
qemu-kvm 8.0.0 and libvirt 9.5.0
2. Which storage type did you use? Cinder has two backends, NFS and
dcache
3. Which networking type did you use? Neutron with OpenVSwitch
Logs & Configs
==============
/var/log/nova/nova-api.log
Trying to list instances:
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi [None
req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85
c824412ae5904653a037e893827aa693 - - default default] Unexpected exception in
API method: neutronclient.common.exceptions.InternalServerError: Request
Failed: internal server error while processing your request.
Neutron server returns request_ids:
['req-8894501a-070a-49be-8e46-e9bac7f1afb0']
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Traceback (most
recent call last):
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py", line 658, in
wrapped
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return
f(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192,
in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192,
in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192,
in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi [Previous line
repeated 2 more times]
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line
143, in detail
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi servers =
self._get_servers(req, is_detail=True)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line
343, in _get_servers
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi response =
self._view_builder.detail(
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py",
line 474, in detail
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi
self._add_security_grps(request, list(servers_dict["servers"]),
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py",
line 684, in _add_security_grps
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi
security_group_api.get_instances_security_groups_bindings(
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line
541, in get_instances_security_groups_bindings
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ports =
_get_ports_from_server_list(servers, neutron)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line
490, in _get_ports_from_server_list
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi
ports.extend(neutron.list_ports(**search_opts).get('ports'))
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret =
obj(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 815, in
list_ports
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return
self.list('ports', self.ports_path, retrieve_all,
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret =
obj(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 372, in
list
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi for r in
self._pagination(collection, path, **params):
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 387, in
_pagination
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi res =
self.get(path, params=params)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret =
obj(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 356, in
get
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return
self.retry_request("GET", action, body=body,
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret =
obj(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 333, in
retry_request
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return
self.do_request(method, action, body=body,
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret =
obj(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 297, in
do_request
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi
self._handle_fault_response(status_code, replybody, resp)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret =
obj(*args, **kwargs)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 272, in
_handle_fault_response
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi
exception_handler_v20(status_code, error_body)
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File
"/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 90, in
exception_handler_v20
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi raise
client_exc(message=error_message,
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi
neutronclient.common.exceptions.InternalServerError: Request Failed: internal
server error while processing your request.
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Neutron server
returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0']
2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi
2024-02-20 08:57:14.750 66460 INFO nova.api.openstack.wsgi [None
req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85
c824412ae5904653a037e893827aa693 - - default default] HTTP exception thrown:
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and
attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.InternalServerError'>
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2054409/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
