Reviewed: https://review.opendev.org/759831 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0d2ca53bb86b8e4a3c44855cb5ef57f223462543 Submitter: Jens Harbott (frickler) ([email protected]) Branch: master
commit 0d2ca53bb86b8e4a3c44855cb5ef57f223462543 Author: Lee Yarwood <[email protected]> Date: Tue Oct 27 08:44:59 2020 +0000 libvirt: Only ask tpool.Proxy to autowrap vir* classes I668643c836d46a25df46d4c99a973af5e50a39db attempted to fix service wide pauses by providing a more complete list of classes to tpool.Proxy. While this excluded libvirtError it can include internal libvirt-python classes pointed to by private globals that have been introduced with the use of type checking within the module. Any attempt to wrap these internal classes will result in the failure seen in bug #1901383. As a result this change simply ignores any class found during inspection that doesn't start with the `vir` string, used by libvirt to denote public methods and classes. Closes-Bug: #1901383 Co-Authored-By: Daniel Berrange <[email protected]> Change-Id: I568b0c4fd6069b9118ff116532f14abb46cc42ab ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1901383 Title: nova-compute fails with "TypeError: Parameterized generics cannot be used with class" with python 3.6 and libvirt-python 6.8.0 Status in OpenStack Compute (nova): Fix Released Bug description: Our gate jobs running python 3.6 and libvirt-python 6.8.0 are currently failing because when we call libvirt methods, the following error is raised [1]: Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service [-] Error starting thread.: TypeError: Parameterized generics cannot be used with class or instance checks Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service Traceback (most recent call last): Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/oslo_service/service.py", line 807, in run_service Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service service.start() Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/opt/stack/nova/nova/service.py", line 159, in start Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service self.manager.init_host() Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 1409, in init_host Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service self.driver.init_host(host=self.host) Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 600, in init_host Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service self._do_quality_warnings() Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 577, in _do_quality_warnings Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service caps = self._host.get_capabilities() Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/host.py", line 710, in get_capabilities Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service xmlstr = self.get_connection().getCapabilities() Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/host.py", line 511, in get_connection Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service conn = self._get_connection() Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/host.py", line 494, in _get_connection Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service {'msg': ex}) Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Oct 23 12:12:21.454426 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service self.force_reraise() Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service six.reraise(self.type_, self.value, self.tb) Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service raise value Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/host.py", line 483, in _get_connection Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service self._wrapped_conn = self._get_new_connection() Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/host.py", line 437, in _get_new_connection Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service wrapped_conn = self._connect(self._uri, self._read_only) Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/host.py", line 288, in _connect Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service return self._libvirt_proxy.openAuth(uri, auth, flags) Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 190, in doit Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service result = proxy_call(self._autowrap, f, *args, **kwargs) Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/usr/local/lib/python3.6/dist-packages/eventlet/tpool.py", line 149, in proxy_call Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service if isinstance(rv, autowrap): Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/usr/lib/python3.6/typing.py", line 1162, in __instancecheck__ Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service return issubclass(instance.__class__, self) Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service File "/usr/lib/python3.6/typing.py", line 1148, in __subclasscheck__ Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service raise TypeError("Parameterized generics cannot be used with class " Oct 23 12:12:21.456615 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service TypeError: Parameterized generics cannot be used with class or instance checks Oct 23 12:12:21.459391 ubuntu-bionic-limestone-regionone-0020957795 nova-compute[13578]: ERROR oslo_service.service The libvirt-python package version was recently bumped in upper-constraints from version 6.6.0 to 6.8.0: https://review.opendev.org/750084 and the tempest job on that change ^ passed running python 3.8. But then our jobs running python 3.6 immediately began to fail. There appears to be some kind of incompatibility between python 3.6 and libvirt-python 6.8.0. I had a look through the libvirt-python repo to see what changed between 6.6.0 and 6.8.0 and can't tell what's the root cause. We get the TypeError raised in nova-compute when we call the libvirt openAuth method but nothing in this diff [2] looks problematic? Not sure what we can do other than revert the upper-constraints bump for libvirt-python and go back to 6.6.0. [1] https://zuul.opendev.org/t/openstack/build/20f1d309663347c28112b711b82b9c03/log/controller/logs/screen-n-cpu.txt?severity=4#861 [2] https://github.com/libvirt/libvirt-python/compare/v6.6.0...v6.8.0#diff-a3d0475c2dfecd386f36307bc70befd9d5bd69daceadc61a5589ee86413d04c7L91 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1901383/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
