Reviewed: https://review.opendev.org/c/openstack/nova/+/850003 Committed: https://opendev.org/openstack/nova/commit/e43bf900dc8ca66578603bed333c56b215b1876e Submitter: "Zuul (22348)" Branch: master
commit e43bf900dc8ca66578603bed333c56b215b1876e Author: Balazs Gibizer <[email protected]> Date: Fri Jul 15 13:48:46 2022 +0200 Gracefully ERROR in _init_instance if vnic_type changed If the vnic_type of a bound port changes from "direct" to "macvtap" and then the compute service is restarted then during _init_instance nova tries to plug the vif of the changed port. However as it now has macvtap vnic_type nova tries to look up the netdev of the parent VF. Still that VF is consumed by the instance so there is no such netdev on the host OS. This error killed the compute service at startup due to unhandled exception. This patch adds the exception handler, logs an ERROR and continue initializing other instances on the host. Also this patch adds a detailed ERROR log when nova detects that the vnic_type changed during _heal_instance_info_cache periodic. Closes-Bug: #1981813 Change-Id: I1719f8eda04e8d15a3b01f0612977164c4e55e85 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1981813 Title: Compute service fails to restart if the vnic_type of a bound port changed from direct to macvtap (CVE-2022-37394) Status in OpenStack Compute (nova): Fix Released Status in OpenStack Security Advisory: In Progress Bug description: We have a downstream bug report with the following reproduction steps: 1) create a neutron port with vnic_type "direct" 2) create an instance with that port 3) after the instance is created successfully change the vnic_type of the bound port from "direct" to "macvtap". This is accepted by Neutron 4) wait until the nova instance info caches is healed by the periodic task in nova-compute 5) restart the nova-compute service. Actual behavior --------------- The nova-compute service fails to start with PciDeviceNotFoundById exception pointing to the PCI address of the VF the port is bound to on the host. Expected behavior ----------------- The nova-compute service should start up successfully. Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service Traceback (most recent call last): Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/pci/utils.py", line 167, in get_ifname_by_pci_address Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service dev_info = os.listdir(dev_path) Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service FileNotFoundError: [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:19:0a.7/net' Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service During handling of the above exception, another exception occurred: Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service Traceback (most recent call last): Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/usr/local/lib/python3.10/site-packages/oslo_service/service.py", line 806, in run_service Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service service.start() Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/service.py", line 159, in start Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service self.manager.init_host() Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 1536, in init_host Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service self._init_instance(context, instance) Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/compute/manager.py", line 1230, in _init_instance Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service self.driver.plug_vifs(instance, net_info) Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1386, in plug_vifs Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service self.vif_driver.plug(instance, vif) Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 730, in plug Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service self.plug_hw_veb(instance, vif) Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 628, in plug_hw_veb Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service set_vf_interface_vlan( Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 99, in set_vf_interface_vlan Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service vf_ifname = pci_utils.get_ifname_by_pci_address(pci_addr) Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service File "/opt/stack/nova/nova/pci/utils.py", line 170, in get_ifname_by_pci_address Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service raise exception.PciDeviceNotFoundById(id=pci_addr) Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service nova.exception.PciDeviceNotFoundById: PCI device 0000:19:0a.7 not found Jul 15 06:39:14 dell-r640-020 nova-compute[278453]: ERROR oslo_service.service To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1981813/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
