Reviewed: https://review.opendev.org/c/openstack/nova/+/926407
Committed:
https://opendev.org/openstack/nova/commit/f8b98390dc99f6cb0101c88223eb840e0d1c7124
Submitter: "Zuul (22348)"
Branch: master
commit f8b98390dc99f6cb0101c88223eb840e0d1c7124
Author: Balazs Gibizer <g...@redhat.com>
Date: Thu Aug 15 13:06:39 2024 +0200
Fix PCI passthrough cleanup on reschedule
The resource tracker Claim object works on a copy of the instance object
got from the compute manager. But the PCI claim logic does not use the
copy but use the original instance object. However the abort claim logic
including the abort PCI claim logic worked on the copy only. Therefore the
claimed PCI devices are visible to the compute manager in the
instance.pci_decives list even after the claim is aborted.
There was another bug in the PCIDevice object where the instance object
wasn't passed to the free() function and therefore the
instance.pci_devices list wasn't updated when the device was freed.
Closes-Bug: #1860555
Change-Id: Iff343d4d78996cd17a6a584fefa7071c81311673
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1860555
Title:
PCI passthrough reschedule race condition
Status in OpenStack Compute (nova):
Fix Released
Bug description:
Steps to reproduce
------------------
Create multiple instances concurrently using a flavor with a PCI
passthrough request (--property
"pci_passthrough:alias"="<alias>:<count>"), and a scheduler hint with
some anti-affinity constraint.
Expected result
---------------
The instances are created successfully, and each have the expected
number of PCI devices attached.
Actual result
-------------
Sometimes, instances may fail during creation, or may be created with
more PCI devices than requested.
Environment
-----------
Nova 18.2.2 (rocky), CentOS 7, libvirt, deployed by kolla-ansible.
Analysis
--------
If an instance with PCI passthrough devices is rescheduled (e.g. due to
affinity violation), the instance can end up with extra PCI devices attached.
If the devices selected on the original and subsequent compute nodes have the
same address, the instance will fail to create, with the following error:
libvirtError: internal error: Device 0000:89:00.0 is already in use
However, if the devices are different, and all available on the first and
second compute nodes, the VM may end up with additional hostdevs.
On investigation, when the node is rescheduled, the instance object passed to
the conductor RPC API contains the PCI devices that should have been freed.
This is because the claim object holds a clone of the instance that is used to
perform the abort on failure [1][2], and the PCI devices removed from its
list are not
reflected in the original object. There is a secondary issue that the PCI
manager was not passing through the instance to the PCI object's free() method
in all cases [3], resulting in the PCI device not being removed from the
instance.pci_devices list.
I have two alternative fixes for this issue, but they will need a
little time to work their way out of an organisation. Essentially:
1. pass the original instance (not the clone) to the abort function in the
Claim.
2. refresh the instance from DB when rescheduling
The former is a more general solution, but I don't know the reasons
for using a clone in the first place. The second works for
reschedules, but may leave a hole for resize or migration. I haven't
reproduced the issue in those cases but it seems possible that it
would be present.
[1]
https://opendev.org/openstack/nova/src/branch/master/nova/compute/claims.py#L64
[2]
https://opendev.org/openstack/nova/src/branch/master/nova/compute/claims.py#L83
[3]
https://opendev.org/openstack/nova/src/branch/master/nova/pci/manager.py#L309
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1860555/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
