Extended the regression considerations by some content from the commit and
discussions I found.
It is safe IMHO, but I need to explain the SRU team why.
** Description changed:
[Impact]
* Libvirt does an overzealous check on concurrent hash usage which breaks
some automation like for example Terraform. Upstream moved the need to
lock up the stack where applicable and dropped the checks as they were
superfluous.
* Backport the upstream change dropping the extra checks on hash usage
[Test Case]
* 1.Start (at least) one guest:
$ virsh start test1
* 2.Do 'virsh list' in a loop:
$ for i in {1..1000}; do virsh list; done
* 3.Open another terminal, do 'virsh domstats' in a loop:
$ for i in {1..1000}; do virsh domstats; done
- # It is a race after all, so more guests and more of the loops above
- concurrently help to raise the chance. With 4 guests and 4 concurrent
- of each loops above I hit it 4 times of which 3 where almost at the
- same moment.
+ # It is a race after all, so more guests and more of the loops above
+ concurrently help to raise the chance. With 4 guests and 4 concurrent
+ of each loops above I hit it 4 times of which 3 where almost at the
+ same moment.
* 4.Check the libvirtd.log:
$ cat /var/log/libvirt/libvirtd.log | grep -i "Hash operation not allowed
during"
2018-09-04 06:57:00.761+0000: 28687: error : virHashForEach:597 : Hash
operation not allowed during iteration
From: https://bugzilla.redhat.com/show_bug.cgi?id=1581364#c7
With the aforementioned patch, this error is not produced any more.
[Regression Potential]
* If the assumption that all remaining callers are safe isn't correct
there could be code using invalid hashes accessing guest information.
This needs thorough tests and regression checks as well as an extra
review if it can be applied to the libvirt 4.0 we have in Bionic.
+ This was done by me and also upstream did the same.
+ Quote from the patch: "Most important uses seem to be covered. Callers
+ have now a greater responsability, notably the ability to execute some
+ operations while iterating were reliably forbidden before are now
+ accepted."
+ It is important to see that the internal locking was inherently racy
+ (boolean) so we are replaceing a broken lock-ckeck with no check as
+ upper layers were supposed to do it correctly. That is true for what is
+ in the archive but there could be out-of archive things not doing so
+ calling directly into the python api bindings for example.
+ Never the less those are today just as affected by the locking being
+ broken, so those do not "loose" a lot.
+ All major exploiters we know of are good - the rest is a regression
+ risk we are willing to take in favor of fixing all the valid use cases
+ being affected by the broken lock handling.
[Other Info]
* n/a
----
In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2], a
tool to automate infrastructure deployments, together with terraform-
provider-libvirt[3], a Terraform provider (plugin) that allow
interaction with libvirt.
By default, this setup will issue several libvirt API calls concurrently
in an hard-to-predict order, as demonstrated by logs that I am
attaching.
In the "bad log" example, the following line appears:
error : virHashSearch:727 : Hash operation not allowed during iteration
According to analysis of a similar problem by Red Hat in an OpenStack
scenario[4], this is has been fixed in upstream libvirt via commit
4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].
I tested the patch and it applies cleanly to the 4.0.0 package shipping
with Bionic and that successfully resolves this issue.
Please evaluate including this patch.
Thanks in advance
[1] http://uyuni-project.org/
[2] https://www.terraform.io/
[3] https://github.com/dmacvicar/terraform-provider-libvirt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
[5]
https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt-daemon 4.0.0-1ubuntu8.3
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
Uname: Linux 4.15.0-33-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 29 16:05:10 2018
InstallationDate: Installed on 2014-06-12 (1539 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: libvirt
UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1789659
Title:
libvirt-daemon error "virHashSearch:727 : Hash operation not allowed
during iteration"
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1789659/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
