** Description changed:
[Impact]
- * Libvirt does an overzealous check on concurrent hash usage which breaks
- some automation like for example Terraform. Upstream moved the need to
- lock up the stack where applicable and dropped the checks as they were
- superfluous.
+ * Libvirt does an overzealous check on concurrent hash usage which breaks
+ some automation like for example Terraform. Upstream moved the need to
+ lock up the stack where applicable and dropped the checks as they were
+ superfluous.
* Backport the upstream change dropping the extra checks on hash usage
[Test Case]
- * 1.Start a guest:
- $ virsh start test1
+ * 1.Start (at least) one guest:
+ $ virsh start test1
- * 2.Do 'virsh list' in a loop:
- $ for i in {1..1000}; do virsh list; done
+ * 2.Do 'virsh list' in a loop:
+ $ for i in {1..1000}; do virsh list; done
- * 3.Open another terminal, do 'virsh domstats' in a loop:
- $ for i in {1..1000}; do virsh domstats; done
+ * 3.Open another terminal, do 'virsh domstats' in a loop:
+ $ for i in {1..1000}; do virsh domstats; done
- * 4.Check the libvirtd.log:
- $ cat /var/log/libvirt/libvirtd.log | grep -i "Hash operation not allowed
during"
- 2018-09-04 06:57:00.761+0000: 28687: error : virHashForEach:597 : Hash
operation not allowed during iteration
+ # It is a race after all, so more guests and more of the loops above
+ concurrently help to raise the chance. With 4 guests and 4 concurrent
+ of each loops above I hit it 4 times of which 3 where almost at the
+ same moment.
+
+ * 4.Check the libvirtd.log:
+ $ cat /var/log/libvirt/libvirtd.log | grep -i "Hash operation not allowed
during"
+ 2018-09-04 06:57:00.761+0000: 28687: error : virHashForEach:597 : Hash
operation not allowed during iteration
From: https://bugzilla.redhat.com/show_bug.cgi?id=1581364#c7
With the aforementioned patch, this error is not produced any more.
[Regression Potential]
- * If the assumption that all remaining callers are safe isn't correct
- there could be code using invalid hashes accessing guest information.
- This needs thorough tests and regression checks as well as an extra
- review if it can be applied to the libvirt 4.0 we have in Bionic.
+ * If the assumption that all remaining callers are safe isn't correct
+ there could be code using invalid hashes accessing guest information.
+ This needs thorough tests and regression checks as well as an extra
+ review if it can be applied to the libvirt 4.0 we have in Bionic.
[Other Info]
* n/a
----
In the SUSE Manager/Uyuni[1] project CIs we make use of Terraform[2], a
tool to automate infrastructure deployments, together with terraform-
provider-libvirt[3], a Terraform provider (plugin) that allow
interaction with libvirt.
By default, this setup will issue several libvirt API calls concurrently
in an hard-to-predict order, as demonstrated by logs that I am
attaching.
In the "bad log" example, the following line appears:
error : virHashSearch:727 : Hash operation not allowed during iteration
According to analysis of a similar problem by Red Hat in an OpenStack
scenario[4], this is has been fixed in upstream libvirt via commit
4d7384eb9ddef2008cb0cc165eb808f74bc83d6b [5].
I tested the patch and it applies cleanly to the 4.0.0 package shipping
with Bionic and that successfully resolves this issue.
Please evaluate including this patch.
Thanks in advance
[1] http://uyuni-project.org/
[2] https://www.terraform.io/
[3] https://github.com/dmacvicar/terraform-provider-libvirt
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1576464#c3
[5]
https://github.com/libvirt/libvirt/commit/4d7384eb9ddef2008cb0cc165eb808f74bc83d6b.patch
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libvirt-daemon 4.0.0-1ubuntu8.3
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
Uname: Linux 4.15.0-33-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Wed Aug 29 16:05:10 2018
InstallationDate: Installed on 2014-06-12 (1539 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
SourcePackage: libvirt
UpgradeStatus: Upgraded to bionic on 2018-05-02 (119 days ago)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1789659
Title:
libvirt-daemon error "virHashSearch:727 : Hash operation not allowed
during iteration"
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1789659/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
