Hello, One of the major sets of issues currently affecting gate testing is Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us and they happen frequently [0][1][2]. These issues appear to only affect Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in #openstack-nova it is clear that Libvirt isn't interested in debugging such an old version of Libvirt (1.3.1). And while it isn't entirely clear to me which exact version would be acceptable to them the Ubuntu Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0).
I have pushed a change to devstack [3] to enable using UCA which pulls in new Libvirt and mostly seems to work. I think we should consider switching to UCA as this may fix our Libvirt problems and if it doesn't, we will be closer to a version of Libvirt that upstream should be willing to fix. This isn't the most straightfoward switch as UCA has a different repo for each OpenStack release. libvirt-python is sensitve to the underlying library changing; it is backward compatible but libvirt-python built against older libvirt won't work against new libvirt. The result is a libvirt-python wheel built on our wheel mirror does not work with UCA. On the positive side both the OpenStack puppet modules and OpenStack Ansible are using UCA with their deployment tooling so this should get us closer to what people are using in the wild. After some thought I think my preferred method of rolling this out would be to blacklist libvirt-python from our wheel mirror building entirely and force installs to happen from source so that we are base Xenial and $UCA_version independent (local testing of these builds show it only takes a few seconds). Then have specific jobs (like devstack) explicitly opt into the UCA repo appropriate for them (if any). This last bit is from feedback from OpenStack Ansible that having the base images be fairly clean is desirable, but it would also be hard to know which version of UCA is appropriate for our Xenial images (this likely differs based on the job). Now its entirely possible that newer Libvirt will be worse than current (old) Libvirt; however, being closer to upstream should make getting fixes easier. Would be great if those with a better understanding of Libvirt could chime in on this if I am completely wrong here. Finally it is worth noting that we will get newer packages of other software as well, most notably openvswitch will be version 2.6.1 instead of 2.5.0. Any other thoughts or ideas? [0] http://status.openstack.org/elastic-recheck/#1646779 [1] http://status.openstack.org/elastic-recheck/#1643911 [2] http://status.openstack.org/elastic-recheck/#1638982 [3] https://review.openstack.org/451492 Thank you, Clark __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
