Excerpts from Tony Breeds's message of 2016-02-18 08:04:00 +1100: > On Wed, Feb 17, 2016 at 12:44:11PM -0500, Doug Hellmann wrote: > > Excerpts from Henry Gessau's message of 2016-02-17 11:00:53 -0500: > > > Doug Hellmann <d...@doughellmann.com> wrote: > > > > Excerpts from Morgan Fainberg's message of 2016-02-17 07:10:34 -0800: > > > >> On Wed, Feb 17, 2016 at 5:55 AM, Sean Dague <s...@dague.net> wrote: > > > >> > > > >>> On 02/17/2016 08:42 AM, Doug Hellmann wrote: > > > >>>> Excerpts from Victor Stinner's message of 2016-02-17 14:14:18 +0100: > > > >>>>> Le 17/02/2016 13:43, Henry Gessau a écrit : > > > >>>>>> And it looks like eventlet 0.18.3 breaks neutron: > > > >>>>>> https://bugs.launchpad.net/neutron/+bug/1546506 > > > >>>>> > > > >>>>> 2 releases, 2 regressions in OpenStack. Should we cap eventlet > > > >>>>> version? > > > >>>>> The requirement bot can produce patches to update eventlet, patches > > > >>>>> which would run integration tests using Nova, Keystone, Neutron on > > > >>>>> the > > > >>>>> new eventlet version. > > > >>>>> > > > >>>>> eventlet 0.18.2 broke OpenStack Keystone and OpenStack Nova > > > >>>>> https://github.com/eventlet/eventlet/issues/296 > > > >>>>> https://github.com/eventlet/eventlet/issues/299 > > > >>>>> https://review.openstack.org/#/c/278147/ > > > >>>>> https://bugs.launchpad.net/nova/+bug/1544801 > > > >>>>> > > > >>>>> eventlet 0.18.3 broke OpenStack Neutron > > > >>>>> https://github.com/eventlet/eventlet/issues/301 > > > >>>>> https://bugs.launchpad.net/neutron/+bug/1546506 > > > >>>>> > > > >>>>> FYI eventlet 0.18.0 broke WSGI servers: > > > >>>>> https://github.com/eventlet/eventlet/issues/295 > > > >>>>> > > > >>>>> It was followed quickly by eventlet 0.18.2 to fix this issue. > > > >>>>> > > > >>>>> Sadly, it looks like bugfix releases of eventlet don't include a > > > >>>>> single > > > >>>>> bugfix, but include also other changes. For example, 0.18.3 fixed > > > >>>>> the > > > >>>>> bug #296 but introduced "wsgi: TCP_NODELAY enabled by default" > > > >>> optimization. > > > >>>>> > > > >>>>> IMHO the problem is not the release manager of eventlet, but more > > > >>>>> the > > > >>>>> lack of tests on eventlet, especially on OpenStack services. > > > >>>>> > > > >>>>> Current "Continious Delivery"-like with gates do detect bugs, yeah, > > > >>>>> but > > > >>>>> also block a lot of developers when the gates are broken. It doesn't > > > >>>>> seem trivial to investigate and fix eventlet issues. > > > >>>>> > > > >>>>> Victor > > > >>>>> > > > >>>> > > > >>>> Whether we cap or not, we should exclude the known broken versions. > > > >>>> It looks like getting back to a good version will also require > > > >>>> lowering the minimum version we support, since we have >=0.18.2 > > > >>>> now. > > > >>>> > > > >>>> What was the last version of eventlet known to work? > > > >>> > > > >>> 0.18.2 works. On the Nova side we had a failure around unit tests > > > >>> which > > > >>> was quite synthetic that we fixed. I don' know what the keystone issue > > > >>> turned out to be. > > > >>> > > > >> > > > >> I believe the keystone issue was a test specific issue, not a runtime > > > >> issue. We disabled the test. > > > >> --Morgan > > > > > > > > OK. Can someone from the neutron team verify that 0.18.2 works? If so, > > > > we can just exclude 0.18.3 and reset the constraint. > > > > > > I can confirm that neutron works with 0.18.2 as far as we know. > > > > > > > Great. If you (or someone else) wants to submit a requirements update, I > > can approve it. Ping me in #openstack-release. > > I think we're in a tough spot. > > My $0.02 is that we have to cap at <0.18.0 however .... > > We're (the openstack community) finding issues with eventlet which is good > (painful I agree, but good) for both communities. However we *are* trying to > lock down our release, and test/gate failures are very unwelcome in this > phase. > > 0.18.2 was "mostly" good but shows the following bug: > https://bugs.launchpad.net/nova/+bug/1544801 > > The eventlet team have been very responsive to our issue reports. I'm worried > if we cap at 0.18.2 (or 0.17.4) the underlying problems will go undressed and > unnoticed for too long and then we'll all loose :( Not least of all because > it's *very* likely if we cap this we'll release with the cap which means we're > basically stuck with it for the whole life-cycle of mitaka[1][2]
The current plan is to not cap, but exclude the bad versions. The patch to do that (https://review.openstack.org/#/c/281479/) does set the constraint to 0.18.2, so if we need to change that please drop a comment on the review. > Calls for "extra vetting" of new eventlet releases are slightly problematic, > and really require a short cross-project team to verify, as the "Yup is passes > dsvm-tempest-full" doesn't cut it. Can we construct a better automated test scenario? Doug > > Yours Tony. > > [1] Given we're @ R-7 and requirements freeze is R-5 > [2] Assuming we cap at <0.18.0, we don't generally allow raising the minimum > version of $library in a stable release __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
