On 3 February 2016 at 04:28, Sean Dague <s...@dague.net> wrote: > On 02/02/2016 10:03 PM, Matthew Treinish wrote: > > On Tue, Feb 02, 2016 at 05:09:47PM -0800, Armando M. wrote: > >> Folks, > >> > >> We have some IPv6 related bugs [1,2,3] that have been lingering for some > >> time now. They have been hurting the gate (e.g. [4] the most recent > >> offending failure) and since it looks like they have been without owners > >> nor a plan of action for some time, I made the hard decision of skipping > >> them [5] ahead of the busy times ahead. > > > > So TBH I don't think the failure rate for these tests are really at a > point > > necessitating a skip: > > > > > http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_slaac > > > http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_dhcp6_stateless_from_os > > > http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_dhcp6_stateless_from_os > > > > (also just a cool side-note, you can see the very obvious performance > regression > > caused by the keystonemiddleware release and when we excluded that > version in > > requirements) > > > > Well, test_dualnet_dhcp6_stateless_from_os is kinda there with a ~10% > failure > > rate, but the other 2 really aren't. I normally would be -1 on the skip > patch > > because of that. We try to save the skips for cases where the bugs are > really > > severe and preventing productivity at a large scale. > > > > But, in this case these ipv6 tests are kinda of out of place in tempest. > Having > > all the permutations of possible ip allocation configurations always > seemed a > > bit too heavy handed. These tests are also consistently in the top 10 > slowest > > for a run. We really should have trimmed down this set a while ago so > we're only > > have a single case in tempest. Neutron should own the other possible > > configurations as an in-tree test. > > > > Brian Haley has a patch up from Dec. that was trying to clean it up: > > > > https://review.openstack.org/#/c/239868/ > > > > We probably should revisit that soon, since quite clearly no one is > looking at > > these right now. > > We definitely shouldn't be running all the IPv6 tests. > > But I also think the assumption that the failure rate is low is not a > valid reason to keep a test. Unreliable tests that don't have anyone > looking into them should be deleted. They are providing negative value. > Because people just recheck past them even if their code made the race > worse. So any legitimate issues they are exposing are being ignored. > > If the neutron PTL wants tests pulled, we should just do it. > > Thanks for the support! Having said, I think it's important to make a judgement call on a case by case basis, because removing tests blindly might as well backfire.
In this specific instance and all things considered, merging [2] (or even better [1]) feel like a net gain. Cheers, Armando [1] https://review.openstack.org/#/c/239868/ [2] https://review.openstack.org/#/c/275457/ > -Sean > > -- > Sean Dague > http://dague.net > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
