On Aug 27, 2014, at 1:47 AM, Salvatore Orlando <sorla...@nicira.com> wrote:
> TL; DR > A few folks are proposing to stop running tests for neutron advanced services > [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on the neutron > gate. > > Reason: projects like nova are 100% orthogonal to neutron advanced services. > Also, there have been episodes in the past of unreliability of tests for > these services, and it would be good to limit affected projects considering > that more api tests and scenarios are being added. Given how many rechecks I’ve had to do to merge what are effectively no-op patches to infra/config, most often due to the full neutron job exhibiting sporadic failures, I fully support this change. I think we need time to stabilize the tests for advanced services against just neutron before we consider slowing down merges for other projects. > > ----- > > So far the neutron full job runs tests (api and scenarios) for neutron "core" > functionality as well as neutron "advanced services", which run as neutron > service plugin. > > It's highly unlikely, if not impossible, that changes in projects such as > nova, glance or ceilometer can have an impact on the stability of these > services. > On the other hand, instability in these services can trigger gate failures in > unrelated projects as long as tests for these services are run in the neutron > full job in the integrated gate. There have already been several > gate-breaking bugs in lbaas scenario tests are firewall api tests. > Admittedly, advanced services do not have the same level of coverage as core > neutron functionality. Therefore as more tests are being added, there is an > increased possibility of unearthing dormant bugs. > > For this reason we are proposing to not run anymore tests for neutron > advanced services in the integrated gate, but keep them running on the > neutron gate. > This means we will have two neutron jobs: > 1) check-tempest-dsvm-neutron-full which will run only "core" neutron > functionality > 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron full > job is today. > > The former will be part of the integrated gate, the latter will be part of > the neutron gate. > Considering that other integrating services should not have an impact on > neutron advanced services, this should not make gate testing asymmetric. > > However, there might be exceptions for: > - "orchestration" project like heat which in the future might leverage > capabilities like load balancing > - oslo-* libraries, as changes in them might have an impact on neutron > advanced services, since they consume those libraries > > Another good question is whether "extended" tests should be performed as part > of functional or tempest checks. My take on this is that scenario tests > should always be part of tempest. On the other hand I reckon API tests should > exclusively be part of functional tests, but as so far tempest is running a > gazillion of API tests, this is probably a discussion for the medium/long > term. As you say, tempest should retain responsibility for ‘golden-path’ integration tests involving other OpenStack services (’scenario tests’). Everything else should eventually be in-tree, though the transition period to achieve this is likely to be multi-cycle. m. > > In order to add this new job there are a few patches under review: > [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it. > [3] Are the patches implementing a blueprint which will enable us to specify > for which extensions test should be executed. > > Finally, one more note about smoketests. Although we're planning to get rid > of them soon, we still have failures in the pg job because of [4]. For this > reasons smoketests are still running for postgres in the integrated gate. As > load balancing and firewall API tests are part of it, they should be removed > from the smoke test executed on the integrated gate ([5], [6]). This is a > temporary measure until the postgres issue is fixed. > > Regards, > Salvatore > > [1] https://review.openstack.org/#/c/114933/ > [2] https://review.openstack.org/#/c/114932/ > [3] > https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z > [4] https://bugs.launchpad.net/nova/+bug/1305892 > [5] https://review.openstack.org/#/c/115022/ > [6] https://review.openstack.org/#/c/115023/ > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
