On Tue, Aug 26, 2014 at 4:47 PM, Salvatore Orlando <sorla...@nicira.com> wrote:
> TL; DR > A few folks are proposing to stop running tests for neutron advanced > services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on > the neutron gate. > > Reason: projects like nova are 100% orthogonal to neutron advanced > services. Also, there have been episodes in the past of unreliability of > tests for these services, and it would be good to limit affected projects > considering that more api tests and scenarios are being added. > > ----- > > So far the neutron full job runs tests (api and scenarios) for neutron > "core" functionality as well as neutron "advanced services", which run as > neutron service plugin. > > It's highly unlikely, if not impossible, that changes in projects such as > nova, glance or ceilometer can have an impact on the stability of these > services. > On the other hand, instability in these services can trigger gate failures > in unrelated projects as long as tests for these services are run in the > neutron full job in the integrated gate. There have already been several > gate-breaking bugs in lbaas scenario tests are firewall api tests. > Admittedly, advanced services do not have the same level of coverage as > core neutron functionality. Therefore as more tests are being added, there > is an increased possibility of unearthing dormant bugs. > > I support this split but for slightly different reasons. I am under the impression that neutron advanced services are not ready for prime time. If that is correct I don't think we should be gating on things that aren't ready. > For this reason we are proposing to not run anymore tests for neutron > advanced services in the integrated gate, but keep them running on the > neutron gate. > This means we will have two neutron jobs: > 1) check-tempest-dsvm-neutron-full which will run only "core" neutron > functionality > 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron full > job is today. > Using my breakdown, the extended job would include experimental neutron features. > > The former will be part of the integrated gate, the latter will be part of > the neutron gate. > Considering that other integrating services should not have an impact on > neutron advanced services, this should not make gate testing asymmetric. > > However, there might be exceptions for: > - "orchestration" project like heat which in the future might leverage > capabilities like load balancing > - oslo-* libraries, as changes in them might have an impact on neutron > advanced services, since they consume those libraries > Once another service starts consuming an advanced feature I think it makes sense to move it to the main neutron-full job. Especially if we assume that things will only depend on neutron features that are not too experimental. > > Another good question is whether "extended" tests should be performed as > part of functional or tempest checks. My take on this is that scenario > tests should always be part of tempest. On the other hand I reckon API > tests should exclusively be part of functional tests, but as so far tempest > is running a gazillion of API tests, this is probably a discussion for the > medium/long term. > > In order to add this new job there are a few patches under review: > [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it. > [3] Are the patches implementing a blueprint which will enable us to > specify for which extensions test should be executed. > > Finally, one more note about smoketests. Although we're planning to get > rid of them soon, we still have failures in the pg job because of [4]. For > this reasons smoketests are still running for postgres in the integrated > gate. As load balancing and firewall API tests are part of it, they should > be removed from the smoke test executed on the integrated gate ([5], [6]). > This is a temporary measure until the postgres issue is fixed. > ++ > > Regards, > Salvatore > > [1] https://review.openstack.org/#/c/114933/ > [2] https://review.openstack.org/#/c/114932/ > [3] > https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z > [4] https://bugs.launchpad.net/nova/+bug/1305892 > [5] https://review.openstack.org/#/c/115022/ > [6] https://review.openstack.org/#/c/115023/ > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
