On 04/05/17 15:01, Ghanshyam Mann wrote: > > On Wed, May 3, 2017 at 21:57 Andrea Frittoli > <andrea.fritt...@gmail.com <mailto:andrea.fritt...@gmail.com>> wrote: > > On Tue, May 2, 2017 at 2:41 PM Jordan Pittier > <jordan.pitt...@scality.com <mailto:jordan.pitt...@scality.com>> > wrote: > > On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann > <ghanshyamm...@gmail.com <mailto:ghanshyamm...@gmail.com>> wrote: > > In Cinder, there are many features/APIs which are backend > specific and > will return 405 or 501 if same is not implemented on any > backend [1]. > If such tests are implemented in Tempest, then it will > break some gate > where that backend job is voting. like ceph job in > glance_store gate. > > There been many such cases recently where ceph jobs were > broken due to > such tests and recently it is for force-delete backup > feature[2]. > Reverting force-delete tests in [3]. To resolve such cases > at some > extend, Jon is going to add a white/black list of tests > which can run > on ceph job [4] depends on what all feature ceph > implemented. But this > does not resolve it completely due to many reason like > 1. External use of Tempest become difficult where user > needs to know > what all tests to skip for which backend > 2. Tempest tests become too specific to backend. > > Now there are few options to resolve this: > 1. Tempest should not tests such API/feature which are backend > specific like mentioned by api-ref like[1]. > > So basically, if one of the 50 Cinder driver doesn't support a > feature, we should never test that feature ? What about the 49 > other drivers ? If a feature exists and can be tested in the > Gate (with whatever default config/driver is shipped) then I > think we should test it. > > > 2. Tempest test can be disabled/skip based on backend. - > This is not > good idea as it increase config options and overhead of > setting those. > > Using regex and blacklist, any 3rd party CI can skip any test > based on the test ID. Without introducing a config flag. > See: > https://github.com/openstack-infra/project-config/blob/1cea31f402b6b0cccc47cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871 > > > This way each 3rd party system has to maintain its own list, which > has the advantage that > different teams maintain their own list (which is nice from an > ownership and scale pov). > > However I think such list of tests are not as re-usable as having > a devstack plugin (or an > ansible or puppet module) changing a few tempest config options. > > > Humm, I am little bit hesitate to go that way. For gate and CI it > might be good solution but for production cloud testing it makes > Tenpest difficult to use. > > If I use Tempest to test my cloud, few tests going to fail as those > were not supported by cinder driver my cloud has or going to have. > I do not have any way to configure something so that test can be > disabled. Instead I need to maintain list of tests to run or skip. And > that list is not static, it grows dynamically. > This makes Tempest difficult to use. > >
Agree. We (Catalyst Cloud based in NZ) are using Tempest as the monitoring tool and CI/CD gate for our cloud. But we do have to maintain a white list of test cases because there are some cases are not fitting our cloud. > > > > > 3. Tempest test can verify behavior with if else condition > as per > backend. This is bad idea and lose the test strength. > > Yeah, that's bad. > > > IMO options 1 is better options. More feedback are welcome. > > > ..1 > > https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup > ..2 https://bugs.launchpad.net/glance/+bug/1687538 > ..3 https://review.openstack.org/#/c/461625/ > ..4 > > http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html > > -gmann > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- > -gmann > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Cheers & Best regards, Feilong Wang (王飞龙) -------------------------------------------------------------------------- Senior Cloud Software Engineer Tel: +64-48032246 Email: flw...@catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
