On 03/02/2016 10:07 AM, Ivan Kolodyazhny wrote: > Eric, > > For now, we test Cinder API with some concurrency only with Rally, so, IMO, > it's reasonable get more scenarios for API races fixes. > > It's not a hard task to implement new scenarios, they are pretty simple: > [11] and [12] >
Sure, these are simple, but I think it's nowhere near that simple to write a scenario which will prove that "remove API races" works correctly. > [11] > https://github.com/openstack/rally/blob/master/rally/plugins/openstack/scenarios/cinder/volumes.py#L535 > [12] > https://github.com/openstack/rally/blob/master/rally-jobs/cinder.yaml#L516 > > Regards, > Ivan Kolodyazhny, > http://blog.e0ne.info/ > > On Wed, Mar 2, 2016 at 4:50 PM, Eric Harney <[email protected]> wrote: > >> On 03/02/2016 09:36 AM, Ivan Kolodyazhny wrote: >>> Eric, >>> >>> There are Gorka's patches [10] to remove API Races >>> >>> >>> [10] >>> >> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified >>> >>> Regards, >>> Ivan Kolodyazhny, >>> http://blog.e0ne.info/ >>> >> >> So the second part of my question is, is writing a Rally job to prove >> out that code a reasonable task? >> >> How hard is that to do and what does it look like? >> >>> On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney <[email protected]> wrote: >>> >>>> On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote: >>>>> Hi Team, >>>>> >>>>> Here are my thoughts and proposals how to make Cinder testing process >>>>> better. I won't cover "3rd party CI's" topic here. I will share my >>>> opinion >>>>> about current and feature jobs. >>>>> >>>>> >>>>> Unit-tests >>>>> >>>>> - Long-running tests. I hope, everybody will agree that unit-tests >>>> must >>>>> be quite simple and very fast. Unit tests which takes more than 3-5 >>>> seconds >>>>> should be refactored and/or moved to 'integration' tests. >>>>> Thanks to Tom Barron for several fixes like [1]. IMO, we it would be >>>>> good to have some hacking checks to prevent such issues in a future. >>>>> >>>>> - Tests coverage. We don't check it in an automatic way on gates. >>>>> Usually, we require to add some unit-tests during code review >>>> process. Why >>>>> can't we add coverage job to our CI and do not merge new patches, >> with >>>>> will decrease tests coverage rate? Maybe, such job could be voting >> in >>>> a >>>>> future to not ignore it. For now, there is not simple way to check >>>> coverage >>>>> because 'tox -e cover' output is not useful [2]. >>>>> >>>>> >>>>> Functional tests for Cinder >>>>> >>>>> We introduced some functional tests last month [3]. Here is a patch to >>>>> infra to add new job [4]. Because these tests were moved from >>>> unit-tests, I >>>>> think we're OK to make this job voting. Such tests should not be a >>>>> replacement for Tempest. They even could tests Cinder with Fake Driver >> to >>>>> make it faster and not related on storage backends issues. >>>>> >>>>> >>>>> Tempest in-tree tests >>>>> >>>>> Sean started work on it [5] and I think it's a good idea to get them in >>>>> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a >> real >>>>> backend. >>>>> >>>>> >>>>> Functional tests for python-brick-cinderclient-ext >>>>> >>>>> There are patches that introduces functional tests [6] and new job [7]. >>>>> >>>>> >>>>> Functional tests for python-cinderclient >>>>> >>>>> We've got a very limited set of such tests and non-voting job. IMO, we >>>> can >>>>> run them even with Cinder Fake Driver to make them not depended on a >>>>> storage backend and make it faster. I believe, we can make this job >>>> voting >>>>> soon. Also, we need more contributors to this kind of tests. >>>>> >>>>> >>>>> Integrated tests for python-cinderclient >>>>> >>>>> We need such tests to make sure that we won't break Nova, Heat or other >>>>> python-cinderclient consumers with a next merged patch. There is a >> thread >>>>> in openstack-dev ML about such tests [8] and proposal [9] to introduce >>>> them >>>>> to python-cinderclient. >>>>> >>>>> >>>>> Rally tests >>>>> >>>>> IMO, it would be good to have new Rally scenarios for every patches >> like >>>>> 'improves performance', 'fixes concurrency issues', etc. Even if we as >> a >>>>> Cinder community don't have enough time to implement them, we have to >> ask >>>>> for them in reviews, openstack-dev ML, file Rally bugs and blueprints >> if >>>>> needed. >>>>> >>>> >>>> Are there any recent examples of a fix like this recently where it would >>>> seem like a reasonable task to write a Rally scenario along with the >> patch? >>>> >>>> Not being very familiar with Rally (as I think most of us aren't), I'm >>>> having a hard time picturing this. >>>> >>>>> >>>>> [1] https://review.openstack.org/#/c/282861/ >>>>> [2] http://paste.openstack.org/show/488925/ >>>>> [3] https://review.openstack.org/#/c/267801/ >>>>> [4] https://review.openstack.org/#/c/287115/ >>>>> [5] https://review.openstack.org/#/c/274471/ >>>>> [6] https://review.openstack.org/#/c/265811/ >>>>> [7] https://review.openstack.org/#/c/265925/ >>>>> [8] >>>>> >>>> >> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html >>>>> [9] https://review.openstack.org/#/c/279432/ >>>>> >>>>> >>>>> Regards, >>>>> Ivan Kolodyazhny, >>>>> http://blog.e0ne.info/ >>>>> >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: [email protected]?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
