That would be great. The boot from volume test remains one of the top
failure scenarios, both in the gate, and for end users. It would be
really good to get a handle on why and get that shored up.
-Sean
On 03/02/2016 08:18 AM, D'Angelo, Scott wrote:
> +1 to making the testing process better.
> It has been discussed that services could/should consider devoting some or
> all of a release cycle to stability and/or quality.
> I propose the Cinder team makes improving and fixing the tests and test
> process a priority for the Newton cycle.
>
> Scott D'Angelo (scottda)
> ________________________________
> From: Ivan Kolodyazhny [e...@e0ne.info]
> Sent: Wednesday, March 02, 2016 4:25 AM
> To: OpenStack Development Mailing List
> Subject: [openstack-dev] [cinder] Proposal: changes to our current testing
> process
>
> Hi Team,
>
> Here are my thoughts and proposals how to make Cinder testing process better.
> I won't cover "3rd party CI's" topic here. I will share my opinion about
> current and feature jobs.
>
>
> Unit-tests
>
> * Long-running tests. I hope, everybody will agree that unit-tests must
> be quite simple and very fast. Unit tests which takes more than 3-5 seconds
> should be refactored and/or moved to 'integration' tests.
> Thanks to Tom Barron for several fixes like [1]. IMO, we it would be good to
> have some hacking checks to prevent such issues in a future.
>
> * Tests coverage. We don't check it in an automatic way on gates.
> Usually, we require to add some unit-tests during code review process. Why
> can't we add coverage job to our CI and do not merge new patches, with will
> decrease tests coverage rate? Maybe, such job could be voting in a future to
> not ignore it. For now, there is not simple way to check coverage because
> 'tox -e cover' output is not useful [2].
>
> Functional tests for Cinder
>
> We introduced some functional tests last month [3]. Here is a patch to infra
> to add new job [4]. Because these tests were moved from unit-tests, I think
> we're OK to make this job voting. Such tests should not be a replacement for
> Tempest. They even could tests Cinder with Fake Driver to make it faster and
> not related on storage backends issues.
>
>
> Tempest in-tree tests
>
> Sean started work on it [5] and I think it's a good idea to get them in
> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
> backend.
>
>
> Functional tests for python-brick-cinderclient-ext
>
> There are patches that introduces functional tests [6] and new job [7].
>
>
> Functional tests for python-cinderclient
>
> We've got a very limited set of such tests and non-voting job. IMO, we can
> run them even with Cinder Fake Driver to make them not depended on a storage
> backend and make it faster. I believe, we can make this job voting soon.
> Also, we need more contributors to this kind of tests.
>
>
> Integrated tests for python-cinderclient
>
> We need such tests to make sure that we won't break Nova, Heat or other
> python-cinderclient consumers with a next merged patch. There is a thread in
> openstack-dev ML about such tests [8] and proposal [9] to introduce them to
> python-cinderclient.
>
>
> Rally tests
>
> IMO, it would be good to have new Rally scenarios for every patches like
> 'improves performance', 'fixes concurrency issues', etc. Even if we as a
> Cinder community don't have enough time to implement them, we have to ask for
> them in reviews, openstack-dev ML, file Rally bugs and blueprints if needed.
>
>
> [1] https://review.openstack.org/#/c/282861/
> [2] http://paste.openstack.org/show/488925/
> [3] https://review.openstack.org/#/c/267801/
> [4] https://review.openstack.org/#/c/287115/
> [5] https://review.openstack.org/#/c/274471/
> [6] https://review.openstack.org/#/c/265811/
> [7] https://review.openstack.org/#/c/265925/
> [8] http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
> [9] https://review.openstack.org/#/c/279432/
>
>
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
--
Sean Dague
http://dague.net
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
