On Mon, Mar 21, 2016 at 9:59 AM, Steven Hardy <sha...@redhat.com> wrote: > On Mon, Mar 21, 2016 at 09:41:42AM -0400, Emilien Macchi wrote: >> On Mon, Mar 21, 2016 at 6:57 AM, Steven Hardy <sha...@redhat.com> wrote: >> > On Fri, Mar 18, 2016 at 01:27:33PM +0000, arkady_kanev...@dell.com wrote: >> >> Emilien, >> >> >> >> Agree on the rant. But not clear on concrete proposal to fix it. >> >> >> >> Spend more time “fixing” CI and use Tempest as a gate is a bit wage. >> >> >> >> Unless we test known working version of each project in TripleO CI you >> >> are >> >> dependent on health of other components. >> > >> > I've so far resisted replying to this thread, because while valid, many of >> > the concerns expressed by Emilien are quite general complaints, and it's >> > hard to reply with specific solutions. >> > >> > However work *is* going on to improve many of these problems, let's see if >> > I can provide a summary, to clarify the various "concrete proposals" which >> > do exist. >> > >> > 1. Core team & review velocity >> > >> > We've had a small and very overloaded core team for a while now, and this >> > will be helped by expanding our community to include those who've been >> > regularly contributing excellent work and reviews as core reviewers: >> > >> > http://lists.openstack.org/pipermail/openstack-dev/2016-February/087774.html >> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/089235.html >> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/089912.html >> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/089913.html >> > >> > Note that I personally think it's absolutely fine for folks to be more >> > expert in some subsystem and to focus review extra attention on e.g API, >> > UI, Puppet or whatever. This subsystem-core model has been well proven in >> > other projects, and folks will naturally broaden their areas of deeper >> > knowledge over time. >> > >> > Related to this is movement of code, such as the puppet-tripleo refactoring >> > mentioned by Michael - this has already started, and will help with >> > providing a cleaner interface between the puppet and heat pieces (which >> > will also help focus reviewer attention appropriately). >> >> Indeed, Michael, Dan & I are working on moving out the Puppet code >> from THT to puppet-tripleo. >> That's a nice move, and I appreciate TripleO team support on it. >> >> > 2. Day 1 developer experience >> > >> > This is closely related to the CI failure rate - there are efforts to >> > integrate with the RDO tripleo-quickstart tooling, which simplifies the >> > initial undercloud setup, and potentially makes consuming pre-built, >> > validated undercloud images (probably output artefacts from our new >> > periodic CI job) much easier. >> > >> > So, this will mean that both developers and CI can potentially be less >> > regularly impacted by trunk regressions which often cause CI to fail, and >> > break developer environments. >> > >> > https://review.openstack.org/#/c/276810/5 >> > >> > 3. CI coverage and trunk failure rate >> > >> > We've been working really hard to improve things here, which are really >> > several inter-related issues: >> > >> > - Lack of Hardware capacity in the tripleo CI cloud >> > - Frequent trunk regressions breaking our CI >> > - Lack of coverage of some key features (network isolation, SSL, IPv6, >> > upgrades) >> > - Lack of coverage for vendor plugin templates/puppet code >> > >> > There's work ongoing to improve this from multiple perspectives: >> > >> > New periodic CI job (to be used for automated promotion of the >> > current-tripleo repo, and for pre-built undercloud images): >> > https://review.openstack.org/#/c/271370/ >> > >> > Add network isolation support to CI: >> > https://review.openstack.org/#/c/288163/ >> > >> > Test SSL enabled in overcloud: >> > https://review.openstack.org/#/c/281988/ >> > >> > CI coverage of IPv6: >> > https://review.openstack.org/#/c/289445/ >> > >> > Discussion around better documented integration for third-party CI: >> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/088972.html >> >> Do we have plans to execute Tempest? > > This is something which has been discussed several times, but right now we > don't have the time available to run it per-commit because we'll hit the > job timeout. > > This situation will improve as we gain time e.g through use of cached > pre-built images, but right now I think we could look at enabling it only > on the periodic job when that is fully proven. > > Having said that, I should point out that tempest doesn't get us great > coverage of some newer projects - e.g all Heat scenario coverage was moved > out of the tempest tree, and other projects have done similar AFAIK, so we > may end up with very sparse API surface tests (or nothing at all) in these > cases.
In Puppet OpenStack CI, we execute smoke tests (a few tests of each service, and some scenarios), and some tests not in smoke, for Ironic, Aodh, Horizon. It takes ~15 minutes to execute them. This is how we proceed: https://github.com/openstack/puppet-openstack-integration/blob/master/run_tests.sh#L116-L134 I'm asking because to me Tempest is the official project for OpenStack testing and if we enable third party CI later, with external CI, we need to find a common way to test TripleO clouds. > There is probably more we can do within the existing pingtest though, it > creates the instance inside a heat stack, so we could just add a bunch more > resources for all the overcloud services, and pretty quickly prove that the > deployed services are at least running (without extending the runtime much). > > Steve > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Emilien Macchi __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
