On Thu, Apr 6, 2017 at 5:32 AM, Sagi Shnaidman <sshna...@redhat.com> wrote: > HI, > > I think Rally or Browbeat and other performance oriented solutions won't > serve our needs, because we run TripleO CI on virtualized environment with > very limited resources. Actually we are pretty close to full utilizing these > resources when deploying openstack, so very little is available for test. > It's not a problem to run tempest API tests because they are cheap - take > little time, little resources, but also gives little coverage. Scenario test > are more interesting and gives us more coverage, but also takes a lot of > resources (which we don't have sometimes).
Sagi, In my original message I mentioned a "targeted" test, I should explained that more. We could configure the specific scenario so that the load on the virt overcloud would be minimal. Justin Kilpatrick already have Browbeat integrated with TripleO Quickstart[1], so there shouldn't be much work to try this proposed solution. > > It may be useful to run a "limited edition" of API tests that maximize > coverage and don't duplicate, for example just to check service working > basically, without covering all its functionality. It will take very little > time (i.e. 5 tests for each service) and will give a general picture of > deployment success. It will cover fields that are not covered by pingtest as > well. > > I think could be an option to develop a special scenario tempest tests for > TripleO which would fit our needs. I haven't looked at Tempest in a long time, so maybe its functionality has improved. I just saw the opportunity to integrate Browbeat/Rally into CI to test the functionality of OpenStack services, while also capturing performance metrics. Joe [1] https://github.com/openstack/browbeat/tree/master/ci-scripts > > Thanks > > > On Wed, Apr 5, 2017 at 11:49 PM, Emilien Macchi <emil...@redhat.com> wrote: >> >> Greetings dear owls, >> >> I would like to bring back an old topic: running tempest in the gate. >> >> == Context >> >> Right now, TripleO gate is running something called pingtest to >> validate that the OpenStack cloud is working. It's an Heat stack, that >> deploys a Nova server, some volumes, a glance image, a neutron network >> and sometimes a little bit more. >> To deploy the pingtest, you obviously need Heat deployed in your >> overcloud. >> >> == Problems: >> >> Although pingtest has been very helpful over the last years: >> - easy to understand, it's an Heat template, like an OpenStack user >> would do to deploy their apps. >> - fast: the stack takes a few minutes to be created and validated >> >> It has some limitations: >> - Limitation to what Heat resources support (example: some OpenStack >> resources can't be managed from Heat) >> - Impossible to run a dynamic workflow (test a live migration for example) >> >> == Solutions >> >> 1) Switch pingtest to Tempest run on some specific tests, with feature >> parity of what we had with pingtest. >> For example, we could imagine to run the scenarios that deploys VM and >> boot from volume. It would test the same thing as pingtest (details >> can be discussed here). >> Each scenario would run more tests depending on the service that they >> run (scenario001 is telemetry, so it would run some tempest tests for >> Ceilometer, Aodh, Gnocchi, etc). >> We should work at making the tempest run as short as possible, and the >> close as possible from what we have with a pingtest. >> >> 2) Run custom scripts in TripleO CI tooling, called from the pingtest >> (heat template), that would run some validations commands (API calls, >> etc). >> It has been investigated in the past but never implemented AFIK. >> >> 3) ? >> >> I tried to make this text short and go straight to the point, please >> bring feedback now. I hope we can make progress on $topic during Pike, >> so we can increase our testing coverage and detect deployment issues >> sooner. >> >> Thanks, >> -- >> Emilien Macchi >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > -- > Best regards > Sagi Shnaidman > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
