On Tue, Apr 18, 2017 at 10:07 AM Arx Cruz <arxc...@redhat.com> wrote:
> On Tue, Apr 18, 2017 at 10:42 AM, Steven Hardy <sha...@redhat.com> wrote: > >> On Mon, Apr 17, 2017 at 12:48:32PM -0400, Justin Kilpatrick wrote: >> > On Mon, Apr 17, 2017 at 12:28 PM, Ben Nemec <openst...@nemebean.com> >> wrote: >> > > Tempest isn't really either of those things. According to another >> message >> > > in this thread it takes around 15 minutes to run just the smoke tests. >> > The smoke filter may take 15min to run, but it is possible to curate a TripleO own filter which could take less. It really depends in which area you see regressions mostly, but I would assume the most important bit for you is that services are running and working fine with each other, so and a couple of scenario tests may be enough to prove that. If there is no existing test that works for you in Tempest, you might even write your own Tempest plugin - that is as long as you are satisfied with API driven checks. andreaf > > > That's unacceptable for a lot of our CI jobs. >> > >> > > I rather spend 15 minutes running tempest than add a regression or a new > bug, which already happen in the past. > > >> > Ben, is the issue merely the time it takes? Is it the affect that time >> > taken has on hardware availability? >> >> It's both, but the main constraint is the infra job timeout, which is >> about >> 2.5hrs - if you look at our current jobs many regularly get close to (and >> sometimes exceed this), so we just don't have the time budget available to >> run exhasutive tests every commit. >> > > We have green light from infra to increase the job timeout to 5 hours, we > do that in our periodic full tempest job. > > >> >> > Should we focus on how much testing we can get into N time period? >> > Then how do we decide an optimal N >> > for our constraints? >> >> Well yeah, but that's pretty much how/why we ended up with pingtest, it's >> simple, fast, and provides an efficient way to do smoke tests, e.g >> creating >> just one heat resource is enough to prove multiple OpenStack services are >> running, as well as the DB/RPC etc etc. >> >> > I've been working on a full up functional test for OpenStack CI builds >> > for a long time now, it works but takes >> > more than 10 hours. IF you're interested in results kick through to >> > Kibana here [0]. Let me know off list if you >> > have any issues, the presentation of this data is all experimental >> still. >> >> This kind of thing is great, and I'd support more exhaustive testing via >> periodic jobs etc, but the reality is we need to focus on "bang for buck" >> e.g the deepest possible coverage in the most minimal amount of time for >> our per-commit tests - we rely on the project gates to provide a full API >> surface test, and we need to focus on more basic things like "did the >> service >> start", and "is the API accessible". Simple crud operations on a subset >> of >> the API's is totally fine for this IMO, whether via pingtest or some other >> means. >> >> > Right now we do have a periodic job running full tempest, with a few > skips, and because of the lack of tempest tests in the patches, it's being > pretty hard to keep it stable enough to have a 100% pass, and of course, > also the installation very often fails (like in the last five days). > For example, [1] is the latest run we have in periodic job that we get > results from tempest, and we have 114 failures that was caused by some new > code/change, and I have no idea which one was, just looking at the > failures, I can notice that smoke tests plus minimum basic scenario tests > would catch these failures and the developer could fix it and make me happy > :) > Now I have to spend several hours installing and debugging each one of > those tests to identify where/why it fails. > Before this run, we got 100% pass, but unfortunately I don't have the > results anymore, it was removed already from logs.openstack.org > > > >> Steve >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > [1] > http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha-tempest-oooq/0072651/logs/oooq/stackviz/#/stdin > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
