Really nice work guys. Let's try and use this and see if it can help automate our d releases and reduce the effort involved in releasing.
/ Chris Sent from my iPhone > On Jul 29, 2016, at 13:09, Jose Lausuch <[email protected]> wrote: > > Hi, > > In order to provide a better user experience and useful feedback from the > daily runs in Jenkins, these features have been implemented in Functest > recently: > > “Blocking flag” for each test case: > For each test case executed in Functest, there is a flag called “blocking” > that can be set to True or False. If the test fails and the flag is set > ‘true’, Functest will stop the execution of the rest of the tests in the > daily loop exiting with error. If the test fails but the flag is set to > false, it will mark the Job as failed, but it will allow the execution of the > remaining test cases. > https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml > The purpose of this is to allow the critical test cases like the > healthcheck/vping be a “gate” or a filter i.e. if you can’t create VMs, why > should you bother in testing sfc or any other feature? Some other tests like > Tempest, will allow the execution of the preceding tests even though if there > is a failure. > > > Summary report: > Some of you gave us some feedback about the confusing output given by > Functest in Jekins if a job has failed. This has been improved a lot, > providing clear output messages to understand what is happening. But now you > don’t even need to go through all that output, you just need to scroll down > and at the bottom of the page you will see the “Functest report” to check > what has worked/failed along with some other useful information. > Example here: > https://build.opnfv.org/ci/job/functest-fuel-baremetal-daily-master/261/console > > > Black listing tests in Tempest: > If you know that in a certain scenario/installer, a tempest test case fails, > you can now blacklist the test case in this yaml: > https://git.opnfv.org/cgit/functest/tree/testcases/OpenStack/tempest/custom_tests/blacklist.txt > For example, for bgpvpn we have deactivated the tests that deal with floating > ips due to lacking support for floating IPs in ODL VPNService (which will be > fixed in the future). > So, if you know that a known bug will make a test case fail, you can add the > test in this list and it will be skipped. > NOTE: this is not meant to avoid running tests that fail without a reason. It > should be justified (known bug, feature lack of support, …). > > > You’ll see a lot of red in Jenkins, and that’s because Functest has become a > bit more strict when giving the final status. It will only return blue if all > the tests succeeded according to the defined success criteria, not only if > the execution of the test script succeeded (even though the test failed). > For example: > https://build.opnfv.org/ci/job/functest-fuel-baremetal-daily-master/261/console > https://build.opnfv.org/ci/job/functest-fuel-baremetal-daily-master/260/console > The exit status is -1 because there was >0 failures in Tempest-smoke, and the > success criteria for Tempest-smoke is 100%. > > This strictness has been changed like this after some feedback collected from > the community, in particular from the installer teams. > > > > Thanks, > > JOSE LAUSUCH > Senior Systems Designer > Ericsson > > _______________________________________________ > opnfv-tech-discuss mailing list > [email protected] > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
