Hi Good try.
Could this be run by command line without CI? summary report are good dimensions, on the other hand, it hides some info. in tempest, minimum/api/scenario/service tests are different category. We usually to use percentage for the same test case pass rate by running it with number of times. /MatthewLi 发件人: [email protected] [mailto:[email protected]] 代表 Jose Lausuch 发送时间: 2016年7月29日 19:09 收件人: [email protected] 主题: [opnfv-tech-discuss] Functest feedback in Jenkins Hi, In order to provide a better user experience and useful feedback from the daily runs in Jenkins, these features have been implemented in Functest recently: “Blocking flag” for each test case: For each test case executed in Functest, there is a flag called “blocking” that can be set to True or False. If the test fails and the flag is set ‘true’, Functest will stop the execution of the rest of the tests in the daily loop exiting with error. If the test fails but the flag is set to false, it will mark the Job as failed, but it will allow the execution of the remaining test cases. https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml The purpose of this is to allow the critical test cases like the healthcheck/vping be a “gate” or a filter i.e. if you can’t create VMs, why should you bother in testing sfc or any other feature? Some other tests like Tempest, will allow the execution of the preceding tests even though if there is a failure. Summary report: Some of you gave us some feedback about the confusing output given by Functest in Jekins if a job has failed. This has been improved a lot, providing clear output messages to understand what is happening. But now you don’t even need to go through all that output, you just need to scroll down and at the bottom of the page you will see the “Functest report” to check what has worked/failed along with some other useful information. Example here: https://build.opnfv.org/ci/job/functest-fuel-baremetal-daily-master/261/console Black listing tests in Tempest: If you know that in a certain scenario/installer, a tempest test case fails, you can now blacklist the test case in this yaml: https://git.opnfv.org/cgit/functest/tree/testcases/OpenStack/tempest/custom_tests/blacklist.txt For example, for bgpvpn we have deactivated the tests that deal with floating ips due to lacking support for floating IPs in ODL VPNService (which will be fixed in the future). So, if you know that a known bug will make a test case fail, you can add the test in this list and it will be skipped. NOTE: this is not meant to avoid running tests that fail without a reason. It should be justified (known bug, feature lack of support, …). You’ll see a lot of red in Jenkins, and that’s because Functest has become a bit more strict when giving the final status. It will only return blue if all the tests succeeded according to the defined success criteria, not only if the execution of the test script succeeded (even though the test failed). For example: https://build.opnfv.org/ci/job/functest-fuel-baremetal-daily-master/261/console https://build.opnfv.org/ci/job/functest-fuel-baremetal-daily-master/260/console The exit status is -1 because there was >0 failures in Tempest-smoke, and the success criteria for Tempest-smoke is 100%. This strictness has been changed like this after some feedback collected from the community, in particular from the installer teams. Thanks, JOSE LAUSUCH Senior Systems Designer Ericsson
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
