Hi,
You might have noticed recently that Functest is reporting BLUE even though
some test cases fail, e.g. [1].
This is due to the new way of running Functest using different Docker images
based on Alpine:
1. healthcheck
connection_check, api_check, snaps_healthcheck...
2. smoke
vping, odl, tempest, rally, ….
3. features
promise, doctor, bgpvpn, sfc, ...
4. vnf
cloudify_ims, ...
Since they are run independently one after each other, we just return BLUE if
the execution succeeds regardless of the test results. This approach is also
done currently in Yardstick as well. You can see the status of the
scenarios/test results on the Functest dashboard [2] which is a snapshot of
what happens in CI.
If anyone thinks this is not a good approach, we could discuss it during the
TSC meeting or the Weekly Technical Community meeting.
Thank you,
Jose
[1]
https://build.opnfv.org/ci/view/functest/job/functest-compass-baremetal-daily-master/1473/
<https://build.opnfv.org/ci/view/functest/job/functest-compass-baremetal-daily-master/1473/>
[2] http://testresults.opnfv.org/reporting/master/functest/status-apex.html
<http://testresults.opnfv.org/reporting/master/functest/status-apex.html>
_______________________________________________
opnfv-tech-discuss mailing list
opnfv-tech-discuss@lists.opnfv.org
https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
