Hi
we initiated the landing page some time ago:
http://testresults.opnfv.org/testing/#!/landingpage/table
front and back end are now ready
Jack started doing an update for Yardstick
It is now possible to call the Test API to set a score and see/modify
trust indicator after a CI run.
during the plugfest we saw that we were not fully aligned on what we
want to show on such landing page
the recent discussion on the reporting pages show also some differences
on the way we report our results
I think we need to agree on a minimum common view during a Testing
working meeting to have a consistent page
Concretely we have 2 scenario parameters for this landing page:
- Scenario score
- Scenario Trust Indicator
Scenario score
***************
Today there are 2 main views
VSPERF, Yardstick and Bottleneck => report Jenkins status assuming that
- SLA/threshold neither defined (VSPERF) nor taken into account
(Yardstick) to set test suite criteria to PASS or FAIL
- if test suite deals with lots of tests => only 1 aggregated result
corresponding to the test suite "runnability" in CI
In Functest we do a calculation based on the sum of all the individual
cases (https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617) that
is why in reporting page we have score like 14/27, 18/18, 1/9 ....
The first figure represent the success (18/18 = max score; 1/9 = very
poor results)
the max score represent the richness of the test suite, i.e. /9 = 3 test
cases in the suite; / 27 = 9 test cases.
We have a x3 factor due to the criteria of 4 consecutive runs. In the
evolution I had in mind to report only the scenario status
In Storperf Mark defined and takes into the threshold to set the test to
PASS or FAIL
So we may have 2 approaches for the landing page
- The Jenkins status (nb run OK / nb attempted over the last N days)
- The testing status (tests are run and results are according to project
success criteria e.g. for functest we expect the max score, 15/18 will
be considered as FAIL)
What is your view?
What kind of score would you give to the CI runs currently running?
shall we be all consistent or is it up to each project to define what it
wants to show in such page?
Trust Indicator
**************
this parameter allows to consider the scenario evolution versus time
in previous discussions we considered 3 states:
- Silver (init)
- Gold
- Platinium
this indicator is related to the scenario on which a test project is
running the test suite
A simple proposal for scenario promotion
/Morgan
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
_______________________________________________
opnfv-tech-discuss mailing list
opnfv-tech-discuss@lists.opnfv.org
https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
