Thanks Morgan ... very good proposals to help move the test community forward.
- We need to make good choices about what tests are run ... to do this we need a view of current test-coverage, test-tool capabilities and accurate test-case descriptions. In the end it needs to be about selecting test-cases ... not the project. - Overlap and differences between test projects ... for example there are many "latency tests" but they measure different things and will have very different results depending on scenarios, tools, methodology, test configurations, etc. - There are no pass/fail criteria for performance tests ... but how do we develop minimum thresholds for passing the tests ... this can't be something arbitrary defined by each project. Sorry I can't be at Open Stack ... please add above points to the test discussions. Trevor -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Gaoliang (kubi) Sent: Thursday, October 20, 2016 1:01 AM To: Jose Lausuch <[email protected]>; [email protected]; [email protected] Cc: yaohelan <[email protected]>; serena feng <[email protected]>; [email protected] Subject: Re: [test-wg] CI evolution on impact on test projects Hi , Thanks for the great slides, CI evolution is definitely an importing thing in Danube release. for yardstick in CI, the status is as below: definition of smoke test / weekly scope / test slicing ->Yardstick have drafted the proposal of test slicing in Colorado release, but the framework do not support it in c, we're planning to support it in Danube Automatic selection of tests according to scenarios ->Yardstick already support executing a scenario via a test-suite yaml in Colorado. Selection is not automatic, it need to be defined in a scenario test-suite yaml. Test used for installer gating -> no scenario scoring -> yes, but yardstick only calculate the score by the scenario fail/pass status, after Slicing of Yardstick Test cases, Yardstick will include Tier score in Danube. I agree with Jose about a layer between CI (Jenkins) and the test project. That would be helpful for all test projects. Let 's discuss details in OpenStack Summit. Regards, Kubi -----邮件原件----- 发件人: [email protected] [mailto:[email protected]] 代表 Jose Lausuch 发送时间: 2016年10月18日 16:01 收件人: [email protected]; [email protected] 抄送: yaohelan; serena feng; [email protected] 主题: Re: [opnfv-tech-discuss] CI evolution on impact on test projects Hi, Although it took a while to go through all your slides and understand them all (the 2 big example slides helped a lot), I agree in general with what you explain. There are some confusing assumptions like "if a case(weekly) fails => still weekly loop but daily status" ... but the concept behind sounds like a good approach to follow. We should definitively move to different keywords to latest/daily/weekly. Silver/Gold/Platinum sounds good enough, and congratulations for those scenarios that get the Platinum badge from Functest because it comprises a lot of long tests.. The promotions are performed in a logical way and everything is calculated and stored in the central DB. Just some concerns come to my mind: - if a couple scenarios (say 3 or 4) are promoted to "Gold", it means that they will be running all the tests (weekly loop) in CI (unless there is a regression on a daily test if I understand correctly) ==> each Funtest weekly run might take 3-5 hours (say 8 hours for Deploy+Functest+Yardstick) ==> CI needs more time to complete an entire installer loop going through all the scenarios. Today, in some cases, a scenario takes ~2 days to run again in CI (with only daily Functest tests) and some scenarios could be considered nowadays as "Gold".. Are we prepared to run weekly tests on so many scenarios? Will we be able to do this the weeks before a release? - This new logic is very CI-oriented. We also need to bypass it for the case that an end user just wants to run all the tests the same way we do now (i.e. functest testcase run all). So, the logic has to be implemented in another layer on top, not in the test project framework as such. I can imagine a layer between CI (Jenkins) and the test project. This layer just tells Functest what to execute according to the info in the DB. Regards, Jose -----Original Message----- From: [email protected] [mailto:[email protected]] Sent: Thursday, October 13, 2016 17:12 PM To: [email protected] Cc: Fatih Degirmenci; [email protected]; serena feng; yaohelan; Jose Lausuch Subject: CI evolution on impact on test projects Hi, I took the action point to describe the possible consequences of CI evolution on test projects. Please find attached a pdf detailing what I tried to explain yesterday :) I will be off next Thursday and will not be able to detail it before Barcelona but if you have comments/remarks I have also the odp and yed files if you want to edit anything @Kubi could you check Yardstick status info? If you prefer I can put that on the wiki, but as there are many things to agree first (to be sure we are sharing the same view), I prefer to initiate first exchanges by mail to converge to a first level of stability. /Morgan _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss _______________________________________________ test-wg mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/test-wg _______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
