Hi, Although it took a while to go through all your slides and understand them all (the 2 big example slides helped a lot), I agree in general with what you explain. There are some confusing assumptions like "if a case(weekly) fails => still weekly loop but daily status" ... but the concept behind sounds like a good approach to follow. We should definitively move to different keywords to latest/daily/weekly. Silver/Gold/Platinum sounds good enough, and congratulations for those scenarios that get the Platinum badge from Functest because it comprises a lot of long tests..
The promotions are performed in a logical way and everything is calculated and stored in the central DB. Just some concerns come to my mind: - if a couple scenarios (say 3 or 4) are promoted to "Gold", it means that they will be running all the tests (weekly loop) in CI (unless there is a regression on a daily test if I understand correctly) ==> each Funtest weekly run might take 3-5 hours (say 8 hours for Deploy+Functest+Yardstick) ==> CI needs more time to complete an entire installer loop going through all the scenarios. Today, in some cases, a scenario takes ~2 days to run again in CI (with only daily Functest tests) and some scenarios could be considered nowadays as "Gold".. Are we prepared to run weekly tests on so many scenarios? Will we be able to do this the weeks before a release? - This new logic is very CI-oriented. We also need to bypass it for the case that an end user just wants to run all the tests the same way we do now (i.e. functest testcase run all). So, the logic has to be implemented in another layer on top, not in the test project framework as such. I can imagine a layer between CI (Jenkins) and the test project. This layer just tells Functest what to execute according to the info in the DB. Regards, Jose -----Original Message----- From: [email protected] [mailto:[email protected]] Sent: Thursday, October 13, 2016 17:12 PM To: [email protected] Cc: Fatih Degirmenci; [email protected]; serena feng; yaohelan; Jose Lausuch Subject: CI evolution on impact on test projects Hi, I took the action point to describe the possible consequences of CI evolution on test projects. Please find attached a pdf detailing what I tried to explain yesterday :) I will be off next Thursday and will not be able to detail it before Barcelona but if you have comments/remarks I have also the odp and yed files if you want to edit anything @Kubi could you check Yardstick status info? If you prefer I can put that on the wiki, but as there are many things to agree first (to be sure we are sharing the same view), I prefer to initiate first exchanges by mail to converge to a first level of stability. /Morgan _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
