For the performance test, sometimes the criteria is set by the feature project.
For example, Doctor project sets a 1-second threshold for the fault handling process. If consuming more than 1s, it is considered as a failure. -- Yujun On Fri, Oct 21, 2016 at 3:28 AM Cooper, Trevor <[email protected]> wrote: > Thanks Morgan ... very good proposals to help move the test community > forward. > > - We need to make good choices about what tests are run ... to do this we > need a view of current test-coverage, test-tool capabilities and accurate > test-case descriptions. In the end it needs to be about selecting > test-cases ... not the project. > - Overlap and differences between test projects ... for example there are > many "latency tests" but they measure different things and will have very > different results depending on scenarios, tools, methodology, test > configurations, etc. > - There are no pass/fail criteria for performance tests ... but how do we > develop minimum thresholds for passing the tests ... this can't be > something arbitrary defined by each project. > > Sorry I can't be at Open Stack ... please add above points to the test > discussions. > > Trevor > > > > -----Original Message----- > From: [email protected] [mailto: > [email protected]] On Behalf Of Gaoliang (kubi) > Sent: Thursday, October 20, 2016 1:01 AM > To: Jose Lausuch <[email protected]>; [email protected]; > [email protected] > Cc: yaohelan <[email protected]>; serena feng <[email protected]>; > [email protected] > Subject: Re: [test-wg] CI evolution on impact on test projects > > Hi , > > Thanks for the great slides, CI evolution is definitely an importing thing > in Danube release. > > for yardstick in CI, the status is as below: > > definition of smoke test / weekly scope / test slicing > ->Yardstick have drafted the proposal of test slicing in Colorado > release, but the framework do not support it in c, we're planning to > support it in Danube > > Automatic selection of tests according to scenarios > ->Yardstick already support executing a scenario via a test-suite yaml in > Colorado. > Selection is not automatic, it need to be defined in a scenario test-suite > yaml. > > Test used for installer gating -> no > > scenario scoring -> yes, but yardstick only calculate the score by the > scenario fail/pass status, after Slicing of Yardstick Test cases, Yardstick > will include Tier score in Danube. > > I agree with Jose about a layer between CI (Jenkins) and the test project. > That would be helpful for all test projects. Let 's discuss details in > OpenStack Summit. > > Regards, > > Kubi > > -----邮件原件----- > 发件人: [email protected] [mailto: > [email protected]] 代表 Jose Lausuch > 发送时间: 2016年10月18日 16:01 > 收件人: [email protected]; [email protected] > 抄送: yaohelan; serena feng; [email protected] > 主题: Re: [opnfv-tech-discuss] CI evolution on impact on test projects > > Hi, > > Although it took a while to go through all your slides and understand them > all (the 2 big example slides helped a lot), I agree in general with what > you explain. There are some confusing assumptions like "if a case(weekly) > fails => still weekly loop but daily status" ... but the concept behind > sounds like a good approach to follow. > We should definitively move to different keywords to latest/daily/weekly. > Silver/Gold/Platinum sounds good enough, and congratulations for those > scenarios that get the Platinum badge from Functest because it comprises a > lot of long tests.. > > The promotions are performed in a logical way and everything is calculated > and stored in the central DB. Just some concerns come to my mind: > > - if a couple scenarios (say 3 or 4) are promoted to "Gold", it means > that they will be running all the tests (weekly loop) in CI (unless there > is a regression on a daily test if I understand correctly) ==> each > Funtest weekly run might take 3-5 hours (say 8 hours for > Deploy+Functest+Yardstick) ==> CI needs more time to complete an entire > installer loop going through all the scenarios. Today, in some cases, a > scenario takes ~2 days to run again in CI (with only daily Functest tests) > and some scenarios could be considered nowadays as "Gold".. Are we prepared > to run weekly tests on so many scenarios? Will we be able to do this the > weeks before a release? > > - This new logic is very CI-oriented. We also need to bypass it for the > case that an end user just wants to run all the tests the same way we do > now (i.e. functest testcase run all). So, the logic has to be implemented > in another layer on top, not in the test project framework as such. I can > imagine a layer between CI (Jenkins) and the test project. This layer just > tells Functest what to execute according to the info in the DB. > > Regards, > Jose > > > -----Original Message----- > From: [email protected] [mailto:[email protected]] > Sent: Thursday, October 13, 2016 17:12 PM > To: [email protected] > Cc: Fatih Degirmenci; [email protected]; serena feng; > yaohelan; Jose Lausuch > Subject: CI evolution on impact on test projects > > Hi, > > I took the action point to describe the possible consequences of CI > evolution on test projects. > > Please find attached a pdf detailing what I tried to explain yesterday :) > > I will be off next Thursday and will not be able to detail it before > Barcelona but if you have comments/remarks > > I have also the odp and yed files if you want to edit anything > > @Kubi could you check Yardstick status info? > > If you prefer I can put that on the wiki, but as there are many things to > agree first (to be sure we are sharing the same view), I prefer to initiate > first exchanges by mail to converge to a first level of stability. > > /Morgan > > > > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc pas etre diffuses, > exploites ou copies sans autorisation. Si vous avez recu ce message par > erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les > pieces jointes. Les messages electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; they should not be distributed, > used or copied without authorisation. > If you have received this email in error, please notify the sender and > delete this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. > > _______________________________________________ > opnfv-tech-discuss mailing list > [email protected] > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > _______________________________________________ > test-wg mailing list > [email protected] > https://lists.opnfv.org/mailman/listinfo/test-wg > _______________________________________________ > test-wg mailing list > [email protected] > https://lists.opnfv.org/mailman/listinfo/test-wg >
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
