Dane, I will second Kyle's idea. Let's discuss this during today IRC meeting if Kevin's suggestion does not work for you.
Thanks, Edgar On 8/25/14, 10:08 AM, "Kyle Mestery" <mest...@mestery.com> wrote: >Dane, thanks for all the great work you're doing in the third-party CI >area. It's great to see you working to share this knowledge with >others as well! > >Did Kevin's idea work for you to move past this issue? If not, I >suggest you put an item on the neutron meeting agenda today and we >cover this there. You could put the item on the third-party meeting >agenda as well. > >Thanks! >Kyle > >On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd) ><lebla...@cisco.com> wrote: >> Hi Kevin: >> >> >> >> Thanks, this is a great idea! I may try just a slight variation of this >> concept. Maybe your idea could be the recommended way to create a 3rd >>party >> CI for plugins that are just being introduced and need to limit the >>scope of >> testing to a small set of plugin-related commits (or plugins blocked on >>a >> certain fix). >> >> >> >> Thanks, >> Dane >> >> >> >> From: Kevin Benton [mailto:blak...@gmail.com] >> Sent: Saturday, August 23, 2014 5:47 AM >> >> >> To: OpenStack Development Mailing List (not for usage questions) >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required >> to be run >> >> >> >> Can you disable posting of results directly from your Jenkins/Zuul >>setup and >> have a script that just checks the log file for special markers to >>determine >> if the vote should be FAILED/PASSED/SKIPPED? Another advantage of this >> approach is that it gives you an opportunity to detect when a job just >> failed to setup due to infrastructure reasons and trigger a recheck >>without >> ever first posting a failure to gerrit. >> >> >> >> On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) >> <lebla...@cisco.com> wrote: >> >> Thanks Edgar for updating the APIC status!!! >> >> Edgar and Kyle: *****PLEASE NOTE******!!!! I need your understanding >>and >> advice on the following: >> >> We are still stuck with a problem stemming from a design limitation of >> Jenkins that prevents us from being compliant with Neutron 3rd Party CI >> requirements for our DFA CI. >> >> The issue is that Jenkins only allows our scripts to (programmatically) >> return either Success or Fail. There is no option to return "Aborted", >>"Not >> Tested", or "Skipped". >> >> Why does this matter? The DFA plugin is just being introduced, and >>initial >> DFA-enabling change sets have not yet been merged. Therefore, all other >> change sets will fail our Tempest tests, since they are not DFA-enabled. >> >> Similarly, we were recently blocked in our APIC CI with a critical bug, >> causing all change sets without this fix to fail on our APIC testbed. >> >> In these cases, we would like to enter a "throttled" or "partially >>blocked" >> mode, where we would skip testing on change sets we know will fail, and >>(in >> an ideal world) signal this shortcoming to Gerrit e.g. by returning a >> "Skipped" status. Unfortunately, this option is not available in Jenkins >> scripts, as Jenkins is currently designed. The only options we have >> available is "Success" or all "Fail", which are both misleading. We >>would >> also incorrectly report success or fail on one of the following test >> commits: >> https://review.openstack.org/#/c/114393/ >> https://review.openstack.org/#/c/40296/ >> >> I've brought this issue up on the openstack-infra IRC, and jeblair >>confirmed >> the Jenkins limitation, but asked me to get consensus from the Neutron >> community as to this being a problem/requirement. I've also sent out an >> e-mail on the Neutron ML trying to start a discussion on this problem >>(no >> traction). I plan on bringing this up in the 3rd Party CI IRC on Monday, >> assuming there is time permitted in the open discussion. >> >> I'm also investigating >> >> For the short term, I would like to propose the following: >> * We bring this up on the 3rd Party CI IRC on Monday to get a solution >>or >> workaround, if available. If a solution is available, let's consider >> including that as a hint when we come up with CI requirements for >>handling >> CIs bocked by some critical fix. >> * I'm also looking into using a REST API to cancel a Jenkins job >> programmatically. >> * If no solution or workaround is available, we work with infra team or >>with >> Jenkins team to create a solution. >> * Until a solution is available, for plugins which are blocked by a >>critical >> bug, we post a status/notes indicating the plugin's situation on our 3rd >> party CI status wiki, e.g.: >> >> Vendor Plugin/Driver Name Contact Name >> Status Notes >> My Vendor Name My Plugin CI My Contact Person T >> Throttled / Partially blocked / Awaiting Intial Commits >> >> The status/notes should be clear and understood by the Neutron team. >>The >> console logs for change sets where the tests were skipped should also >> contain a message that all testing is being skipped for that commit. >> >> Note that when the DFA initial commits are merged, then this issue >>would go >> away for the DFA CI. However, this problem will reappear every time a >> blocking critical bug shows up for a 3rd party CI setup, or a new >>plugin is >> introduced and the hardware-enabling commits are not yet merged. (That >>is, >> until we have a solution for the Jenkins limitation). >> >> Let me know what you think. >> >> >> Thanks, >> Dane >> >> -----Original Message----- >> From: Edgar Magana [mailto:edgar.mag...@workday.com] >> >> Sent: Friday, August 22, 2014 1:57 PM >> To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for >> usage questions) >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required >> to be run >> >> Sorry my bad but I just changed. >> >> Edgar >> >> On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> >>wrote: >> >>>Edgar: >>> >>>I'm still seeing the comment "Results are not accurate. Needs >>>clarification..." >>> >>>Dane >>> >>>-----Original Message----- >>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>Sent: Thursday, August 21, 2014 2:58 PM >>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >>>for usage questions) >>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>required to be run >>> >>>Dane, >>> >>>Wiki has been updated. >>> >>>Thanks, >>> >>>Edgar >>> >>>On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> >>>wrote: >>> >>>>Edgar: >>>> >>>>The status on the wiki page says "Results are not accurate. Needs >>>>clarification from Cisco". >>>>Can you please tell me what we are missing? >>>> >>>>-Dane >>>> >>>>-----Original Message----- >>>>From: Dane Leblanc (leblancd) >>>>Sent: Tuesday, August 19, 2014 3:05 PM >>>>To: 'Edgar Magana'; OpenStack Development Mailing List (not for usage >>>>questions) >>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >>>>required to be run >>>> >>>>The APIC CI did run tests against that commit (after some queue >>>>latency): >>>> >>>>http://128.107.233.28:8080/job/apic/1860/ >>>>http://cisco-neutron-ci.cisco.com/logs/apic/1860/ >>>> >>>>But the review comments never showed up on Gerrit. This seems to be an >>>>intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered from >>>>this Jenkins/Gerrit server. Whenever we disable another one of our >>>>other Jenkins jobs (in this case, we disabled DFA for some rework), >>>>the review comments sometimes stop showing up on Gerrit. >>>> >>>>-----Original Message----- >>>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>>Sent: Tuesday, August 19, 2014 1:33 PM >>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >>>>for usage questions) >>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>>required to be run >>>> >>>>I was looking to one of the most recent Neutron commits: >>>>https://review.openstack.org/#/c/115175/ >>>> >>>> >>>>I could not find the APIC report. >>>> >>>>Edgar >>>> >>>>On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> >>>>wrote: >>>> >>>>>From which commit is it missing? >>>>>https://review.openstack.org/#/c/114629/ >>>>>https://review.openstack.org/#/c/114393/ >>>>> >>>>>-----Original Message----- >>>>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>>>Sent: Tuesday, August 19, 2014 12:28 PM >>>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >>>>>for usage questions) >>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>>>required to be run >>>>> >>>>>Dane, >>>>> >>>>>Are you sure about it? >>>>>I just went to this commit and I could not find the APIC tests. >>>>> >>>>>Thanks, >>>>> >>>>>Edgar >>>>> >>>>>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> >>>>>wrote: >>>>> >>>>>>Edgar: >>>>>> >>>>>>The Cisco APIC should be reporting results for both APIC-related and >>>>>>non-APIC related changes now. >>>>>>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/). >>>>>> >>>>>>Will you be updating the wiki page? >>>>>> >>>>>>-Dane >>>>>> >>>>>>-----Original Message----- >>>>>>From: Dane Leblanc (leblancd) >>>>>>Sent: Friday, August 15, 2014 8:18 PM >>>>>>To: OpenStack Development Mailing List (not for usage questions) >>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>>>>required to be run >>>>>> >>>>>>Also, you can add me as a contact person for the Cisco VPNaaS driver. >>>>>> >>>>>>-----Original Message----- >>>>>>From: Dane Leblanc (leblancd) >>>>>>Sent: Friday, August 15, 2014 8:14 PM >>>>>>To: OpenStack Development Mailing List (not for usage questions) >>>>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >>>>>>required to be run >>>>>> >>>>>>Edgar: >>>>>> >>>>>>For the Notes for the Cisco APIC, can you change the comment >>>>>>"results are fake" to something like "results are only valid for >>>>>>APIC-related commits"? I think this more accurately represents our >>>>>>current results (for reasons we chatted about on another thread). >>>>>> >>>>>>Thanks, >>>>>>Dane >>>>>> >>>>>>-----Original Message----- >>>>>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>>>>Sent: Friday, August 15, 2014 6:36 PM >>>>>>To: OpenStack Development Mailing List (not for usage questions) >>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>>>>required to be run >>>>>>Importance: High >>>>>> >>>>>>Team, >>>>>> >>>>>>I did a quick audit on the Neutron CI. Very sad results. Only few >>>>>>plugins and drivers are running properly and testing all Neutron >>>>>>commits. >>>>>>I created a report here: >>>>>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing >>>>>>_ >>>>>>P >>>>>>l >>>>>>ugi >>>>>>n >>>>>>_and_Drivers >>>>>> >>>>>> >>>>>>We will discuss the actions to take on the next Neutron IRC meeting. >>>>>>So please, reach me out to clarify what is the status of your CI. >>>>>>I had two commits to quickly verify the CI reliability: >>>>>> >>>>>>https://review.openstack.org/#/c/114393/ >>>>>> >>>>>>https://review.openstack.org/#/c/40296/ >>>>>> >>>>>> >>>>>>I would expect all plugins and drivers passing on the first one and >>>>>>failing for the second but I got so many surprises. >>>>>> >>>>>>Neutron code quality and reliability is a top priority, if you >>>>>>ignore this report that plugin/driver will be candidate to be remove >>>>>>from Neutron tree. >>>>>> >>>>>>Cheers, >>>>>> >>>>>>Edgar >>>>>> >>>>>>P.s. I hate to be the inquisitor hereŠ but someone has to do the >>>>>>dirty job! >>>>>> >>>>>> >>>>>>On 8/14/14, 8:30 AM, "Kyle Mestery" <mest...@mestery.com> wrote: >>>>>> >>>>>>>Folks, I'm not sure if all CI accounts are running sufficient tests. >>>>>>>Per the requirements wiki page here [1], everyone needs to be >>>>>>>running more than just Tempest API tests, which I still see most >>>>>>>neutron third-party CI setups doing. I'd like to ask everyone who >>>>>>>operates a third-party CI account for Neutron to please look at the >>>>>>>link below and make sure you are running appropriate tests. If you >>>>>>>have questions, the weekly third-party meeting [2] is a great place >>>>>>>to ask questions. >>>>>>> >>>>>>>Thanks, >>>>>>>Kyle >>>>>>> >>>>>>>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>>>>>>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >>>>>>> >>>>>>>_______________________________________________ >>>>>>>OpenStack-dev mailing list >>>>>>>OpenStack-dev@lists.openstack.org >>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>> >>>>>>_______________________________________________ >>>>>>OpenStack-dev mailing list >>>>>>OpenStack-dev@lists.openstack.org >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>>_______________________________________________ >>>>>>OpenStack-dev mailing list >>>>>>OpenStack-dev@lists.openstack.org >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>> >>> >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> >> >> -- >> >> Kevin Benton >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >_______________________________________________ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
