Hi Edgar, We are also facing CI issues when the neutron patch set is not rebased with latest changes. For e.g. CI audit that you posted today (https://review.openstack.org/#/c/114393/) is not rebased with neutron test_lib related changes. We had refactored the Brocade Vyatta plugin unit tests to accommodate the test_lib related changes. But our plugin is not compatible with the patch you have posted. So CI is failing.
I had a discussion with Dane Leblanc on this. We also need to post the SKIPPED status for such patch sets. We will also experiment with Kevin's suggestion. Thanks, Karthik -----Original Message----- From: Dane Leblanc (leblancd) [mailto:lebla...@cisco.com] Sent: Monday, August 25, 2014 10:02 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Edgar, Kyle: Kevin's suggestion should work for me (still hashing out the implementation). I've added an item to the 3rd Party IRC agenda anyway to discuss this corner case. Thanks! Dane -----Original Message----- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Monday, August 25, 2014 12:44 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Dane, I will second Kyle's idea. Let's discuss this during today IRC meeting if Kevin's suggestion does not work for you. Thanks, Edgar On 8/25/14, 10:08 AM, "Kyle Mestery" <mest...@mestery.com> wrote: >Dane, thanks for all the great work you're doing in the third-party CI >area. It's great to see you working to share this knowledge with others >as well! > >Did Kevin's idea work for you to move past this issue? If not, I >suggest you put an item on the neutron meeting agenda today and we >cover this there. You could put the item on the third-party meeting >agenda as well. > >Thanks! >Kyle > >On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd) ><lebla...@cisco.com> wrote: >> Hi Kevin: >> >> >> >> Thanks, this is a great idea! I may try just a slight variation of >>this concept. Maybe your idea could be the recommended way to create >>a 3rd party CI for plugins that are just being introduced and need to >>limit the scope of testing to a small set of plugin-related commits >>(or plugins blocked on a certain fix). >> >> >> >> Thanks, >> Dane >> >> >> >> From: Kevin Benton [mailto:blak...@gmail.com] >> Sent: Saturday, August 23, 2014 5:47 AM >> >> >> To: OpenStack Development Mailing List (not for usage questions) >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >> >> >> Can you disable posting of results directly from your Jenkins/Zuul >>setup and have a script that just checks the log file for special >>markers to determine if the vote should be FAILED/PASSED/SKIPPED? >>Another advantage of this approach is that it gives you an >>opportunity to detect when a job just failed to setup due to >>infrastructure reasons and trigger a recheck without ever first >>posting a failure to gerrit. >> >> >> >> On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) >> <lebla...@cisco.com> wrote: >> >> Thanks Edgar for updating the APIC status!!! >> >> Edgar and Kyle: *****PLEASE NOTE******!!!! I need your understanding >>and advice on the following: >> >> We are still stuck with a problem stemming from a design limitation >> of Jenkins that prevents us from being compliant with Neutron 3rd >> Party CI requirements for our DFA CI. >> >> The issue is that Jenkins only allows our scripts to >>(programmatically) return either Success or Fail. There is no option >>to return "Aborted", "Not Tested", or "Skipped". >> >> Why does this matter? The DFA plugin is just being introduced, and >>initial DFA-enabling change sets have not yet been merged. Therefore, >>all other change sets will fail our Tempest tests, since they are not >>DFA-enabled. >> >> Similarly, we were recently blocked in our APIC CI with a critical >> bug, causing all change sets without this fix to fail on our APIC testbed. >> >> In these cases, we would like to enter a "throttled" or "partially >>blocked" >> mode, where we would skip testing on change sets we know will fail, >>and (in an ideal world) signal this shortcoming to Gerrit e.g. by >>returning a "Skipped" status. Unfortunately, this option is not >>available in Jenkins scripts, as Jenkins is currently designed. The >>only options we have available is "Success" or all "Fail", which are >>both misleading. We would also incorrectly report success or fail on >>one of the following test >> commits: >> https://review.openstack.org/#/c/114393/ >> https://review.openstack.org/#/c/40296/ >> >> I've brought this issue up on the openstack-infra IRC, and jeblair >>confirmed the Jenkins limitation, but asked me to get consensus from >>the Neutron community as to this being a problem/requirement. I've >>also sent out an e-mail on the Neutron ML trying to start a >>discussion on this problem (no traction). I plan on bringing this up >>in the 3rd Party CI IRC on Monday, assuming there is time permitted >>in the open discussion. >> >> I'm also investigating >> >> For the short term, I would like to propose the following: >> * We bring this up on the 3rd Party CI IRC on Monday to get a >>solution or workaround, if available. If a solution is available, >>let's consider including that as a hint when we come up with CI >>requirements for handling CIs bocked by some critical fix. >> * I'm also looking into using a REST API to cancel a Jenkins job >>programmatically. >> * If no solution or workaround is available, we work with infra team >>or with Jenkins team to create a solution. >> * Until a solution is available, for plugins which are blocked by a >>critical bug, we post a status/notes indicating the plugin's >>situation on our 3rd party CI status wiki, e.g.: >> >> Vendor Plugin/Driver Name Contact Name >> Status Notes >> My Vendor Name My Plugin CI My Contact Person T >> Throttled / Partially blocked / Awaiting Intial Commits >> >> The status/notes should be clear and understood by the Neutron team. >>The >> console logs for change sets where the tests were skipped should also >>contain a message that all testing is being skipped for that commit. >> >> Note that when the DFA initial commits are merged, then this issue >>would go away for the DFA CI. However, this problem will reappear >>every time a blocking critical bug shows up for a 3rd party CI setup, >>or a new plugin is introduced and the hardware-enabling commits are >>not yet merged. (That is, until we have a solution for the Jenkins >>limitation). >> >> Let me know what you think. >> >> >> Thanks, >> Dane >> >> -----Original Message----- >> From: Edgar Magana [mailto:edgar.mag...@workday.com] >> >> Sent: Friday, August 22, 2014 1:57 PM >> To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >>for usage questions) >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >> Sorry my bad but I just changed. >> >> Edgar >> >> On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> >>wrote: >> >>>Edgar: >>> >>>I'm still seeing the comment "Results are not accurate. Needs >>>clarification..." >>> >>>Dane >>> >>>-----Original Message----- >>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>Sent: Thursday, August 21, 2014 2:58 PM >>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >>>for usage questions) >>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>required to be run >>> >>>Dane, >>> >>>Wiki has been updated. >>> >>>Thanks, >>> >>>Edgar >>> >>>On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> >>>wrote: >>> >>>>Edgar: >>>> >>>>The status on the wiki page says "Results are not accurate. Needs >>>>clarification from Cisco". >>>>Can you please tell me what we are missing? >>>> >>>>-Dane >>>> >>>>-----Original Message----- >>>>From: Dane Leblanc (leblancd) >>>>Sent: Tuesday, August 19, 2014 3:05 PM >>>>To: 'Edgar Magana'; OpenStack Development Mailing List (not for >>>>usage >>>>questions) >>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >>>>required to be run >>>> >>>>The APIC CI did run tests against that commit (after some queue >>>>latency): >>>> >>>>http://128.107.233.28:8080/job/apic/1860/ >>>>http://cisco-neutron-ci.cisco.com/logs/apic/1860/ >>>> >>>>But the review comments never showed up on Gerrit. This seems to be >>>>an intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered >>>>from this Jenkins/Gerrit server. Whenever we disable another one of >>>>our other Jenkins jobs (in this case, we disabled DFA for some >>>>rework), the review comments sometimes stop showing up on Gerrit. >>>> >>>>-----Original Message----- >>>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>>Sent: Tuesday, August 19, 2014 1:33 PM >>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >>>>for usage questions) >>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>>required to be run >>>> >>>>I was looking to one of the most recent Neutron commits: >>>>https://review.openstack.org/#/c/115175/ >>>> >>>> >>>>I could not find the APIC report. >>>> >>>>Edgar >>>> >>>>On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> >>>>wrote: >>>> >>>>>From which commit is it missing? >>>>>https://review.openstack.org/#/c/114629/ >>>>>https://review.openstack.org/#/c/114393/ >>>>> >>>>>-----Original Message----- >>>>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>>>Sent: Tuesday, August 19, 2014 12:28 PM >>>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List >>>>>(not for usage questions) >>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>>>required to be run >>>>> >>>>>Dane, >>>>> >>>>>Are you sure about it? >>>>>I just went to this commit and I could not find the APIC tests. >>>>> >>>>>Thanks, >>>>> >>>>>Edgar >>>>> >>>>>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> >>>>>wrote: >>>>> >>>>>>Edgar: >>>>>> >>>>>>The Cisco APIC should be reporting results for both APIC-related >>>>>>and non-APIC related changes now. >>>>>>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/). >>>>>> >>>>>>Will you be updating the wiki page? >>>>>> >>>>>>-Dane >>>>>> >>>>>>-----Original Message----- >>>>>>From: Dane Leblanc (leblancd) >>>>>>Sent: Friday, August 15, 2014 8:18 PM >>>>>>To: OpenStack Development Mailing List (not for usage questions) >>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests >>>>>>are required to be run >>>>>> >>>>>>Also, you can add me as a contact person for the Cisco VPNaaS driver. >>>>>> >>>>>>-----Original Message----- >>>>>>From: Dane Leblanc (leblancd) >>>>>>Sent: Friday, August 15, 2014 8:14 PM >>>>>>To: OpenStack Development Mailing List (not for usage questions) >>>>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests >>>>>>are required to be run >>>>>> >>>>>>Edgar: >>>>>> >>>>>>For the Notes for the Cisco APIC, can you change the comment >>>>>>"results are fake" to something like "results are only valid for >>>>>>APIC-related commits"? I think this more accurately represents our >>>>>>current results (for reasons we chatted about on another thread). >>>>>> >>>>>>Thanks, >>>>>>Dane >>>>>> >>>>>>-----Original Message----- >>>>>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>>>>Sent: Friday, August 15, 2014 6:36 PM >>>>>>To: OpenStack Development Mailing List (not for usage questions) >>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests >>>>>>are required to be run >>>>>>Importance: High >>>>>> >>>>>>Team, >>>>>> >>>>>>I did a quick audit on the Neutron CI. Very sad results. Only few >>>>>>plugins and drivers are running properly and testing all Neutron >>>>>>commits. >>>>>>I created a report here: >>>>>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existi >>>>>>ng >>>>>>_ >>>>>>P >>>>>>l >>>>>>ugi >>>>>>n >>>>>>_and_Drivers >>>>>> >>>>>> >>>>>>We will discuss the actions to take on the next Neutron IRC meeting. >>>>>>So please, reach me out to clarify what is the status of your CI. >>>>>>I had two commits to quickly verify the CI reliability: >>>>>> >>>>>>https://review.openstack.org/#/c/114393/ >>>>>> >>>>>>https://review.openstack.org/#/c/40296/ >>>>>> >>>>>> >>>>>>I would expect all plugins and drivers passing on the first one >>>>>>and failing for the second but I got so many surprises. >>>>>> >>>>>>Neutron code quality and reliability is a top priority, if you >>>>>>ignore this report that plugin/driver will be candidate to be >>>>>>remove from Neutron tree. >>>>>> >>>>>>Cheers, >>>>>> >>>>>>Edgar >>>>>> >>>>>>P.s. I hate to be the inquisitor hereŠ but someone has to do the >>>>>>dirty job! >>>>>> >>>>>> >>>>>>On 8/14/14, 8:30 AM, "Kyle Mestery" <mest...@mestery.com> wrote: >>>>>> >>>>>>>Folks, I'm not sure if all CI accounts are running sufficient tests. >>>>>>>Per the requirements wiki page here [1], everyone needs to be >>>>>>>running more than just Tempest API tests, which I still see most >>>>>>>neutron third-party CI setups doing. I'd like to ask everyone who >>>>>>>operates a third-party CI account for Neutron to please look at >>>>>>>the link below and make sure you are running appropriate tests. >>>>>>>If you have questions, the weekly third-party meeting [2] is a >>>>>>>great place to ask questions. >>>>>>> >>>>>>>Thanks, >>>>>>>Kyle >>>>>>> >>>>>>>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>>>>>>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >>>>>>> >>>>>>>_______________________________________________ >>>>>>>OpenStack-dev mailing list >>>>>>>OpenStack-dev@lists.openstack.org >>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>> >>>>>>_______________________________________________ >>>>>>OpenStack-dev mailing list >>>>>>OpenStack-dev@lists.openstack.org >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>>_______________________________________________ >>>>>>OpenStack-dev mailing list >>>>>>OpenStack-dev@lists.openstack.org >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>> >>> >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> >> >> -- >> >> Kevin Benton >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >_______________________________________________ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
