Hi Karthik, what do you mean that the plugin is incompatible with https://review.openstack.org/#/c/114393/? you're mentioning a rebase issue - but the patch in question appears to cleanly apply to master.
Is your probably because patch #114393 does not have in its log some changes you need to accommodate test_lib changes? Are these changes you need already merged? In this case you might try to rebase the patch you're going to test on master before running devstack, which I think it's also what happens in the upstream gate. Salvatore On 26 August 2014 21:57, Karthik Natarajan <natar...@brocade.com> wrote: > Hi Edgar, > > We are also facing CI issues when the neutron patch set is not rebased > with latest changes. > For e.g. CI audit that you posted today ( > https://review.openstack.org/#/c/114393/) is not rebased with neutron > test_lib related changes. > We had refactored the Brocade Vyatta plugin unit tests to accommodate the > test_lib related changes. > But our plugin is not compatible with the patch you have posted. So CI is > failing. > > I had a discussion with Dane Leblanc on this. We also need to post the > SKIPPED status for such patch sets. > We will also experiment with Kevin's suggestion. > > Thanks, > Karthik > > -----Original Message----- > From: Dane Leblanc (leblancd) [mailto:lebla...@cisco.com] > Sent: Monday, August 25, 2014 10:02 AM > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > required to be run > > Edgar, Kyle: > > Kevin's suggestion should work for me (still hashing out the > implementation). I've added an item to the 3rd Party IRC agenda anyway to > discuss this corner case. > > Thanks! > Dane > > -----Original Message----- > From: Edgar Magana [mailto:edgar.mag...@workday.com] > Sent: Monday, August 25, 2014 12:44 PM > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > required to be run > > Dane, > > I will second Kyle's idea. Let's discuss this during today IRC meeting if > Kevin's suggestion does not work for you. > > Thanks, > > Edgar > > On 8/25/14, 10:08 AM, "Kyle Mestery" <mest...@mestery.com> wrote: > > >Dane, thanks for all the great work you're doing in the third-party CI > >area. It's great to see you working to share this knowledge with others > >as well! > > > >Did Kevin's idea work for you to move past this issue? If not, I > >suggest you put an item on the neutron meeting agenda today and we > >cover this there. You could put the item on the third-party meeting > >agenda as well. > > > >Thanks! > >Kyle > > > >On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd) > ><lebla...@cisco.com> wrote: > >> Hi Kevin: > >> > >> > >> > >> Thanks, this is a great idea! I may try just a slight variation of > >>this concept. Maybe your idea could be the recommended way to create > >>a 3rd party CI for plugins that are just being introduced and need to > >>limit the scope of testing to a small set of plugin-related commits > >>(or plugins blocked on a certain fix). > >> > >> > >> > >> Thanks, > >> Dane > >> > >> > >> > >> From: Kevin Benton [mailto:blak...@gmail.com] > >> Sent: Saturday, August 23, 2014 5:47 AM > >> > >> > >> To: OpenStack Development Mailing List (not for usage questions) > >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > >>required to be run > >> > >> > >> > >> Can you disable posting of results directly from your Jenkins/Zuul > >>setup and have a script that just checks the log file for special > >>markers to determine if the vote should be FAILED/PASSED/SKIPPED? > >>Another advantage of this approach is that it gives you an > >>opportunity to detect when a job just failed to setup due to > >>infrastructure reasons and trigger a recheck without ever first > >>posting a failure to gerrit. > >> > >> > >> > >> On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) > >> <lebla...@cisco.com> wrote: > >> > >> Thanks Edgar for updating the APIC status!!! > >> > >> Edgar and Kyle: *****PLEASE NOTE******!!!! I need your understanding > >>and advice on the following: > >> > >> We are still stuck with a problem stemming from a design limitation > >> of Jenkins that prevents us from being compliant with Neutron 3rd > >> Party CI requirements for our DFA CI. > >> > >> The issue is that Jenkins only allows our scripts to > >>(programmatically) return either Success or Fail. There is no option > >>to return "Aborted", "Not Tested", or "Skipped". > >> > >> Why does this matter? The DFA plugin is just being introduced, and > >>initial DFA-enabling change sets have not yet been merged. Therefore, > >>all other change sets will fail our Tempest tests, since they are not > >>DFA-enabled. > >> > >> Similarly, we were recently blocked in our APIC CI with a critical > >> bug, causing all change sets without this fix to fail on our APIC > testbed. > >> > >> In these cases, we would like to enter a "throttled" or "partially > >>blocked" > >> mode, where we would skip testing on change sets we know will fail, > >>and (in an ideal world) signal this shortcoming to Gerrit e.g. by > >>returning a "Skipped" status. Unfortunately, this option is not > >>available in Jenkins scripts, as Jenkins is currently designed. The > >>only options we have available is "Success" or all "Fail", which are > >>both misleading. We would also incorrectly report success or fail on > >>one of the following test > >> commits: > >> https://review.openstack.org/#/c/114393/ > >> https://review.openstack.org/#/c/40296/ > >> > >> I've brought this issue up on the openstack-infra IRC, and jeblair > >>confirmed the Jenkins limitation, but asked me to get consensus from > >>the Neutron community as to this being a problem/requirement. I've > >>also sent out an e-mail on the Neutron ML trying to start a > >>discussion on this problem (no traction). I plan on bringing this up > >>in the 3rd Party CI IRC on Monday, assuming there is time permitted > >>in the open discussion. > >> > >> I'm also investigating > >> > >> For the short term, I would like to propose the following: > >> * We bring this up on the 3rd Party CI IRC on Monday to get a > >>solution or workaround, if available. If a solution is available, > >>let's consider including that as a hint when we come up with CI > >>requirements for handling CIs bocked by some critical fix. > >> * I'm also looking into using a REST API to cancel a Jenkins job > >>programmatically. > >> * If no solution or workaround is available, we work with infra team > >>or with Jenkins team to create a solution. > >> * Until a solution is available, for plugins which are blocked by a > >>critical bug, we post a status/notes indicating the plugin's > >>situation on our 3rd party CI status wiki, e.g.: > >> > >> Vendor Plugin/Driver Name Contact Name > >> Status Notes > >> My Vendor Name My Plugin CI My Contact Person T > >> Throttled / Partially blocked / Awaiting Intial Commits > >> > >> The status/notes should be clear and understood by the Neutron team. > >>The > >> console logs for change sets where the tests were skipped should also > >>contain a message that all testing is being skipped for that commit. > >> > >> Note that when the DFA initial commits are merged, then this issue > >>would go away for the DFA CI. However, this problem will reappear > >>every time a blocking critical bug shows up for a 3rd party CI setup, > >>or a new plugin is introduced and the hardware-enabling commits are > >>not yet merged. (That is, until we have a solution for the Jenkins > >>limitation). > >> > >> Let me know what you think. > >> > >> > >> Thanks, > >> Dane > >> > >> -----Original Message----- > >> From: Edgar Magana [mailto:edgar.mag...@workday.com] > >> > >> Sent: Friday, August 22, 2014 1:57 PM > >> To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not > >>for usage questions) > >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > >>required to be run > >> > >> Sorry my bad but I just changed. > >> > >> Edgar > >> > >> On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> > >>wrote: > >> > >>>Edgar: > >>> > >>>I'm still seeing the comment "Results are not accurate. Needs > >>>clarification..." > >>> > >>>Dane > >>> > >>>-----Original Message----- > >>>From: Edgar Magana [mailto:edgar.mag...@workday.com] > >>>Sent: Thursday, August 21, 2014 2:58 PM > >>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not > >>>for usage questions) > >>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > >>>required to be run > >>> > >>>Dane, > >>> > >>>Wiki has been updated. > >>> > >>>Thanks, > >>> > >>>Edgar > >>> > >>>On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> > >>>wrote: > >>> > >>>>Edgar: > >>>> > >>>>The status on the wiki page says "Results are not accurate. Needs > >>>>clarification from Cisco". > >>>>Can you please tell me what we are missing? > >>>> > >>>>-Dane > >>>> > >>>>-----Original Message----- > >>>>From: Dane Leblanc (leblancd) > >>>>Sent: Tuesday, August 19, 2014 3:05 PM > >>>>To: 'Edgar Magana'; OpenStack Development Mailing List (not for > >>>>usage > >>>>questions) > >>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are > >>>>required to be run > >>>> > >>>>The APIC CI did run tests against that commit (after some queue > >>>>latency): > >>>> > >>>>http://128.107.233.28:8080/job/apic/1860/ > >>>>http://cisco-neutron-ci.cisco.com/logs/apic/1860/ > >>>> > >>>>But the review comments never showed up on Gerrit. This seems to be > >>>>an intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered > >>>>from this Jenkins/Gerrit server. Whenever we disable another one of > >>>>our other Jenkins jobs (in this case, we disabled DFA for some > >>>>rework), the review comments sometimes stop showing up on Gerrit. > >>>> > >>>>-----Original Message----- > >>>>From: Edgar Magana [mailto:edgar.mag...@workday.com] > >>>>Sent: Tuesday, August 19, 2014 1:33 PM > >>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not > >>>>for usage questions) > >>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > >>>>required to be run > >>>> > >>>>I was looking to one of the most recent Neutron commits: > >>>>https://review.openstack.org/#/c/115175/ > >>>> > >>>> > >>>>I could not find the APIC report. > >>>> > >>>>Edgar > >>>> > >>>>On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> > >>>>wrote: > >>>> > >>>>>From which commit is it missing? > >>>>>https://review.openstack.org/#/c/114629/ > >>>>>https://review.openstack.org/#/c/114393/ > >>>>> > >>>>>-----Original Message----- > >>>>>From: Edgar Magana [mailto:edgar.mag...@workday.com] > >>>>>Sent: Tuesday, August 19, 2014 12:28 PM > >>>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List > >>>>>(not for usage questions) > >>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > >>>>>required to be run > >>>>> > >>>>>Dane, > >>>>> > >>>>>Are you sure about it? > >>>>>I just went to this commit and I could not find the APIC tests. > >>>>> > >>>>>Thanks, > >>>>> > >>>>>Edgar > >>>>> > >>>>>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> > >>>>>wrote: > >>>>> > >>>>>>Edgar: > >>>>>> > >>>>>>The Cisco APIC should be reporting results for both APIC-related > >>>>>>and non-APIC related changes now. > >>>>>>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/). > >>>>>> > >>>>>>Will you be updating the wiki page? > >>>>>> > >>>>>>-Dane > >>>>>> > >>>>>>-----Original Message----- > >>>>>>From: Dane Leblanc (leblancd) > >>>>>>Sent: Friday, August 15, 2014 8:18 PM > >>>>>>To: OpenStack Development Mailing List (not for usage questions) > >>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests > >>>>>>are required to be run > >>>>>> > >>>>>>Also, you can add me as a contact person for the Cisco VPNaaS driver. > >>>>>> > >>>>>>-----Original Message----- > >>>>>>From: Dane Leblanc (leblancd) > >>>>>>Sent: Friday, August 15, 2014 8:14 PM > >>>>>>To: OpenStack Development Mailing List (not for usage questions) > >>>>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests > >>>>>>are required to be run > >>>>>> > >>>>>>Edgar: > >>>>>> > >>>>>>For the Notes for the Cisco APIC, can you change the comment > >>>>>>"results are fake" to something like "results are only valid for > >>>>>>APIC-related commits"? I think this more accurately represents our > >>>>>>current results (for reasons we chatted about on another thread). > >>>>>> > >>>>>>Thanks, > >>>>>>Dane > >>>>>> > >>>>>>-----Original Message----- > >>>>>>From: Edgar Magana [mailto:edgar.mag...@workday.com] > >>>>>>Sent: Friday, August 15, 2014 6:36 PM > >>>>>>To: OpenStack Development Mailing List (not for usage questions) > >>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests > >>>>>>are required to be run > >>>>>>Importance: High > >>>>>> > >>>>>>Team, > >>>>>> > >>>>>>I did a quick audit on the Neutron CI. Very sad results. Only few > >>>>>>plugins and drivers are running properly and testing all Neutron > >>>>>>commits. > >>>>>>I created a report here: > >>>>>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existi > >>>>>>ng > >>>>>>_ > >>>>>>P > >>>>>>l > >>>>>>ugi > >>>>>>n > >>>>>>_and_Drivers > >>>>>> > >>>>>> > >>>>>>We will discuss the actions to take on the next Neutron IRC meeting. > >>>>>>So please, reach me out to clarify what is the status of your CI. > >>>>>>I had two commits to quickly verify the CI reliability: > >>>>>> > >>>>>>https://review.openstack.org/#/c/114393/ > >>>>>> > >>>>>>https://review.openstack.org/#/c/40296/ > >>>>>> > >>>>>> > >>>>>>I would expect all plugins and drivers passing on the first one > >>>>>>and failing for the second but I got so many surprises. > >>>>>> > >>>>>>Neutron code quality and reliability is a top priority, if you > >>>>>>ignore this report that plugin/driver will be candidate to be > >>>>>>remove from Neutron tree. > >>>>>> > >>>>>>Cheers, > >>>>>> > >>>>>>Edgar > >>>>>> > >>>>>>P.s. I hate to be the inquisitor hereŠ but someone has to do the > >>>>>>dirty job! > >>>>>> > >>>>>> > >>>>>>On 8/14/14, 8:30 AM, "Kyle Mestery" <mest...@mestery.com> wrote: > >>>>>> > >>>>>>>Folks, I'm not sure if all CI accounts are running sufficient tests. > >>>>>>>Per the requirements wiki page here [1], everyone needs to be > >>>>>>>running more than just Tempest API tests, which I still see most > >>>>>>>neutron third-party CI setups doing. I'd like to ask everyone who > >>>>>>>operates a third-party CI account for Neutron to please look at > >>>>>>>the link below and make sure you are running appropriate tests. > >>>>>>>If you have questions, the weekly third-party meeting [2] is a > >>>>>>>great place to ask questions. > >>>>>>> > >>>>>>>Thanks, > >>>>>>>Kyle > >>>>>>> > >>>>>>>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting > >>>>>>>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > >>>>>>> > >>>>>>>_______________________________________________ > >>>>>>>OpenStack-dev mailing list > >>>>>>>OpenStack-dev@lists.openstack.org > >>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>>>> > >>>>>> > >>>>>>_______________________________________________ > >>>>>>OpenStack-dev mailing list > >>>>>>OpenStack-dev@lists.openstack.org > >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>>>> > >>>>>>_______________________________________________ > >>>>>>OpenStack-dev mailing list > >>>>>>OpenStack-dev@lists.openstack.org > >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>>> > >>>> > >>> > >> > >> > >> _______________________________________________ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >> > >> > >> > >> > >> -- > >> > >> Kevin Benton > >> > >> > >> _______________________________________________ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > > >_______________________________________________ > >OpenStack-dev mailing list > >OpenStack-dev@lists.openstack.org > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
