Dane, thanks for all the great work you're doing in the third-party CI area. It's great to see you working to share this knowledge with others as well!
Did Kevin's idea work for you to move past this issue? If not, I suggest you put an item on the neutron meeting agenda today and we cover this there. You could put the item on the third-party meeting agenda as well. Thanks! Kyle On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd) <lebla...@cisco.com> wrote: > Hi Kevin: > > > > Thanks, this is a great idea! I may try just a slight variation of this > concept. Maybe your idea could be the recommended way to create a 3rd party > CI for plugins that are just being introduced and need to limit the scope of > testing to a small set of plugin-related commits (or plugins blocked on a > certain fix). > > > > Thanks, > Dane > > > > From: Kevin Benton [mailto:blak...@gmail.com] > Sent: Saturday, August 23, 2014 5:47 AM > > > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required > to be run > > > > Can you disable posting of results directly from your Jenkins/Zuul setup and > have a script that just checks the log file for special markers to determine > if the vote should be FAILED/PASSED/SKIPPED? Another advantage of this > approach is that it gives you an opportunity to detect when a job just > failed to setup due to infrastructure reasons and trigger a recheck without > ever first posting a failure to gerrit. > > > > On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) > <lebla...@cisco.com> wrote: > > Thanks Edgar for updating the APIC status!!! > > Edgar and Kyle: *****PLEASE NOTE******!!!! I need your understanding and > advice on the following: > > We are still stuck with a problem stemming from a design limitation of > Jenkins that prevents us from being compliant with Neutron 3rd Party CI > requirements for our DFA CI. > > The issue is that Jenkins only allows our scripts to (programmatically) > return either Success or Fail. There is no option to return "Aborted", "Not > Tested", or "Skipped". > > Why does this matter? The DFA plugin is just being introduced, and initial > DFA-enabling change sets have not yet been merged. Therefore, all other > change sets will fail our Tempest tests, since they are not DFA-enabled. > > Similarly, we were recently blocked in our APIC CI with a critical bug, > causing all change sets without this fix to fail on our APIC testbed. > > In these cases, we would like to enter a "throttled" or "partially blocked" > mode, where we would skip testing on change sets we know will fail, and (in > an ideal world) signal this shortcoming to Gerrit e.g. by returning a > "Skipped" status. Unfortunately, this option is not available in Jenkins > scripts, as Jenkins is currently designed. The only options we have > available is "Success" or all "Fail", which are both misleading. We would > also incorrectly report success or fail on one of the following test > commits: > https://review.openstack.org/#/c/114393/ > https://review.openstack.org/#/c/40296/ > > I've brought this issue up on the openstack-infra IRC, and jeblair confirmed > the Jenkins limitation, but asked me to get consensus from the Neutron > community as to this being a problem/requirement. I've also sent out an > e-mail on the Neutron ML trying to start a discussion on this problem (no > traction). I plan on bringing this up in the 3rd Party CI IRC on Monday, > assuming there is time permitted in the open discussion. > > I'm also investigating > > For the short term, I would like to propose the following: > * We bring this up on the 3rd Party CI IRC on Monday to get a solution or > workaround, if available. If a solution is available, let's consider > including that as a hint when we come up with CI requirements for handling > CIs bocked by some critical fix. > * I'm also looking into using a REST API to cancel a Jenkins job > programmatically. > * If no solution or workaround is available, we work with infra team or with > Jenkins team to create a solution. > * Until a solution is available, for plugins which are blocked by a critical > bug, we post a status/notes indicating the plugin's situation on our 3rd > party CI status wiki, e.g.: > > Vendor Plugin/Driver Name Contact Name > Status Notes > My Vendor Name My Plugin CI My Contact Person T > Throttled / Partially blocked / Awaiting Intial Commits > > The status/notes should be clear and understood by the Neutron team. The > console logs for change sets where the tests were skipped should also > contain a message that all testing is being skipped for that commit. > > Note that when the DFA initial commits are merged, then this issue would go > away for the DFA CI. However, this problem will reappear every time a > blocking critical bug shows up for a 3rd party CI setup, or a new plugin is > introduced and the hardware-enabling commits are not yet merged. (That is, > until we have a solution for the Jenkins limitation). > > Let me know what you think. > > > Thanks, > Dane > > -----Original Message----- > From: Edgar Magana [mailto:edgar.mag...@workday.com] > > Sent: Friday, August 22, 2014 1:57 PM > To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for > usage questions) > Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required > to be run > > Sorry my bad but I just changed. > > Edgar > > On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> wrote: > >>Edgar: >> >>I'm still seeing the comment "Results are not accurate. Needs >>clarification..." >> >>Dane >> >>-----Original Message----- >>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>Sent: Thursday, August 21, 2014 2:58 PM >>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >>for usage questions) >>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >>Dane, >> >>Wiki has been updated. >> >>Thanks, >> >>Edgar >> >>On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> wrote: >> >>>Edgar: >>> >>>The status on the wiki page says "Results are not accurate. Needs >>>clarification from Cisco". >>>Can you please tell me what we are missing? >>> >>>-Dane >>> >>>-----Original Message----- >>>From: Dane Leblanc (leblancd) >>>Sent: Tuesday, August 19, 2014 3:05 PM >>>To: 'Edgar Magana'; OpenStack Development Mailing List (not for usage >>>questions) >>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >>>required to be run >>> >>>The APIC CI did run tests against that commit (after some queue latency): >>> >>>http://128.107.233.28:8080/job/apic/1860/ >>>http://cisco-neutron-ci.cisco.com/logs/apic/1860/ >>> >>>But the review comments never showed up on Gerrit. This seems to be an >>>intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered from >>>this Jenkins/Gerrit server. Whenever we disable another one of our >>>other Jenkins jobs (in this case, we disabled DFA for some rework), >>>the review comments sometimes stop showing up on Gerrit. >>> >>>-----Original Message----- >>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>Sent: Tuesday, August 19, 2014 1:33 PM >>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >>>for usage questions) >>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>required to be run >>> >>>I was looking to one of the most recent Neutron commits: >>>https://review.openstack.org/#/c/115175/ >>> >>> >>>I could not find the APIC report. >>> >>>Edgar >>> >>>On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> >>>wrote: >>> >>>>From which commit is it missing? >>>>https://review.openstack.org/#/c/114629/ >>>>https://review.openstack.org/#/c/114393/ >>>> >>>>-----Original Message----- >>>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>>Sent: Tuesday, August 19, 2014 12:28 PM >>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >>>>for usage questions) >>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>>required to be run >>>> >>>>Dane, >>>> >>>>Are you sure about it? >>>>I just went to this commit and I could not find the APIC tests. >>>> >>>>Thanks, >>>> >>>>Edgar >>>> >>>>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> >>>>wrote: >>>> >>>>>Edgar: >>>>> >>>>>The Cisco APIC should be reporting results for both APIC-related and >>>>>non-APIC related changes now. >>>>>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/). >>>>> >>>>>Will you be updating the wiki page? >>>>> >>>>>-Dane >>>>> >>>>>-----Original Message----- >>>>>From: Dane Leblanc (leblancd) >>>>>Sent: Friday, August 15, 2014 8:18 PM >>>>>To: OpenStack Development Mailing List (not for usage questions) >>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>>>required to be run >>>>> >>>>>Also, you can add me as a contact person for the Cisco VPNaaS driver. >>>>> >>>>>-----Original Message----- >>>>>From: Dane Leblanc (leblancd) >>>>>Sent: Friday, August 15, 2014 8:14 PM >>>>>To: OpenStack Development Mailing List (not for usage questions) >>>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >>>>>required to be run >>>>> >>>>>Edgar: >>>>> >>>>>For the Notes for the Cisco APIC, can you change the comment >>>>>"results are fake" to something like "results are only valid for >>>>>APIC-related commits"? I think this more accurately represents our >>>>>current results (for reasons we chatted about on another thread). >>>>> >>>>>Thanks, >>>>>Dane >>>>> >>>>>-----Original Message----- >>>>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>>>Sent: Friday, August 15, 2014 6:36 PM >>>>>To: OpenStack Development Mailing List (not for usage questions) >>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>>>required to be run >>>>>Importance: High >>>>> >>>>>Team, >>>>> >>>>>I did a quick audit on the Neutron CI. Very sad results. Only few >>>>>plugins and drivers are running properly and testing all Neutron >>>>>commits. >>>>>I created a report here: >>>>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing >>>>>_ >>>>>P >>>>>l >>>>>ugi >>>>>n >>>>>_and_Drivers >>>>> >>>>> >>>>>We will discuss the actions to take on the next Neutron IRC meeting. >>>>>So please, reach me out to clarify what is the status of your CI. >>>>>I had two commits to quickly verify the CI reliability: >>>>> >>>>>https://review.openstack.org/#/c/114393/ >>>>> >>>>>https://review.openstack.org/#/c/40296/ >>>>> >>>>> >>>>>I would expect all plugins and drivers passing on the first one and >>>>>failing for the second but I got so many surprises. >>>>> >>>>>Neutron code quality and reliability is a top priority, if you >>>>>ignore this report that plugin/driver will be candidate to be remove >>>>>from Neutron tree. >>>>> >>>>>Cheers, >>>>> >>>>>Edgar >>>>> >>>>>P.s. I hate to be the inquisitor hereŠ but someone has to do the >>>>>dirty job! >>>>> >>>>> >>>>>On 8/14/14, 8:30 AM, "Kyle Mestery" <mest...@mestery.com> wrote: >>>>> >>>>>>Folks, I'm not sure if all CI accounts are running sufficient tests. >>>>>>Per the requirements wiki page here [1], everyone needs to be >>>>>>running more than just Tempest API tests, which I still see most >>>>>>neutron third-party CI setups doing. I'd like to ask everyone who >>>>>>operates a third-party CI account for Neutron to please look at the >>>>>>link below and make sure you are running appropriate tests. If you >>>>>>have questions, the weekly third-party meeting [2] is a great place >>>>>>to ask questions. >>>>>> >>>>>>Thanks, >>>>>>Kyle >>>>>> >>>>>>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>>>>>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >>>>>> >>>>>>_______________________________________________ >>>>>>OpenStack-dev mailing list >>>>>>OpenStack-dev@lists.openstack.org >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>>>_______________________________________________ >>>>>OpenStack-dev mailing list >>>>>OpenStack-dev@lists.openstack.org >>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>>_______________________________________________ >>>>>OpenStack-dev mailing list >>>>>OpenStack-dev@lists.openstack.org >>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> >> > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > -- > > Kevin Benton > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
