Balázs Gibizer <balazs.gibi...@ericsson.com> writes: > Hi, > > I'm getting more and more confused how the zuul job hierarchy works or > is supposed to work.
Hi! First, you (or others) may or may not have seen this already -- some of it didn't exist when we first rolled out v3, and some of it has changed -- but here are the relevant bits of the documentation that should help explain what's going on. It helps to understand freezing: https://docs.openstack.org/infra/zuul/user/config.html#job and matching: https://docs.openstack.org/infra/zuul/user/config.html#matchers > First there was a bug in nova that some functional tests are not > triggered although the job (re-)definition in the nova part of the > project-config should not prevent it to run [1]. > > There we figured out that irrelevant-files parameter of the jobs are > not something that can be overriden during re-definition or through > parent-child relationship. The base job openstack-tox-functional has > an irrelevant-files attribute that lists '^doc/.*$' as a path to be > ignored [2]. In the other hand the nova part of the project-config > tries to make this ignore less broad by adding only '^doc/source/.*$' > . This does not work as we expected and the job did not run on changes > that only affected ./doc/notification_samples path. We are fixing it > by defining our own functional job in nova tree [4]. > > [1] https://bugs.launchpad.net/nova/+bug/1742962 > [2] > https://github.com/openstack-infra/openstack-zuul-jobs/blob/1823e3ea20e6dfaf37786a6ff79c56cb786bf12c/zuul.d/jobs.yaml#L380 > [3] > https://github.com/openstack-infra/project-config/blob/1145ab1293f5fa4d34c026856403c22b091e673c/zuul.d/projects.yaml#L10509 > [4] https://review.openstack.org/#/c/533210/ This is correct. The issue here is that the irrelevant-files definition on openstack-tox-functional is too broad. We need to be *extremely* careful applying matchers to jobs like that. Generally I think that irrelevant-files should be reserved for the project-pipeline invocations only. That's how they were effectively used in Zuul v2, after all. Essentially, when someone puts an irrelevant-files section on a job like that, they are saying "this job will never apply to these files, ever." That's clearly not correct in this case. So our solutions are to acknowledge that it's over-broad, and reduce or eliminate the list in [2] and expand it elsewhere (as in [3]). Or we can say "we were generally correct, but nova is extra special so it needs its own job". If that's the choice, then I think [4] is a fine solution. > Then I started looking into other jobs to see if we made similar > mistakes. I found two other examples in the nova related jobs where > redefining the irrelevant-files of a job caused problems. In these > examples nova tried to ignore more paths during the override than what > was originally ignored in the job definition but that did not work > [5][6]. > > [5] https://bugs.launchpad.net/nova/+bug/1745405 (temptest-full) As noted in that bug, the tempest-full job is invoked on nova via this stanza: https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10674-L10688 As expected, that did not match. There is a second invocation of tempest-full on nova here: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/zuul-legacy-project-templates.yaml#n126 That has no irrelevant-files matches, and so matches everything. If you drop the use of that template, it will work as expected. Or, if you can say with some certainty that nova's irrelevant-files set is not over-broad, you could move the irrelevant-files from nova's invocation into the template, or even the job, and drop nova's individual invocation. > [6] https://bugs.launchpad.net/nova/+bug/1745431 (neutron-grenade) The same template invokes this job as well. > So far the problem seemed to be consistent (i.e. override does not > work). But then I looked into neutron-grenade-multinode. That job is > defined in neutron tree (like neutron-grenade) but nova also refers to > it in nova section of the project-config with different > irrelevant-files than their original definition. So I assumed that > this will lead to similar problem than in case of neutron-grenade, but > it doesn't. > > The neutron-grenade-multinode original definition [1] does not try to > ignore the 'nova/tests' path but the nova side of the definition in > the project config does try to ignore that path [8]. Interestingly a > patch in nova that only changes under the path: nova/tests/ does not > trigger the job [9]. So in this case overriding the irrelevant-files > of a job works. (It seems that overriding neutron-tempest-linuxbridge > irrelevant-files works too). > > [7] > https://github.com/openstack/neutron/blob/7e3d6a18fb928bcd303a44c1736d0d6ca9c7f0ab/.zuul.yaml#L140-L159 > [8] > https://github.com/openstack-infra/project-config/blob/5ddbd62a46e17dd2fdee07bec32aa65e3b637ff3/zuul.d/projects.yaml#L10516-L10530 > [9] https://review.openstack.org/#/c/537936/ > > I don't see what is the difference between neutron-grenade and > neutron-grenade-multinode jobs definitions from this perspective but > it seems that the irrelevent-files attribute behaves inconsistently > in these two jobs. Could you please help me undestand how > irrelevant-files in overriden jobs supposed to work? These jobs only have the one invocation -- on the nova project -- and are not added via a template. Hopefully that explains the difference. Basically, the irrelevant-files on at least one project-pipeline invocation of a job have to match, as well as at least one definition of the job. So if both things have irrelevant-files, then it's effectively a union of the two. I used a tool to help verify some of the information in this message, especially the bugs [5] and [6]. You can ask Zuul to output debug information about its job selection if you're dealing with confusing situations like this. I went ahead and pushed a new patchset to your test change to demonstrate how: https://review.openstack.org/537936 When it finishes running all the tests (in a few hours), it should include in its report debug information about the decision-making process for the jobs it ran. It outputs similar information into the debug logs; so that we don't have to wait for it to see what it looks like here is that copy: http://paste.openstack.org/show/653729/ The relevant lines for [5] are: 2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job tempest-full branches: None source: openstack-infra/openstack-zuul-jobs/zuul.d/zuul-legacy-project-templates.yaml@master#126> matched <Change 0x7f2fed8883c8 537936,2> 2018-01-26 13:07:53,560 DEBUG zuul.layout: Pipeline variant <Job tempest-full branches: None source: openstack-infra/project-config/zuul.d/projects.yaml@master#10485> did not match <Change 0x7f2fed8883c8 537936,2> Note the project-file-branch-line-number references are especially helpful. -Jim __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
