maht commented on issue #2: Multibranch pipeline job example URL: https://github.com/apache/incubator-nuttx-testing/pull/2#issuecomment-582702122 > > Yes, regarding time concerns check option should take less time than full and can be shortened if needed by simply removing/commenting out some test cases. > > Why? If full does not work the build is broken and the PR should not come in. Well, in theory, that should be true. In practice, usually not, as far as I have seen in all my career. Simply put, running all the possible tests is impossible, because usually the number of possible tests to run is a number that can increase exponentially. Tomorrow I add a "on/off" flag, and I magically have doubled the number of possible tests: all the previous test running with "on", and all the previous tests running with "off". So, at some point, despite how big your capacity is, you have to think about what is minimum you want to try ('check') for the reasonable amount of resources you have. I am open to debate about what to include in "check" vs "full", or how to improve the way they are executed so more can be included or consume less resources or execution time. But saying that quality will be bad because we are not using the "full" set is meaningless. I can create an even "fuller" set of tests with performance, static code analysis, coverage, chaos testing, etc. Would that mean that we need to use "fuller" instead of "full"? Or should we avoid to create too much tests for fear of not including them? We need to set a line, or several, about what to test or not in each "test hierarchy" level (per pull request, nightly, releases, etc) so everything scales and is smooth, and we should adjust these lines over time. So let's focus on define which tests are more important/minimal for the pull request trigger. > Yet another great argument for github actions. I have never use GitHub actions, CircleCI, or any other CI service as much as Jenkins, but all of them are welcome if they help to check the pull request quality in a reliable way (i.e: without false alarms). I don't see why Jenkins has to be the only or preferred option (as long as Apache SF allow it, which seems the case). In the proposal [1] I wrote, I talk about Jenkins, but I also say: "The pipeline term is used here in a loose way. It means whatever arrange of tools and scripts to execute the CI. Specifically it does not restrict to the Jenkins CI pipelines.", so implementing the pipeline with other service is perfectly fine. > > Or make some changes to do out of tree build to speedup parallelize configs build time. > Yes. This will help. You do realize this is all in the PX4 CI and we are reinventing the wheel one spoke at a time here @davids5 I did not know the PX4 CI, but looks like very mature and curated. It would be wonderful to reuse as much from their know-how in our CI. I will take a look, but if you already know the internals and "tricks" used to improve the efficiency of the CI you are more than welcome. But this debate should be done in mail list or wiki and not in the comments of a pull request, I think... so, going back to the original matter of reviewing this pull request: does it worth to be merged despite it is not even near to be the optimal CI or the inertia/bad practices it will introduce will create a monster on the long term so it is preferred to work for something better? (I don't want to be the one creating that monster and be haunt forever :ghost:). [1] https://cwiki.apache.org/confluence/display/NUTTX/Continuous+integration+--+Miguel+Herranz
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
