Hi Ludo, all, On Sat, 31 Aug 2024 at 15:03, Ludovic Courtès <l...@gnu.org> wrote:
> To reduce world rebuilds, perhaps we’ll sometimes create “merge trains”, > whereby we’ll merge, say, the branch upgrading CMake and that ungrafting > ibus on top of ‘core-packages-team’, and then merge this combination in > ‘master’. I do not see how the world rebuild would be reduced. Correct me if my understanding is misleading, the branch ’core-updates’ had been extended to more than core packages because the project had (has?) not enough CPU power for continuously building any change with a massive impact. Therefore, this ’core-updates’ branch was built only every X months in order to reduce the workload. As you perfectly know, all these grouped changes is a mess and it requires a lot of effort to stabilize. Hence, a very boring task… i.e., an arbitrary X. ;-) For sure, that’s a good idea to have a core-packages-team that focuses only on core packages – the initial aim. :-) However, it does not address the issue with changes that have a massive impact. This new core-packages-team just becomes one of other branches that will also contain packages triggering world rebuilds. And the question is how do we manage that? > The key being: these branches will have been developed and > tested independently of one another by dedicated teams, and the merge > train will be a mere formality. In this picture of “merge train”, the CI workload and world rebuilds will increase, no? Consider the two teams: science and this new core-packages. Then science takes care of openblas (6000+ dependent packages) and core-packages of grep (6000+ dependent packages). When science team wants to merge to master, they ask for a rebuild. When core-packages team wants to merge to master, they ask for a rebuild. Therefore, we have two world rebuilds. Or the both teams need to synchronize and thus the branches are indeed developed independently but not tested independently. Are they? Well, my understanding of “merge train” is an automatic synchronization workflow: science team does not ask for a rebuild but ask for a merge and core-packages team idem – so they cannot evaluate their impact and what needs to be fixed by their changes – then both merges are the two wagons of the train. Well, it’s not clear for me what CI builds: first one wagon then the two wagons? Or first the two wagons and then what? Aside, the core-packages or science merges might have an impact to packages tracked by other teams. Other said, fixes unrelated to their scope. Somehow, it requires that other teams also follow – it means a kind of synchronization. That’s why core-updates became the big catch-all and complex mess. That’s just a poor man way to synchronize: first it minimizes the number of world rebuilds because they are more or less manually triggered, and second it eases to have fixes unrelated to core changes. For sure, I agree that having more branches will help to reduce the complexity of merging core-updates-like changes. However, it’s not clear how it will work in practise. Especially, the number of world rebuilds will increase, IMHO. Well, all in all, I understand the “merge train” story with “regular” branches. But I do not see how it would work for different branches that trigger world rebuilds. Could you detail a bit your perspective with “merge train”? Or the workflow for merging branches that contains world rebuild changes. I know about [1] but I am lacking imagination for applying it to our situation. Because, if we have 3 merge requests (A, B and C), the “merge train” workflow implies the pipeline is run 3 times: with A on top of master, with A+B (B on the top of A on the top of master), with A+B+C (C on the top of B on the top of A on the top of master). If A+B fails then B is removed from the train and the pipeline runs A+C. And you can also consider if A fails then B and B+C is built. The “merge train” workflow makes sens when the number of changes in A, B and C is high and thus it helps to keep the merges under control. However, one implicit assumption is a low-cost pipeline, IMHO. Bah the question how to merge different branches containing world rebuilds without the big catch-all old core-updates branch is not addressed under the constraint of reducing as much as possible the number of world rebuilds, for what my opinion is worth. Cheers, simon 1: https://docs.gitlab.com/ee/ci/pipelines/merge_trains.html PS: My understanding of Gitlab Merge Train selling point is that several pipelines are run in parallel. How to waste power for increasing the throughput?
