Hi Ludo, all On Fri, 06 Sep 2024 at 11:11, Ludovic Courtès <l...@gnu.org> wrote:
> In the end, perhaps we’ll have to negotiate on a case-by-case basis. > The important thing to me is: independent testing as much as possible, > and well-defined responsibilities and scope for the people/teams > engaging in such changes. I agree. My main concern is about the potential (excessive) unnecessary wasted rebuilds. We need a policy that clarifies how to start large rebuilds, which implies cross the final line with a merge. Let take the example from this another message [1] as one case. On Fri, 06 Sep 2024 at 11:01, Ludovic Courtès <l...@gnu.org> wrote: > • Attach to a “merge train”. For instance, assume there’s a branch > changing ‘gsl’: this is totally unrelated to ‘ffmpeg’ but it also > triggers a lot of rebuilds. We could tack that second branch on top > of the known-good ‘ffmpeg’ branch, and, once it’s all good, merge > that “train” into ‘master’. > > (To be clear, the term “merge train” originates from GitLab-CI and > similar CI tool, which use it as a merge scheduling strategy: > <https://docs.gitlab.com/ee/ci/pipelines/merge_trains.html>. GitLab-CI > can create merge trains automatically I believe, but in our case we’d do > that manually, at least for now.) If we consider this specific case, the “merge train” workflow means: a) the branch changing “gsl” is built, so 1473 rebuilds. b) the branch changing “gsl” and “ffmpeg” is also built in parallel, so 521 rebuilds. The “merge train” workflow also reads: i) the branch changing “ffmpeg” is built, so 521 rebuilds. ii) the branch changing “gsl” and “ffmpeg” is also built in parallel, so 1473 rebuilds including all the 521 rebuilds of i). Therefore, for each scenario, 521 builds are “wasted“, compared to the optimal 1473 in this case. I do not think it’s what you have in mind when speaking about “merge train”. Is it? Maybe it points that “merge train” as described by Gitlab is not what we need. The “merge train” workflow runs *in parallel* the builds of several branches and I am not convinced this is wanted for heavy branches as it might happens. Therefore in order to avoid some confusion, maybe we could avoid the name “merge train” and use another name for the strategy we are discussing and we would like to implement. For instance “rebase train” or “build train”. Or yet another name. :-) Let try to not waste resource. I think we could have a policy when a branch contains “heavy” rebuilds. Because we cannot continuously rebuild the world. ;-) Somehow, the team of this “heavy” rebuild branch says: “hey build train of branch X starts soon”, meaning the team is ready and the state of the branch X looks good so it require to set up the CI, i.e., the team is going to trigger more than 1000 rebuilds. The message “build train of branch X starts soon” is sent to guix-devel and one week is let to the other teams to rebase their own branch onto branch X, if they have something to build, are ready, etc. The team who asked the “build train” is responsible for crossing the final line (the merge to master) and also responsible to synchronize with the wagons, e.g., by sending some updates about how is going the world rebuild, when the final merge is planned, etc. If no one takes this train, no big deal because a next train is probably going to start soon. ;-) WDYT? Cheers, simon --8<---------------cut here---------------start------------->8--- $ guix refresh -l gsl | cut -d':' -f2 | tr ' ' '\n' | tail -n +2 | sort | uniq > gsl.deps $ for i in $(seq 2 7); do guix refresh -l ffmpeg@$i \ | cut -d':' -f2 | tr ' ' '\n' | tail -n +2 ;done | sort | uniq > ffmpeg.deps $ wc -l gsl.deps ffmpeg.deps 1473 gsl.deps 521 ffmpeg.deps 1994 total $ for line in $(cat ffmpeg.deps); do grep -n ^$line gsl.deps ;done | wc -l 521 --8<---------------cut here---------------end--------------->8--- PS: Please note the number here are not true the number of rebuilds but the number of packages that triggers all the necessary rebuilds. 1: Re: ‘core-updates’ is gone; long live ‘core-packages-team’! Ludovic Courtès <l...@gnu.org> Fri, 06 Sep 2024 11:01:46 +0200 id:87h6at2m11....@gnu.org https://lists.gnu.org/archive/html/guix-devel/2024-09 https://yhetil.org/guix/87h6at2m11....@gnu.org
