I'd like to propose that we enable GitLab CI and use it for automatically testing our merge requests. It would run 'make', 'make test', and 'make doc' like the current manual testing process. Once we have that, GitLab can be configured to enforce a successful pipeline before merging. As we only allow fast-forward merges, this means each MR has received testing in the form it hits master. This would effectively replace the current setup of pushing to staging.
The system would use Docker containers which is similar in spirit to Han-Wen's proposal in February [1]. However the system is much simpler and the containers don't come with a full development environment. Instead they have barely enough to compile and test LilyPond. For the actual changes and to get a feel of the UX, see the MR: https://gitlab.com/lilypond/lilypond/-/merge_requests/57 Advantages ========== - We get automatic testing, tightly integrated into GitLab. - It helps the Patch Meister (James); see below for one caveat. - No patchy required for pushing to master. Disadvantages ============= - Using Docker images, we get a single standardized environment for the tests. While being great for reproducibility, it looses the diversity we had so far (please refer to earlier discussions). - Comparison of regression tests is not yet integrated, the main problem being the need for a baseline. I already have an idea or two how this could work, but for now I'm focusing on the initial setup. This means James still needs to download the patches and run 'make check' (make doc being run automatically). - There's no possibility to push without a merge request, and it needs to be merged through the UI. I don't consider this a loss personally. - There's no possibility to push multiple changes at once: Merge requests are rebased, tested and merged one by one. GitLab is planning to implement a queuing mechanism, and if all else fails we can invent our own scripts. Where to Run ============ Now towards infrastructure: Being a public project and open-source GitLab currently offers 50,000 CI minutes per month for free. [1] This has the advantage that test results are available immediately. 50,000 minutes divided by ~60 minutes per build, this is ~800 builds per month, or around 25 per day. Not sure what happens if we exceed this, comments claim it's "enforced" in some way. However there's no page to see the current usage (for a public group project)... If this is too little, if GitLab reduces the amount of free minutes, or if we want to get faster builds, we can always add our own machines. That is a matter of installing Docker and the runner (packages provided by GitLab). Configuration is as simple as running one command and pasting the URL as well as a token. We can even go hybrid and use the best of both worlds. My tests suggest that specific runners are preferred, using the shared runners as fallback. Let me know what you think Jonas 1: https://lists.gnu.org/archive/html/lilypond-devel/2020-02/msg00416.html 2: Our 'lilypond' group is older than https://about.gitlab.com/releases/2020/03/18/ci-minutes-for-free-users/
signature.asc
Description: This is a digitally signed message part
