On Mon, Jun 2, 2025, at 5:42 PM, Christoph Cullmann wrote:
> Hi,
>
>
> On Monday, June 2nd, 2025 at 13:40, Ben Cooksley <bcooks...@kde.org> wrote:
>
>> Hi all,
>> For some time now we have had a variety of issues with our Docker/Podman
>> based CI builds. These have included the lack of GUI test support on
>> Windows, periodic crashes on FreeBSD, poor IO performance of Windows builds,
>> issues supporting builds for Flatpak and Snaps and inability to support
>> either builds or tests where elevated privileges or system session resources
>> are needed.
>>
>
>> In addition to this we've had issues where Linux CI builds have the
>> capability to trigger OOM events on the CI hosts, which in turn takes out
>> Windows and (less often) FreeBSD builders. While this does not occur too
>> often, it does happen from time to time and eventually negatively impacts
>> the build queue for those platforms.
>>
>
>> The need to have dedicated VMs for FreeBSD and Windows on our builders also
>> makes setting up of a CI build node for KDE software a more complicated and
>> time intensive task than it otherwise needs to be (and means that the amount
>> of systems to care for increases by 3 for every CI node we add).
>>
>
>> While individually relatively minor, together these issues more than justify
>> making a significant change to the way we run our CI system - in this case
>> transitioning from container based builds to VM based builds.
>>
>
>> These builds will still take place on dedicated hardware that we control,
>> however instead of taking place within a container (managed by Podman on
>> Linux and FreeBSD, or Docker on Windows) they will instead take place within
>> a VM using a copy-on-write disk image.
>> VM based builds will unfortunately take a little longer to start (it takes
>> ~10 seconds for a VM from any of Linux, FreeBSD or Windows to boot on my
>> personal system) however the benefits we gain should more than outweigh this
>> small downside.
>>
>
>> This has been under development for the past couple of weeks and is now
>> reaching the point where the only remaining steps are to get it integrated
>> with the Gitlab CI agent (gitlab-runner) for which prototype code is already
>> in place, and complete porting of our images over. Once that happens a
>> complete rebuild of all of our builders will be swiftly undertaken to
>> transition them completely over to the new VM based infrastructure.
>>
>
>> Specs wise, at this time it is planned for each spawned standard VM to be
>> provided with 2/3's of the system CPU cores (so 12 cores), 16GB RAM and
>> 100GB of disk space (although some of that will be occupied by the system
>> image - approximately 10GB for standard Linux builds and ~30GB or so for
>> Windows builds). There will be a higher resource tier available for certain
>> builds however that will be on request only and would need to be justified
>> (such as Craft needing to build QtWebEngine).
>>
>
>> As launching VMs is not the most efficient approach for all workloads,
>> limited support for running Docker containers will be preserved, however
>> this support is primarily intended for running linters, sanity checks and
>> website builds, and is not intended for running general CI/CD builds.
>>
>
>> The tooling used by the CI nodes to run VMs is something that should be
>> fairly trivial for people to run on their own local system should they wish
>> to run any of those images (say for FreeBSD or Android), although you will
>> need to setup libvirt yourself (SUSE has very good instructions for this,
>> Debian less so as their instructions lack installing the packages needed to
>> provide UEFI and TPM support). The tooling itself was merged this evening to
>> sysadmin/ci-images (vm-common/ folder) and can be used with the VM images
>> found at https://storage.kde.org/vm-images/
>>
>
>> There is however one downside to this - Qt 5 support.
>>
>
>> Over the past few months distributions have been steadily removing packages
>> and other supporting infrastructure needed to keep Qt 5 builds alive. In the
>> case of Windows, support for the entire Qt 5 tree has been unmaintained for
>> some time. For FreeBSD and SUSE a significant number of packages have been
>> removed - which in the case of SUSE also includes packages needed to support
>> the building of KJS. Accordingly, because builds of Frameworks are a first
>> stepping stone to support building anything else, it will not be possible
>> for us to produce Qt 5 based VM build images for any of the 3 platforms.
>>
>
>> We will therefore have to remove Qt 5 support from the CI system with the
>> transition to VM based CI.
>>
>
>> Please let me know if there are any questions on the above.
>
>
>
> Have we some overview how many things on invent.kde.org will loose the
> the CI as they are still Qt 5 only?
Happy to help upgrade some.
Upgrade for marK:
https://invent.kde.org/education/mark/-/merge_requests/10
Examining:
https://invent.kde.org/education/khipu
https://invent.kde.org/education/kstars
https://invent.kde.org/education/rocs
https://invent.kde.org/education/ktouch
Within education, these seem to be the only ones still needing migration. Can
help
migrate these. Can also help with some of the Games software.
>
> Greetings
> Christoph
> Attachments:
> * signature.asc
