This sounds great for being able to run CI locally! Very nice work! :) Is this process intended to be documented, at https://develop.kde.org/docs/ maybe?
On Mon, Jun 30, 2025 at 7:17 AM Ben Cooksley <bcooks...@kde.org> wrote: > On Mon, Jun 30, 2025 at 8:42 AM Albert Vaca Cintora <albertv...@gmail.com> > wrote: > >> What will be the procedure if we ever need to add a dependency now? With >> Docker it was easy because we could just edit the Dockerfile. What will be >> the equivalent? >> > > This is still being managed through sysadmin/ci-images, which now contains > everything needed to build the VM templates. > Much like with the Docker setup, there is a folder still for each VM image > which contains the logic for that particular image. > > If you look in each folder there you'll see various scripts which are used > to perform the build - with all except for Windows having two common entry > points of "build.sh" and "sysprep.sh". > > These two entry point scripts are executed by sudo / as root within the VM > image - with build.sh essentially being the new "Dockerfile". If you need > to bring files in like patches or other scripts, for all OSes (except > Windows) they can be simply placed in that folder and will become available > at /var/lib/ci-files/ within the VM. I'd suggest looking at flatpak-builder > / alpine-qt68 as examples of how this is used. > > The CI jobs in the sysadmin/ci-images repository are still used to build > and publish images in a fully automated fashion. > > For those wondering how this works, in sysadmin/ci-images you'll find a > utility "vm-runner" which is used for fetching images as well as > provisioning and tearing down VMs based on those images (or any other qcow2 > disk image you happen to have locally). It can be run on any system that > has a functioning libvirt instance and KVM setup, which most distributions > should provide - so you can run anything the CI system runs on your local > system with minimal effort. > > For Linux and FreeBSD vm-runner works by leveraging cloud-init to > configure the system, which allows it to set a hostname, ensure the disk is > fully expanded, provision user accounts for access, etc. You can see what > the default templates do by looking in vm-runner/templates/. Among other > things this includes dropping the files from those folders in > sysadmin/ci-images into the image (which is what the --cloudinit-resources > flag to vm-runner run triggers). > > To build an image all we do (for Linux/FreeBSD at least) is: > a) Download a cloud-init enabled image for the distribution in question > b) Boot that image using vm-runner, passing in the resources from the > sysadmin/ci-images folder for the image being built > c) Via SSH, run the build.sh script within the VM (using sudo to give it > root privileges) > d) Via SSH, run the sysprep.sh script within the VM (once again using sudo) > e) Once the VM has shutdown, upload it to storage.kde.org/vm-images/ for > use by VM based CI builds / vm-runner clients run elsewhere. > > Developers are more than welcome to use vm-runner locally should they wish > to. It's requirements are pretty light (ignoring the VM specs you specify) > and i've been using it on my local system while developing this setup (and > it actually makes some sysadmin tasks easier by letting me quickly spin up > VMs to test things in). The images used by the CI system live at > storage.kde.org/vm-images/ and should be specified without the .qcow2 > extension. > > If I want to run an Alpine CI image locally for instance i'd run: > vm-runner run --cpu 4 --ram 8G --disk 100G --image > storage.kde.org/vm-images/alpine-qt68 --pull --mount ~/shared/ --ssh-keys > ~/.ssh/id_ed25519.pub myalpineci > > Once booted it will advise me of the IP address and I can login by doing a > relatively standard ssh user@$IP to obtain a shell - with ~/shared/ from > my host system mounted at /mnt in the VM. Note that shared mounts don't > currently work for FreeBSD due to missing drivers on the FreeBSD side - I > believe those are in review for FreeBSD 15.0 currently. > > Hope that illustrates things a bit. > > Cheers, > Ben > > >> >> On Sat, Jun 28, 2025 at 1:23 PM Ben Cooksley <bcooks...@kde.org> wrote: >> >>> Hi all, >>> >>> As part of our transition to VM based CI, i'm about to start cutting >>> over the build infrastructure for the underlying images used to support our >>> CI system from the current Docker model to the new VM based model. >>> >>> This follows the successful on-infrastructure build of a Windows image >>> completed this evening, which was the last milestone component remaining to >>> be validated when it came to building images. >>> >>> Because the system will continue to rely on the Docker images for a >>> period of time while we're transitioning over the next week or two, we will >>> temporarily be unable to add, update or remove any system level >>> dependencies (including Qt) during this time period. >>> >>> More information on the transition will be published in the coming week >>> once the images have been ported over, all successfully built, and a >>> variety of test builds have been successfully accomplished using them. >>> >>> I'm anticipating the completion of migration to VM based CI to be >>> completed within the next 2 weeks, if not sooner. >>> >>> Thanks, >>> Ben >>> >> -- Cheers, Kristen
