Hello again,
So, the last time I checked the notary service, what it does is
basically sends the artifact to a remote server, where the artifact is
signed with a valid key that's recognisable as KDE's to validate the
artifact as KDE's artifact in various stores. Am I explaining it correctly?
If I am right, then, I don't think that's needed for snaps. To upload a
snap to the store, one basically needs the auth token of the KDE account
in the snap store and pass it when uploading the snap. IIRC, till now,
some of the vms were having this auth token embedded. Can't we replicate
something like this?
Thanks,
Soumyadeep
On 08/07/25 00:20, Ben Cooksley wrote:
On Mon, Jul 7, 2025 at 11:55 PM Soumyadeep Ghosh
<soumyadeepghosh2...@zohomail.in> wrote:
Hi Ben,
Hi Soumyadeep,
This is an awesome news! I just wanted to know what would be the
problem in publishing those snaps built inside the new VM based CI.
There is currently no Notary built that supports handling the
publishing process at this time, which is why we're unable to publish.
Thanks,
Soumyadeep Ghosh
Cheers,
Ben
---- On Mon, 07 Jul 2025 17:19:14 +0530 *bcooks...@kde.org * wrote
----
Hi all,
I'm happy to announce that VM based CI is now in a state where
it is ready to begin making the transition into general
availability. As part of this i'm happy to advise that builds
for Snaps (but not their publishing) are also becoming
generally available, and FreeBSD will be updating to Qt 6.9 as
well.
This transition will also mark the general retirement of
Docker based CI, with Docker functionality only being retained
for website builds and linter jobs (such as xml-lint,
cppcheck, etc).
As part of this support for Qt 5 (except for general Linux) is
also being retired. This includes all FreeBSD and Windows
support, as well as all binary support (Appimages, Windows
exes, etc)
_Steps you need to take:_
If you only inherit the templates from sysadmin/ci-utilities,
and do not have custom jobs, then no action is needed on your
part.
If you have custom jobs which are based on the templates
currently in sysadmin/ci-utilities then you should examine
the diff in
https://invent.kde.org/sysadmin/ci-utilities/-/merge_requests/507
to ensure that no changes are needed to your job.
Custom jobs not based on the sysadmin provided templates
likely do not need to be changed.
Projects that are Qt 5 based should remove includes for
everything except Linux general (linux.yml) as those templates
will be removed as part of this roll out.
_Timeline for the transition:_
Over the next week the first node (node6.ci.kde.org
<http://node6.ci.kde.org>) will be taken out of rotation for
running existing container based jobs to facilitate testing of
the automated deployment. Once this has been validated, the
conversion of the remaining nodes (node2, node3, node4, node5)
will be scheduled - converting them all to be VM based CI runners.
To allow for final legacy jobs to be completed, node1 will be
left to run remaining legacy jobs for a period of time (to be
determined). Once that finishes up, that node will also be
converted.
As part of this the dedicated VM runners currently supplied
for building KDE Linux and Snaps will also be retired. Support
for Yocto builds is out of scope for this transition at this
time but that may be re-evaluated in the future.
_What specs will the VMs be provided with?_
By default builds will be provided with 8 vCPU cores, 16GB RAM
and 200GB of disk space, although some of that disk space will
be occupied by the VM OS, development tools, etc.
Builds requiring additional resources should file a Sysadmin
ticket.
VMs will have a shared mount provided from the host at /mnt
that allows for artifacts to be cached between builds on that
node. This is mainly intended to be used for caching
dependencies and other downloaded artifacts and should not be
relied on to transfer results between jobs (as there is no
guarantee contents won't go away, or that jobs will be
allocated to the same build node).
_What benefits will we get as part of this change?_
For Linux based builds, as these are now running in a fully
fledged system, anything that depends on system level services
(like CUPS or systemd/logind) or which needs to undertake
actions normally restricted in a Docker/Podman container (like
manipulating FUSE mounts or interacting with hardware like a
TPM) should now be able to be unit tested.
For FreeBSD based builds, these images no longer have to be
built manually by hand, and are now easily available to be run
locally on developer systems for local testing. Reliability of
FreeBSD builds should also improve with the elimination of the
dependency on the Podman daemon being running on the build node.
For Windows based builds, these should run faster due to the
elimination of the overhead of the Docker overlay file system,
which on Windows introduces significant overhead to IO
operations. For Kate, regular Windows CI build times were
reduced from 6 minutes (under Docker) to 3 minutes 40 seconds
(in a VM based CI setup). This includes the overhead of
provisioning the VM and waiting for Windows to start. We
should also see improvements to builder availability, as the
out of memory issues that have caused Windows build nodes to
be killed by the OOM-killer are not possible in a VM based CI
environment.
For binary builds, Linux support is enhanced by it now being
possible to build Snaps, as well as Flatpaks no longer
requiring workarounds to be built, while Appimages are now
able to be run if needed. These are actions that have all
previously been restricted by operating in a container
environment.
The system will also generally benefit from being able to
scale build capacity for different OSes on an as needed basis
within our available hardware (so long build queues for
Windows post major release events should become less of an issue).
As an additional benefit, the system will require
significantly less work to maintain. Currently each build
node, along with the FreeBSD and Windows VM thereon, have to
be maintained by hand and disk space allocated between them in
a fixed fashion. This means that any cleanup from stale disk
images, over-filled caches, etc. has to be done 3 times on
each build node (being the Linux host as well as the FreeBSD
and Windows guest VMs). Currently provisioning new nodes is
significantly labour intensive as well (see
https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/README.md
for the instructions).
This is essentially completely eliminated with the transition
to VM based CI, with the majority of the deployment now being
possible using Ansible with the only manual step being the
registration with Gitlab - which is a fairly quick process
taking less than 20 minutes per node. Maintenance is
significantly reduced as each node only needs one set of
cleanup - not three.
Should there be any questions on the above please let me know.
Many thanks,
Ben
