Re: Debugging KDE CI runners

2022-10-12 Thread Ben Cooksley 
On Wed, Oct 12, 2022 at 2:51 AM Steven Robbins  wrote:

> On Tuesday, October 11, 2022 3:20:06 A.M. CDT Ben Cooksley wrote:
> > On Tue, Oct 11, 2022 at 6:49 PM Steven Robbins  wrote:
>
> > > I am trying to understand how libraries are selected for installation
> on
> > > the
> > > KDE CI/CD machines.
>
> > There are a couple of things here to pull apart.
> >
> > First, the Gitlab CI jobs you mention above themselves don't use Craft at
> > all at build time, so the above configurations you are referring to have
> no
> > impact whatsoever.
>
> Ah, that's one crucial detail, thanks!
>
> > Those jobs rely on KDE specific Docker containers on Linux and Windows,
> and
> > on FreeBSD is a fixed statically provisioned machine.
> > For Linux and FreeBSD, we use only distribution provided packages for
> those
> > systems - while on Windows we use Craft to deploy a static and
> > centrally managed list of packages on the system.
>
> OK.  Can you help me understand the full scope of Craft?  Is it solely for
> the
> windows CI configuration?  I've heard that there are also distribution
> builders
> (for flatpack and the like) -- is Craft involved in any of that?
>

There are two parts of our systems:
a) Continuous Integration, which runs unit tests, code quality checks and
monitors code to ensure it compiles
b) Continuous Delivery, which produces binaries for end users to use.

Prior to the move to Gitlab, (a) was handled by build.kde.org, while (b)
was handled by binary-factory.kde.org.
While the move to GItlab for (a) is complete, it has yet to be completed
for (b) but will be in the next few months.

Craft is for the most part only involved in (b).

The only places Craft is used in (a) is to produce Docker images for
Windows and Android which the system uses to carry out its CI runs.
These image builds are always using a static configuration which is
centrally maintained and cannot be varied on a per-project basis.

With regards to the production of Binaries, Craft is used for the following
types of builds:
(a) Windows installers, portable archives and store upload bundles
(b) Android APKs
(c) Linux Appimages
(d) macOS *.app / DMG bundles

Linux Flatpak bundles are built using their own tooling, which is what the
existing Gitlab Flatpak job we already have in place does.


>
> > You are therefore subject to whatever versions of ffmpeg are provided by
> > OpenSUSE (our Linux Docker image provider) and FreeBSD for those
> platforms,
> > and on Windows whatever Craft has specified as the default version of
> > ffmpeg.
>
> OK -- this explains windows I suppose, as the default ffmpeg is indeed set
> to
> 5.0.1 [1].  Is this the default setting you refer to?
>
> [1]
> https://invent.kde.org/packaging/craft-blueprints-kde/-/blob/master/libs/
> ffmpeg/ffmpeg.py#L25
> 


Yes, that is the setting.


>
>
> For OpenSUSE and FreeBSD: is there a central location I can check to
> understand what version of suse/freebsd is being presently used?  How
> often
> are OS updates done?  Is there an annoucement list to track this?
>

The SUSE images are updated periodically on an as-needed basis to bring in
updates or additional packages we need.
There is no set schedule for this. (same applies for the other Docker
images we have)

You can view the build logs for all our Docker images (covering LInux,
Android and Windows) in the CI section of
https://invent.kde.org/sysadmin/ci-images/
Our Dockerfile definitions are hosted in that same repository.

For FreeBSD, those VMs are maintained by the KDE FreeBSD team on Virtual
Machines provided by KDE Sysadmin.
These are maintained manually, and once again there is no schedule for this.


>
> Now comes my real question.  At present, Digikam aims to support both
> FFMPEG 4
> and 5.  Therefore it is best to test both configurations.  Is there any
> mechanism to have TWO CI runners per OS -- one with each version of ffmpeg?
>

Sorry, but it is unnecessary to have two builds per operating system when
you will get just as good coverage by having Linux build with ffmpeg 4 and
Windows build with ffmpeg 5.
We adopt the same process for testing compiler coverage, with Linux
handling GCC, FreeBSD handling Clang and Windows handling MSVC.

Additionally, Digikam is one of three projects that consume significant
amounts of CI resources, so i'd very much rather that didn't grow further.


>
>
> > With regards to your Craft Blueprint above, my understanding is that what
> > you have there is not valid syntax, as Craft doesn't look at individual
> > blueprints to resolve versions to use.
> > This is why in your Binary Factory configuration you have the following
> > specified:
> >
> > 'Digikam':
> >  buildBlueprint: "digikam"
> >  versions:
> >  - name: 'Nightly'
> >target: 'master'
> >packageAppx: True
> >options:
> >- 'libs/ffmpeg.version=4.4'
> >  platforms:
> >  - 'macos'
> >  - 'win64'
> >  - 'ming

Re: Copying po/docbook files to repositories nightly

2022-10-12 Thread Albert Astals Cid 
El dilluns, 10 d’octubre de 2022, a les 22:00:59 (CEST), Albert Astals Cid va 
escriure:
> El dilluns, 10 d’octubre de 2022, a les 15:03:14 (CEST), Alvin Wong va
> 
> escriure:
> > Hi,
> > 
> > On 10/10/2022 5:20, Albert Astals Cid wrote:
> > > El divendres, 7 d’octubre de 2022, a les 23:07:38 (CEST), Albert Astals
> > > Cid va>
> > > 
> > > escriure:
> > >> El dimecres, 5 d’octubre de 2022, a les 18:59:08 (CEST), Alvin Wong va
> > >> 
> > >> escriure:
> > >>> Hi,
> > >>> 
> > >>> I notice this has been applied to docs-krita-org
> > >>> 
> > >>> .
> > >>> However, being a Sphinx doc project it already has a
> > >>> `StaticMessages.sh`
> > >>> script to copy the PO files into the `locale/` directory and
> > >>> "unflatten"
> > >>> the directory layout to what the Sphinx build expects (the files in
> > >>> the
> > >>> l10n SVN tree have the directory layout flattened by mangling the
> > >>> filenames). Now the files are also being copied to the `po/`
> > >>> directory,
> > >>> but
> > >>> in the flattened layout, which in practice are unused duplicated
> > >>> copies.
> > >>> Should we opt-out of this copying step to avoid the duplicated files,
> > >>> or
> > >>> is
> > >>> there a better way to handle this?
> > >> 
> > >> We will make it so that for StaticMessages.sh the po files are not
> > >> copied
> > >> back.
> > > 
> > > So we kind of fixed that but that didn't work for your use case because
> > > you're using the EXPORTS_POT_DIR=1 "special" use case
> > > 
> > > The gist of the "fix" we did is that we don't copy back to the git repo
> > > the
> > > files that end in _static_.po
> > > 
> > > Would able to adapt your StaticMessages.sh so that's the suffix of the
> > > files being generated?
> > > 
> > > Cheers,
> > > 
> > >Albert
> > 
> > I am not sure we want to change the file names of almost 300 .pot files
> > which may be disrupting to translators.
> 
> File renaming will be [mostly] transparent for translators, hardly any
> disruption at all.

Forgot to say. You need to tell us when you do the renaming so we rename all 
existing files.

Also the po/ fix you suggested would also work, i feel it's non optimal but if 
you prefer to do that it's also acceptable if you documented it well in the 
.gitignore for future ourselves to remember why it's there.

Cheers,
  Albert

> 
> Cheers,
>   Albert
> 
> > Will it be a better idea to just
> > remove the `po/` directory and add it to `.gitignore`?
> > 
> > Best Regards,
> > Alvin
> > 
> > >>> By the way, it seems the existing `StaticMessages.sh` copy step is
> > >>> slightly
> > >>> out of sync with the new copy step (the git commits don't have
> > >>> identical
> > >>> diff contents). Is this something to be concerned about?
> > >> 
> > >> Can you point me to such discrepancy?
> > >> 
> > >> Cheers,
> > >> 
> > >>Albert
> > >>> 
> > >>> Best Regards,
> > >>> Alvin
> > >>> 
> >  El divendres, 2 de setembre de 2022, a les 23:24:21 (CEST), Albert
> >  Astals
> >  Cid>
> >  
> >  va escriure:
> > > /As you may know, translations for apps don't live in the same place
> > > as
> > > the />/code for the apps themselves. />//>/This greatly benefits
> > > translators but is not awesome for the release />/management side of
> > > things since it means that for each release we need to />/not forget
> > > to
> > > copy the appropriate files to the appropriate place, makes
> > > />/tagging
> > > somewhat harder, etc. />//>/For a while now we have been running an
> > > "experimental" copy-po-qm-docbook- />/back-to-repository in a number
> > > of
> > > "select" repositories and it seems to have />/worked quite well, you
> > > can
> > > seem one example in
> > > />/https://invent.kde.org/plasma-mobile/alligator/-/commits/master/p
> > > o
> > > />//>/The idea is to enable this for all repositories. />
> >  
> >  This has now been enabled for master branch and according to scripty
> >  logs
> >  all seems to have worked.
> >  
> >  Please inspect your repositories and make sure the po files are there
> >  where
> >  they should and nothing broke.
> >  
> >  Also please make sure you adapt your releasing scripts if you release
> >  from
> >  master.
> >  
> >  Cheers,
> >  
> > Albert
> > > 
> > > //>/This is a heads up, as a developer there's nothing you need to
> > > do,
> > > at
> > > most />/remove the po/ folder from .gitignore if for some reason it
> > > is
> > > there. />//>/If you're a packager you will need to make sure your
> > > scripts don't try to />/copy po/qm/docbook files anymore when doing
> > > a
> > > release once this is />/activated. />//>/My plan would be to enable
> > > this
> > > scripts over Akademy so we have the high />/bandwidth there to fix
> > > things if needed.

[ANNOUNCE] CMake 3.25.0-rc1 is ready for testing

2022-10-12 Thread John Parent 
I am proud to announce the first CMake 3.25 release candidate.
  https://cmake.org/download/

Documentation is available at:
  https://cmake.org/cmake/help/v3.25

Release notes appear below and are also published at
  https://cmake.org/cmake/help/v3.25/release/3.25.html

Some of the more significant changes in CMake 3.25 are:


* The "find_file()", "find_path()", "find_library()", and
  "find_program()" commands gained a "VALIDATOR" option to specify a
  function to be called for each candidate item to validate it.

* The "try_compile()" and "try_run()" commands gained new signatures
  that more consistently use keyword dispatch and do not require a
  binary directory to be specified.  Additionally, these signatures
  use a unique directory for each invocation, which allows multiple
  outputs to be preserved when using "cmake --debug-trycompile".

* The "add_subdirectory()" command gained a "SYSTEM" option to enable
  the "SYSTEM" directory property in the subdirectory.

* The "block()" and "endblock()" commands were added to manage
  specific scopes (policy or variable) for a contained block of
  commands.

* The "return()" command gained a "PROPAGATE" option to propagate
  variables to the scope to which control returns. See policy
  "CMP0140".

* The "BSD" and "CMAKE_HOST_BSD" variables are now set to a string
  value when the target or host system is BSD, respectively.

* The "LINUX" and "CMAKE_HOST_LINUX" variables are now set to true
  when the target or host system is Linux, respectively.

* The "CMAKE_MSVC_DEBUG_INFORMATION_FORMAT" variable and
  "MSVC_DEBUG_INFORMATION_FORMAT" target property were introduced to
  select the debug information format for compilers targeting the MSVC
  ABI. See policy "CMP0141".


CMake 3.25 Release Notes


Changes made since CMake 3.24 include the following.


New Features



Presets
---

* The "cmake-presets(7)" schema version has been bumped to "6".

* The "cmake-presets(7)" format now supports a "packagePresets" field
  to specify presets for "cpack --preset".

* The "cmake-presets(7)" format now supports a "workflowPresets" field
  to specify presets for "cmake --workflow".


Languages
-

* The "Compile Features" functionality is now aware of C++26, and
  defines a "cxx_std_26" meta-feature. C++26 compiler modes may also
  be specified via the "CXX_STANDARD", "CUDA_STANDARD",
  "HIP_STANDARD", or "OBJCXX_STANDARD" target properties.

* "CUDA" language support now includes device link-time optimization
  when using "nvcc".  The "CMAKE_INTERPROCEDURAL_OPTIMIZATION"
  variable and the associated "INTERPROCEDURAL_OPTIMIZATION" target
  property will activate device LTO.


Command-Line


* A "cmake --workflow --preset" mode was added to drive sequences of
  configure, build, test, and package operations through a single
  command.

* The "cmake -E capabilities" command gained a new "tls" field that
  tells whether or not TLS is enabled.

* The "cmake -E env" command-line tool gained a "--modify" flag to
  support "ENVIRONMENT_MODIFICATION" operations.

* The "cmake --debug-trycompile" option now prints log messages
  reporting the directory in which each try-compile check is done.


Compilers
-

* Support for the Tasking compiler toolsets (SmartCode, TriCore,
  Standalone: ARM, MCS, 8051) was added with compiler id "Tasking".
  See the "CMAKE_TASKING_TOOLSET" variable.


Commands


* The "add_subdirectory()" command gained a "SYSTEM" option to enable
  the "SYSTEM" directory property in the subdirectory.

* The "block()" and "endblock()" commands were added to manage
  specific scopes (policy or variable) for a contained block of
  commands.

* The "cmake_language()" command gained a new "GET_MESSAGE_LOG_LEVEL"
  sub-command.  It can be used to query the current message logging
  level.

* The "find_file()", "find_path()", "find_library()", and
  "find_program()" commands gained a "VALIDATOR" option to specify a
  function to be called for each candidate item to validate it.

* The "find_package()" command now considers paths of the form
  "/*/(cmake|CMake)/*/" when searching for package
  configuration files.

* The "return()" command gained a "PROPAGATE" option to propagate
  variables to the scope to which control returns. See policy
  "CMP0140".

* The "try_compile()" and "try_run()" commands gained new signatures
  that more consistently use keyword dispatch and do not require a
  binary directory to be specified.  Additionally, these signatures
  use a unique directory for each invocation, which allows multiple
  outputs to be preserved when using "cmake --debug-trycompile".

* The "try_compile()" and "try_run()" commands gained the option
  "NO_CACHE" to store results in normal variables.

* The "try_run()" command gained "RUN_OUTPUT_STDOUT_VARIABLE" and
  "RUN_OUTPUT_STDERR_VARIABLE" options to capture stdout and stderr
  separately from the output of the compiled program.


Variables

Why is windows CI so variable?

2022-10-12 Thread Steven Robbins 
Ben,

Thanks very much again for all the information you provided.  It was very 
illuminating and answered all my questions to this point.


On Wednesday, October 12, 2022 3:48:50 A.M. CDT Ben Cooksley wrote:

> Additionally, Digikam is one of three projects that consume significant
> amounts of CI resources, so i'd very much rather that didn't grow further.

This brings a separate question to mind.  I've noticed that the windows build 
is hugely variable.  It can take anywhere from about 25 minutes to over an 
hour (at which point the job is killed).

Have you any insight into why the windows machine is so variable compared to  
linux and bsd?

THanks,
-Steve


signature.asc
Description: This is a digitally signed message part.