Hi all,
I have two git repos for multiple layers, let's say layerA, layerB and
layerC. I cannot merge them because one repo is "closed-source" while
the other is public. Our testing scripts in a testing/demo image are in
layerC, so to validate that changes made to layerA actually are fine,
one needs to build layerC with the content of unmerged layerA changes
(merge requests) before merging the changes. Manually doing this is
possible but is error-prone and not automated.
So, I took some time to look at how to do this in GitLab and I am
sharing here how I managed to do it. The goal is to see if I missed
something, if there's an easier way to deal with this but mainly, maybe
someone will have interest in that and can reuse it. This needs maybe
some cleanup so it isn't available just yet in our public repo, I'm
basically waiting for feedback on it before pushing :)
Mapping of layerX used above to actual names:
- layerA is meta-bsp in meta-cherry-es (public)
- layerB is meta-extended in meta-cherry-es (public)
- layerC is meta-theobroma-systems-demo (private)
The important parts in .gitlab-ci.yml in meta-cherry-es:
"""
[snip]
stages:
- build
- build-demo
[snip]
tiger-extended:
[snip; just a job in build stage]
build-demo:
stage: build-demo
variables:
# Required to make sure that checking that nothing newer/different
# is now present in the parent merge request which triggered the
# child pipeline (e.g. a push happened before the child pipeline
# got triggered, the old child pipeline would use code from the
# new state of the parent merge request).
# Can be used to fail the child pipeline, or enforce a commit hash
# if and only if the GitLab server is configured to keep old
# commits of merge requests (which is not the default).
PARENT_CI_COMMIT_SHA: $CI_COMMIT_SHA
# Required to manually fetch parent code from child pipeline
# without requiring permissions to access fork project from where
# the merge request originates
PARENT_CI_MERGE_REQUEST_REF_PATH: $CI_MERGE_REQUEST_REF_PATH
# Required to generate CI_REPOSITORY_URL pointing to the parent
# project but with a job token from child pipeline.
PARENT_CI_PROJECT_PATH: $CI_PROJECT_PATH
PARENT_CI_SERVER_URL: $CI_SERVER_URL
trigger:
# FIXME: note: should be configurable via a GitLab CI/CD project
variable to avoid it being hardcoded?
project: qschulz/meta-theobroma-systems-demo
# Fail the parent pipeline if the child pipeline does
# Will likely trigger a bunch of chicken and the egg problems
strategy: depend
# Make sure we're building the same release in dependent layer and
# not the default branch
branch: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
"""
My kas-cherry-es.yml files which will be used by
meta-theobroma-systems-demo's:
meta-cherry-es/meta-bsp/kas-cherry-es.yml
"""
---
header:
version: 14
build_system: openembedded
machine: puma-haikou
distro: poky
local_conf_header:
debug: |
EXTRA_IMAGE_FEATURES = "debug-tweaks"
no-ptest: |
DISTRO_FEATURES:remove = "ptest"
repos:
poky:
url: "https://git.yoctoproject.org/poky";
# yocto-5.0.2 tag
commit: f7def85be9f99dcb4ba488bead201f670304379b
branch: scarthgap
layers:
meta:
meta-poky:
meta-yocto-bsp:
meta-arm:
url: "https://git.yoctoproject.org/meta-arm";
# yocto-5.0 tag
commit: 8aa8a1f17f5b64bc691544f989f04fc83df98adb
branch: scarthgap
layers:
meta-arm-toolchain:
meta-arm:
meta-rockchip:
url: "https://git.yoctoproject.org/meta-rockchip";
commit: bf9ade59abc15a5f6fb5450265c214450f35857f
branch: scarthgap
meta-openembedded:
url: "https://git.openembedded.org/meta-openembedded";
commit: 6de0ab744341ad61b0661aa28d78dc6767ce0786
branch: scarthgap
layers:
meta-oe:
meta-python:
meta-cherry-es:
layers:
meta-bsp:
"""
meta-cherry-es/meta-extended/kas-cherry-es.yml
"""
---
header:
version: 14
includes:
- repo: meta-cherry-es
file: meta-bsp/kas-cherry-es.yml
target:
- cherry-es-extended-image
repos:
meta-cherry-es:
layers:
meta-extended:
meta-openembedded:
layers:
meta-oe:
meta-networking:
"""
The important parts in .gitlab-ci.yml in meta-theobroma-systems-demo:
"""
---
workflow:
rules:
# Allow multi-project pipeline trigger, c.f.
#
https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#multi-project-pipelines
# so that a merge request in meta-cherry-es can trigger a pipeline
# in meta-theobroma-systems-demo
- if: $CI_PIPELINE_SOURCE == "pipeline"
[snip]
.build:
tags:
- docker
image: ghcr.io/siemens/kas/kas:4.3.2
stage: build
script:
- |
if [ "$CI_PIPELINE_SOURCE" = "pipeline" ]; then
# FIXME: not generic (though not an issue for us :) )
ADD_KAS_FILE="kas-mr-meta-cherry-es.yml"
parent_dir=$(basename "$PARENT_CI_PROJECT_PATH")
mkdir -p "$parent_dir"
cd "$parent_dir"
git init .
# Horrendous multi-line commands to make yamllint happy with
80-character-long lines
repository="$(echo "$PARENT_CI_SERVER_URL" | \
sed "s^\(http.*://\)\(.*\)^\1gitlab-ci-token:$CI_JOB_TOKEN@\2^")"
repository="$repository/$PARENT_CI_PROJECT_PATH"
ref="$PARENT_CI_MERGE_REQUEST_REF_PATH"
# FIXME: Note: only "required" because our internal gitlab
server is using a self-signed certificate
git -c http.sslVerify=false fetch "$repository" "$ref" || exit 1
# Make sure we're testing what we're supposed to test
test "$PARENT_CI_COMMIT_SHA" = "$(git rev-parse FETCH_HEAD)" ||
exit 1
git checkout FETCH_HEAD || exit 1
cd ..
else
# FIXME: not generic (though not an issue for us :) )
ADD_KAS_FILE="kas-commit-meta-cherry-es.yml"
fi
export ADD_KAS_FILE
- KAS_MACHINE="$CI_JOB_NAME" kas build
"kas-cherry-es.yml:$ADD_KAS_FILE"
artifacts:
paths:
- build/tmp/deploy/images/$CI_JOB_NAME/*.rootfs-*.wic
- build/tmp/deploy/images/$CI_JOB_NAME/*.rootfs-*.wic.bmap
- build/tmp/deploy/images/$CI_JOB_NAME/fitImage*$CI_JOB_NAME-*.bin
- build/tmp/deploy/images/$CI_JOB_NAME/idbloader.img-$CI_JOB_NAME-*
- build/tmp/deploy/images/$CI_JOB_NAME/u-boot-$CI_JOB_NAME-*.itb
exclude:
- build/tmp/deploy/images/$CI_JOB_NAME/fitImage*linux*.bin
timeout: 10h
tiger-haikou:
extends: .build
"""
My kas-*.yml files which will be used by the kas build command:
kas-cherry-es.yml
"""
---
header:
version: 14
includes:
- repo: meta-cherry-es
file: meta-extended/kas-cherry-es.yml
target:
- cherry-es-demo-image
repos:
meta-openembedded:
layers:
meta-oe:
meta-networking:
meta-theobroma-systems-demo:
"""
kas-commit-meta-cherry-es.yml
"""
---
header:
version: 14
repos:
meta-cherry-es:
url: https://git.embedded.cherry.de/yocto-layers/meta-cherry-es.git
commit: 71a6eee9c4cfe1a3c922b9a3c36acc6e0766a5d8
branch: scarthgap
"""
kas-mr-meta-cherry-es.yml
"""
---
header:
version: 14
repos:
meta-cherry-es:
path: meta-cherry-es
"""
Things could be easier if kas allowed refs/merge-requests/X/head as
branch and using environment variables, in which case I could manage
with only one kas configuration file and avoid this horrendous series of
commands to manually fetch the source code. Maybe I missed something here?
This also is not really capable of handling more than one git repo you
depend one. Say you have a layerF which layerC depends on, this wouldn't
be able to know which changes to test, layerF or layerA, but there's
probably some smart way to handle that in a generic way. I had this
issue in the past because i used to have three git repos, one for the
BSP layer, one for the extended layer (basically demo images with open
source code; or at least public) and one for the demo layer (private).
This was an absolute nightmare, so I merged the BSP layer and extended
layer into a single repo when upgrading to scarthgap, and I now feel
better and don't have the need to look into 2-depth multi-project
pipeline CI :).
Additional note, this was tested on GitLab 17.4 and kas-container 4.3.2
so YMMV if you have different versions.
Any thoughts?
@cc Tim, Trevor and Pidge solely based on
https://libera.irclog.whitequark.org/yocto/2024-03-04#35933698;
https://libera.irclog.whitequark.org/yocto/2024-03-04#35934046;
(has 8 months really already passed since then? no it couldn't have!)
Cheers,
Quentin
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#64095): https://lists.yoctoproject.org/g/yocto/message/64095
Mute This Topic: https://lists.yoctoproject.org/mt/109171222/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
