Hi, Thank you for documenting this! Really interesting
Le mer. 23 oct. 2024 à 16:26, Quentin Schulz via lists.yoctoproject.org <quentin.schulz=cherry...@lists.yoctoproject.org> a écrit : > Hi all, > > I have two git repos for multiple layers, let's say layerA, layerB and > layerC. I cannot merge them because one repo is "closed-source" while > the other is public. Our testing scripts in a testing/demo image are in > layerC, so to validate that changes made to layerA actually are fine, > one needs to build layerC with the content of unmerged layerA changes > (merge requests) before merging the changes. Manually doing this is > possible but is error-prone and not automated. > > So, I took some time to look at how to do this in GitLab and I am > sharing here how I managed to do it. The goal is to see if I missed > something, if there's an easier way to deal with this but mainly, maybe > someone will have interest in that and can reuse it. This needs maybe > some cleanup so it isn't available just yet in our public repo, I'm > basically waiting for feedback on it before pushing :) > > Mapping of layerX used above to actual names: > - layerA is meta-bsp in meta-cherry-es (public) > - layerB is meta-extended in meta-cherry-es (public) > - layerC is meta-theobroma-systems-demo (private) > > The important parts in .gitlab-ci.yml in meta-cherry-es: > """ > [snip] > > stages: > - build > - build-demo > > [snip] > > tiger-extended: > [snip; just a job in build stage] > > build-demo: > stage: build-demo > variables: > # Required to make sure that checking that nothing newer/different > # is now present in the parent merge request which triggered the > # child pipeline (e.g. a push happened before the child pipeline > # got triggered, the old child pipeline would use code from the > # new state of the parent merge request). > # Can be used to fail the child pipeline, or enforce a commit hash > # if and only if the GitLab server is configured to keep old > # commits of merge requests (which is not the default). > PARENT_CI_COMMIT_SHA: $CI_COMMIT_SHA > # Required to manually fetch parent code from child pipeline > # without requiring permissions to access fork project from where > # the merge request originates > PARENT_CI_MERGE_REQUEST_REF_PATH: $CI_MERGE_REQUEST_REF_PATH > # Required to generate CI_REPOSITORY_URL pointing to the parent > # project but with a job token from child pipeline. > PARENT_CI_PROJECT_PATH: $CI_PROJECT_PATH > PARENT_CI_SERVER_URL: $CI_SERVER_URL > trigger: > # FIXME: note: should be configurable via a GitLab CI/CD project > variable to avoid it being hardcoded? > project: qschulz/meta-theobroma-systems-demo > # Fail the parent pipeline if the child pipeline does > # Will likely trigger a bunch of chicken and the egg problems > strategy: depend > # Make sure we're building the same release in dependent layer and > # not the default branch > branch: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME > rules: > - if: $CI_PIPELINE_SOURCE == "merge_request_event" > """ > > My kas-cherry-es.yml files which will be used by > meta-theobroma-systems-demo's: > > meta-cherry-es/meta-bsp/kas-cherry-es.yml > """ > --- > header: > version: 14 > build_system: openembedded > machine: puma-haikou > distro: poky > local_conf_header: > debug: | > EXTRA_IMAGE_FEATURES = "debug-tweaks" > no-ptest: | > DISTRO_FEATURES:remove = "ptest" > repos: > poky: > url: "https://git.yoctoproject.org/poky"; > # yocto-5.0.2 tag > commit: f7def85be9f99dcb4ba488bead201f670304379b > branch: scarthgap > layers: > meta: > meta-poky: > meta-yocto-bsp: > meta-arm: > url: "https://git.yoctoproject.org/meta-arm"; > # yocto-5.0 tag > commit: 8aa8a1f17f5b64bc691544f989f04fc83df98adb > branch: scarthgap > layers: > meta-arm-toolchain: > meta-arm: > meta-rockchip: > url: "https://git.yoctoproject.org/meta-rockchip"; > commit: bf9ade59abc15a5f6fb5450265c214450f35857f > branch: scarthgap > meta-openembedded: > url: "https://git.openembedded.org/meta-openembedded"; > commit: 6de0ab744341ad61b0661aa28d78dc6767ce0786 > branch: scarthgap > layers: > meta-oe: > meta-python: > meta-cherry-es: > layers: > meta-bsp: > """ > > meta-cherry-es/meta-extended/kas-cherry-es.yml > """ > --- > header: > version: 14 > includes: > - repo: meta-cherry-es > file: meta-bsp/kas-cherry-es.yml > target: > - cherry-es-extended-image > repos: > meta-cherry-es: > layers: > meta-extended: > meta-openembedded: > layers: > meta-oe: > meta-networking: > """ > > The important parts in .gitlab-ci.yml in meta-theobroma-systems-demo: > > """ > --- > workflow: > rules: > # Allow multi-project pipeline trigger, c.f. > # > > https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#multi-project-pipelines > # so that a merge request in meta-cherry-es can trigger a pipeline > # in meta-theobroma-systems-demo > - if: $CI_PIPELINE_SOURCE == "pipeline" > > [snip] > > .build: > tags: > - docker > image: ghcr.io/siemens/kas/kas:4.3.2 > stage: build > script: > - | > if [ "$CI_PIPELINE_SOURCE" = "pipeline" ]; then > # FIXME: not generic (though not an issue for us :) ) > ADD_KAS_FILE="kas-mr-meta-cherry-es.yml" > parent_dir=$(basename "$PARENT_CI_PROJECT_PATH") > mkdir -p "$parent_dir" > cd "$parent_dir" > git init . > # Horrendous multi-line commands to make yamllint happy with > 80-character-long lines > repository="$(echo "$PARENT_CI_SERVER_URL" | \ > sed "s^\(http.*://\)\(.*\)^\1gitlab-ci-token:$CI_JOB_TOKEN@ > \2^")" > repository="$repository/$PARENT_CI_PROJECT_PATH" > ref="$PARENT_CI_MERGE_REQUEST_REF_PATH" > # FIXME: Note: only "required" because our internal gitlab > server is using a self-signed certificate > git -c http.sslVerify=false fetch "$repository" "$ref" || exit 1 > # Make sure we're testing what we're supposed to test > test "$PARENT_CI_COMMIT_SHA" = "$(git rev-parse FETCH_HEAD)" || > exit 1 > git checkout FETCH_HEAD || exit 1 > cd .. > else > # FIXME: not generic (though not an issue for us :) ) > ADD_KAS_FILE="kas-commit-meta-cherry-es.yml" > fi > export ADD_KAS_FILE > - KAS_MACHINE="$CI_JOB_NAME" kas build > "kas-cherry-es.yml:$ADD_KAS_FILE" > > artifacts: > paths: > - build/tmp/deploy/images/$CI_JOB_NAME/*.rootfs-*.wic > - build/tmp/deploy/images/$CI_JOB_NAME/*.rootfs-*.wic.bmap > - build/tmp/deploy/images/$CI_JOB_NAME/fitImage*$CI_JOB_NAME-*.bin > - build/tmp/deploy/images/$CI_JOB_NAME/idbloader.img-$CI_JOB_NAME-* > - build/tmp/deploy/images/$CI_JOB_NAME/u-boot-$CI_JOB_NAME-*.itb > exclude: > - build/tmp/deploy/images/$CI_JOB_NAME/fitImage*linux*.bin > timeout: 10h > > tiger-haikou: > extends: .build > """ > > My kas-*.yml files which will be used by the kas build command: > > kas-cherry-es.yml > """ > --- > header: > version: 14 > includes: > - repo: meta-cherry-es > file: meta-extended/kas-cherry-es.yml > target: > - cherry-es-demo-image > repos: > meta-openembedded: > layers: > meta-oe: > meta-networking: > meta-theobroma-systems-demo: > """ > > kas-commit-meta-cherry-es.yml > """ > --- > header: > version: 14 > repos: > meta-cherry-es: > url: https://git.embedded.cherry.de/yocto-layers/meta-cherry-es.git > commit: 71a6eee9c4cfe1a3c922b9a3c36acc6e0766a5d8 > branch: scarthgap > """ > > kas-mr-meta-cherry-es.yml > """ > --- > header: > version: 14 > repos: > meta-cherry-es: > path: meta-cherry-es > """ > > Things could be easier if kas allowed refs/merge-requests/X/head as > branch and using environment variables, in which case I could manage > with only one kas configuration file and avoid this horrendous series of > commands to manually fetch the source code. Maybe I missed something here? > > This also is not really capable of handling more than one git repo you > depend one. Say you have a layerF which layerC depends on, this wouldn't > be able to know which changes to test, layerF or layerA, but there's > probably some smart way to handle that in a generic way. I had this > issue in the past because i used to have three git repos, one for the > BSP layer, one for the extended layer (basically demo images with open > source code; or at least public) and one for the demo layer (private). > This was an absolute nightmare, so I merged the BSP layer and extended > layer into a single repo when upgrading to scarthgap, and I now feel > better and don't have the need to look into 2-depth multi-project > pipeline CI :). > > Additional note, this was tested on GitLab 17.4 and kas-container 4.3.2 > so YMMV if you have different versions. > > Any thoughts? > For what it's worth, I've arrived more or less at the same setup in our contribution project : - each oe-core, bitbake, meta-yocto has its own repo in our gitlab, integrated in a Yocto cooker based project. - a MR on these will trigger a pipeline running patchtest on the commits in the MR. - I did not find a way to map repos to their paths in a generic way so I have a manual "$CI_PROJECT_NAME -> path in integration project" mapping in my .gitlab-ci.yml file. > @cc Tim, Trevor and Pidge solely based on > https://libera.irclog.whitequark.org/yocto/2024-03-04#35933698; > https://libera.irclog.whitequark.org/yocto/2024-03-04#35934046; > > (has 8 months really already passed since then? no it couldn't have!) > > Cheers, > Quentin > > > > -- Yoann Congal Smile ECS - Tech expert
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#64176): https://lists.yoctoproject.org/g/yocto/message/64176 Mute This Topic: https://lists.yoctoproject.org/mt/109171222/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
