On Fri, May 19, 2023 at 10:00 AM Daniel P. Berrangé <berra...@redhat.com> wrote: > > On Fri, Apr 07, 2023 at 03:52:51PM +0100, Camilla Conte wrote: > > Configure Gitlab CI to run on Kubernetes > > according to the official documentation. > > https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#docker-in-docker-with-tls-enabled-in-kubernetes > > > > These changes are needed because of the CI jobs > > using Docker-in-Docker (dind). > > As soon as Docker-in-Docker is replaced with Kaniko, > > these changes can be reverted. > > > > I documented what I did to set up the Kubernetes runner on the wiki: > > https://wiki.qemu.org/Testing/CI/KubernetesRunners > > > > Signed-off-by: Camilla Conte <cco...@redhat.com> > > --- > > .gitlab-ci.d/container-template.yml | 6 +++--- > > .gitlab-ci.d/default.yml | 3 +++ > > .gitlab-ci.d/opensbi.yml | 8 +++----- > > .gitlab-ci.d/qemu-project.yml | 17 +++++++++++++++++ > > 4 files changed, 26 insertions(+), 8 deletions(-) > > create mode 100644 .gitlab-ci.d/default.yml > > > > diff --git a/.gitlab-ci.d/container-template.yml > > b/.gitlab-ci.d/container-template.yml > > index 519b8a9482..f55a954741 100644 > > --- a/.gitlab-ci.d/container-template.yml > > +++ b/.gitlab-ci.d/container-template.yml > > @@ -1,14 +1,14 @@ > > .container_job_template: > > extends: .base_job_template > > - image: docker:stable > > + image: docker:20.10.16 > > stage: containers > > services: > > - - docker:dind > > + - docker:20.10.16-dind > > before_script: > > - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" > > - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" > > - apk add python3 > > - - docker info > > + - until docker info; do sleep 1; done > > - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p > > "$CI_REGISTRY_PASSWORD" > > script: > > - echo "TAG:$TAG" > > diff --git a/.gitlab-ci.d/default.yml b/.gitlab-ci.d/default.yml > > new file mode 100644 > > index 0000000000..292be8b91c > > --- /dev/null > > +++ b/.gitlab-ci.d/default.yml > > @@ -0,0 +1,3 @@ > > +default: > > + tags: > > + - $RUNNER_TAG > > Can we just put this in base.yml instead of creating a new file.
Sure. > > diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml > > index 9a651465d8..5b0b47b57b 100644 > > --- a/.gitlab-ci.d/opensbi.yml > > +++ b/.gitlab-ci.d/opensbi.yml > > @@ -42,17 +42,15 @@ > > docker-opensbi: > > extends: .opensbi_job_rules > > stage: containers > > - image: docker:stable > > + image: docker:20.10.16 > > services: > > - - docker:stable-dind > > + - docker:20.10.16-dind > > Can you elaborate on this ? I know the docs about use that particular > version tag, but they don't appear to explain why. If this is not > actually a hard requirements, we should keep using the stable tag. Yes, we can keep using "stable". Then, we should be ready to address future issues that may arise from "stable" not being compatible with the runner. > > variables: > > GIT_DEPTH: 3 > > IMAGE_TAG: $CI_REGISTRY_IMAGE:opensbi-cross-build > > - # We don't use TLS > > - DOCKER_HOST: tcp://docker:2375 > > - DOCKER_TLS_CERTDIR: "" > > So IIUC, this was always redundant when using gitlab CI. We should just > remove these in a standalone commit. Okay, I'll put this in a separate commit. > > before_script: > > - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD > > $CI_REGISTRY > > + - until docker info; do sleep 1; done > > Was this really needed ? The docs don't show that, and docker login is > synchronous, so I wouldn't expect us to them poll on 'docker info'. Unfortunately, yes. We need to wait until the "docker info" command is successful. This ensures that the Docker server has started and the subsequent docker commands won't fail. > In container-template.yml we in fact do the reverse > > - docker info > - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p > "$CI_REGISTRY_PASSWORD" About "docker login", as far as I understand it's a client-only command. It doesn't involve the Docker server at all. These two commands are not related to each other, it doesn't matter if "docker login" runs before or after "docker info". > imho best make this opensbi.yml file match contanier-template.yml, and > could be part of the same cleanup commit that removes thhose two docker > env vars. You mean to replace the "docker-opensbi" job in the "opensbi.yml" file with the same as the ".container_job_template" from the "container-template.yml" file? These two look too much different to me. I think we need to keep both. > > script: > > - docker pull $IMAGE_TAG || true > > - docker build --cache-from $IMAGE_TAG --tag > > $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA > > diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml > > index a7ed447fe4..57b175f5c2 100644 > > --- a/.gitlab-ci.d/qemu-project.yml > > +++ b/.gitlab-ci.d/qemu-project.yml > > @@ -1,7 +1,24 @@ > > # This file contains the set of jobs run by the QEMU project: > > # https://gitlab.com/qemu-project/qemu/-/pipelines > > > > +variables: > > + RUNNER_TAG: "" > > + > > +workflow: > > + rules: > > + # Set additional variables when running on Kubernetes. > > + # https://wiki.qemu.org/Testing/CI/KubernetesRunners > > + - if: $RUNNER_TAG == "k8s" > > + variables: > > + DOCKER_HOST: tcp://docker:2376 > > + DOCKER_TLS_CERTDIR: "/certs" > > + DOCKER_TLS_VERIFY: 1 > > + DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client" > > Is there anyway we can get the runner itself to set these > correctly by default ? Yes, the runner can set environment variables for its jobs. My concern here is that over time we lose visibility of these customizations if we put them in the runner configuration. This can be solved by having a repo in the qemu-project namespace to host the runner configuration, something like I did here: https://gitlab.com/spotlesstofu/qemu-ci-kubernetes. > IMHO the ideal would be that the k8s runners are registerd with the > qemu project to run *any* jobs without requiring tags. That way the > runners will "just work" when share runners are unavailable/exhausted, > like we have with Eldon's runner The problem here is that the Kubernetes (k8s) runner can't run windows jobs at the moment. If we wait for the shared runners to be exhausted, those few windows jobs in the pipeline won't be able to run. > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
