On Fri, Apr 07, 2023 at 03:52:51PM +0100, Camilla Conte wrote:
> Configure Gitlab CI to run on Kubernetes
> according to the official documentation.
> https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#docker-in-docker-with-tls-enabled-in-kubernetes
>
> These changes are needed because of the CI jobs
> using Docker-in-Docker (dind).
> As soon as Docker-in-Docker is replaced with Kaniko,
> these changes can be reverted.
>
> I documented what I did to set up the Kubernetes runner on the wiki:
> https://wiki.qemu.org/Testing/CI/KubernetesRunners
>
> Signed-off-by: Camilla Conte <cco...@redhat.com>
> ---
> .gitlab-ci.d/container-template.yml | 6 +++---
> .gitlab-ci.d/default.yml | 3 +++
> .gitlab-ci.d/opensbi.yml | 8 +++-----
> .gitlab-ci.d/qemu-project.yml | 17 +++++++++++++++++
> 4 files changed, 26 insertions(+), 8 deletions(-)
> create mode 100644 .gitlab-ci.d/default.yml
>
> diff --git a/.gitlab-ci.d/container-template.yml
> b/.gitlab-ci.d/container-template.yml
> index 519b8a9482..f55a954741 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml
> @@ -1,14 +1,14 @@
> .container_job_template:
> extends: .base_job_template
> - image: docker:stable
> + image: docker:20.10.16
> stage: containers
> services:
> - - docker:dind
> + - docker:20.10.16-dind
> before_script:
> - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
> - apk add python3
> - - docker info
> + - until docker info; do sleep 1; done
> - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p
> "$CI_REGISTRY_PASSWORD"
> script:
> - echo "TAG:$TAG"
> diff --git a/.gitlab-ci.d/default.yml b/.gitlab-ci.d/default.yml
> new file mode 100644
> index 0000000000..292be8b91c
> --- /dev/null
> +++ b/.gitlab-ci.d/default.yml
> @@ -0,0 +1,3 @@
> +default:
> + tags:
> + - $RUNNER_TAG
Can we just put this in base.yml instead of creating a new file.
> diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
> index 9a651465d8..5b0b47b57b 100644
> --- a/.gitlab-ci.d/opensbi.yml
> +++ b/.gitlab-ci.d/opensbi.yml
> @@ -42,17 +42,15 @@
> docker-opensbi:
> extends: .opensbi_job_rules
> stage: containers
> - image: docker:stable
> + image: docker:20.10.16
> services:
> - - docker:stable-dind
> + - docker:20.10.16-dind
Can you elaborate on this ? I know the docs about use that particular
version tag, but they don't appear to explain why. If this is not
actually a hard requirements, we should keep using the stable tag.
> variables:
> GIT_DEPTH: 3
> IMAGE_TAG: $CI_REGISTRY_IMAGE:opensbi-cross-build
> - # We don't use TLS
> - DOCKER_HOST: tcp://docker:2375
> - DOCKER_TLS_CERTDIR: ""
So IIUC, this was always redundant when using gitlab CI. We should just
remove these in a standalone commit.
> before_script:
> - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
> + - until docker info; do sleep 1; done
Was this really needed ? The docs don't show that, and docker login is
synchronous, so I wouldn't expect us to them poll on 'docker info'.
In container-template.yml we in fact do the reverse
- docker info
- docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p
"$CI_REGISTRY_PASSWORD"
imho best make this opensbi.yml file match contanier-template.yml, and
could be part of the same cleanup commit that removes thhose two docker
env vars.
> script:
> - docker pull $IMAGE_TAG || true
> - docker build --cache-from $IMAGE_TAG --tag
> $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
> diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
> index a7ed447fe4..57b175f5c2 100644
> --- a/.gitlab-ci.d/qemu-project.yml
> +++ b/.gitlab-ci.d/qemu-project.yml
> @@ -1,7 +1,24 @@
> # This file contains the set of jobs run by the QEMU project:
> # https://gitlab.com/qemu-project/qemu/-/pipelines
>
> +variables:
> + RUNNER_TAG: ""
> +
> +workflow:
> + rules:
> + # Set additional variables when running on Kubernetes.
> + # https://wiki.qemu.org/Testing/CI/KubernetesRunners
> + - if: $RUNNER_TAG == "k8s"
> + variables:
> + DOCKER_HOST: tcp://docker:2376
> + DOCKER_TLS_CERTDIR: "/certs"
> + DOCKER_TLS_VERIFY: 1
> + DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
Is there anyway we can get the runner itself to set these
correctly by default ?
IMHO the ideal would be that the k8s runners are registerd with the
qemu project to run *any* jobs without requiring tags. That way the
runners will "just work" when share runners are unavailable/exhausted,
like we have with Eldon's runner
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
