On Fri, Feb 24, 2023 at 06:08:57PM +0000, Alex Bennée wrote:
> Apparently the docker-in-docker approach has some flaws including
> needing privileged mode to run and being quite slow. An alternative
> approach is to use Google's kaniko tool. It also works across
> different gitlab executors.
Interesting, I've not come across this tool before.
> Following the gitlab example code we drop all the direct docker calls
> and usage of the script and make a direct call to kaniko and hope the
> images are cacheable by others.
>
> Signed-off-by: Alex Bennée <alex.ben...@linaro.org>
> ---
> .gitlab-ci.d/container-template.yml | 23 +++++++++--------------
> 1 file changed, 9 insertions(+), 14 deletions(-)
>
> diff --git a/.gitlab-ci.d/container-template.yml
> b/.gitlab-ci.d/container-template.yml
> index c434b9c8f3..c2d7950df8 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml
> @@ -1,22 +1,17 @@
> .container_job_template:
> extends: .base_job_template
> - image: docker:stable
> + image:
> + name: gcr.io/kaniko-project/executor:v1.9.0-debug
> + entrypoint: [""]
> stage: containers
> - services:
> - - docker:dind
> before_script:
> - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
> - - apk add python3
> - - docker info
> - - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p
> "$CI_REGISTRY_PASSWORD"
> + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
> script:
> - echo "TAG:$TAG"
> - echo "COMMON_TAG:$COMMON_TAG"
> - - ./tests/docker/docker.py --engine docker build
> - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
> - -r $CI_REGISTRY/qemu-project/qemu
> - - docker tag "qemu/$NAME" "$TAG"
> - - docker push "$TAG"
> - after_script:
> - - docker logout
> + - /kaniko/executor
> + --reproducible
> + --context "${CI_PROJECT_DIR}"
> + --dockerfile
> "${CI_PROJECT_DIR}/tests/docker/dockerfiles/$NAME.docker"
> + --destination "${TAG}"
AFAICT from reading the docs, this does not allow for caching of layers
from the previous built image, so will always do a from scratch build.
This feels like it would kill any performance benefits over docker.
https://github.com/GoogleContainerTools/kaniko#caching
"Users can opt into caching by setting the --cache=true flag. A
remote repository for storing cached layers can be provided via
the --cache-repo flag. If this flag isn't provided, a cached
repo will be inferred from the --destination provided."
So it would sound like we need --cache=true adding to this command,
and in theory it should prime its cache from ${TAG}, but i'm not
entirely confident in that without testing. Also does not seem
like its psosible to have the dual cache COMMON_TAG + TAG, which
is particularly important for forks, since COMMON_TAG is more
likely to be up2date.
We could live with the latter restriction if we *always* used
COMMON_TAG as the cache, because 99% of the time forks are
going to just be using dockerfiles that match upstream. Only
those few contributors who have a branch that's changing
the dockerfiles would benefit from pointing the cache at
their fork.
So we could achieve that I believe using
- /kaniko/executor
--reproducible
--context "${CI_PROJECT_DIR}"
--cache=true
--cache-repo "${COMMON_TAG}"
--dockerfile
"${CI_PROJECT_DIR}/tests/docker/dockerfiles/$NAME.docker"
--destination "${TAG}"
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
