ilya-biryukov created this revision.
ilya-biryukov added reviewers: mehdi_amini, klimek.
Herald added a subscriber: llvm-commits.
We previously emulated multi-staged builds using two dockerfiles,
native support from Docker allows us to merge them into one,
simplifying our scripts.
For more details about multi-stage builds, see:
https://docs.docker.com/develop/develop-images/multistage-build/
Repository:
rL LLVM
https://reviews.llvm.org/D44787
Files:
docs/Docker.rst
utils/docker/build_docker_image.sh
utils/docker/debian8/Dockerfile
utils/docker/debian8/build/Dockerfile
utils/docker/debian8/release/Dockerfile
utils/docker/example/Dockerfile
utils/docker/example/build/Dockerfile
utils/docker/example/release/Dockerfile
utils/docker/nvidia-cuda/Dockerfile
utils/docker/nvidia-cuda/build/Dockerfile
utils/docker/nvidia-cuda/release/Dockerfile
utils/docker/scripts/build_install_llvm.sh
Index: utils/docker/scripts/build_install_llvm.sh
===================================================================
--- utils/docker/scripts/build_install_llvm.sh
+++ utils/docker/scripts/build_install_llvm.sh
@@ -16,8 +16,8 @@
Checkout svn sources and run cmake with the specified arguments. Used
inside docker container.
-Passes additional -DCMAKE_INSTALL_PREFIX and archives the contents of
-the directory to /tmp/clang.tar.gz.
+Passes additional -DCMAKE_INSTALL_PREFIX and puts the build results into
+/tmp/clang-install/ directory.
Available options:
-h|--help show this help message
@@ -244,12 +244,7 @@
popd
-# Pack the installed clang into an archive.
-echo "Archiving clang installation to /tmp/clang.tar.gz"
-cd "$CLANG_INSTALL_DIR"
-tar -czf /tmp/clang.tar.gz *
-
# Cleanup.
-rm -rf "$CLANG_BUILD_DIR" "$CLANG_INSTALL_DIR"
+rm -rf "$CLANG_BUILD_DIR"
echo "Done"
Index: utils/docker/nvidia-cuda/release/Dockerfile
===================================================================
--- utils/docker/nvidia-cuda/release/Dockerfile
+++ /dev/null
@@ -1,23 +0,0 @@
-#===- llvm/utils/docker/nvidia-cuda/release/Dockerfile -------------------===//
-#
-# The LLVM Compiler Infrastructure
-#
-# This file is distributed under the University of Illinois Open Source
-# License. See LICENSE.TXT for details.
-#
-#===----------------------------------------------------------------------===//
-# This is an example Dockerfile that copies a clang installation, compiled
-# by the 'build/' container into a fresh docker image to get a container of
-# minimal size.
-# Replace FIXMEs to prepare a new Dockerfile.
-
-# FIXME: Replace 'ubuntu' with your base image.
-FROM nvidia/cuda:8.0-devel
-
-# FIXME: Change maintainer name.
-LABEL maintainer "LLVM Developers"
-
-# Unpack clang installation into this container.
-ADD clang.tar.gz /usr/local/
-
-# C++ standard library and binutils are already included in the base package.
Index: utils/docker/nvidia-cuda/Dockerfile
===================================================================
--- utils/docker/nvidia-cuda/Dockerfile
+++ utils/docker/nvidia-cuda/Dockerfile
@@ -6,26 +6,26 @@
# License. See LICENSE.TXT for details.
#
#===----------------------------------------------------------------------===//
-# Produces an image that compiles and archives clang, based on nvidia/cuda
-# image.
-FROM nvidia/cuda:8.0-devel
-
+# Stage 1. Check out LLVM source code and run the build.
+FROM nvidia/cuda:8.0-devel as builder
LABEL maintainer "LLVM Developers"
-
-# Arguments to pass to build_install_clang.sh.
-ARG buildscript_args
-
# Install llvm build dependencies.
RUN apt-get update && \
apt-get install -y --no-install-recommends ca-certificates cmake python \
- subversion ninja-build && \
+ subversion ninja-build && \
rm -rf /var/lib/apt/lists/*
ADD checksums /tmp/checksums
ADD scripts /tmp/scripts
-
# Arguments passed to build_install_clang.sh.
ARG buildscript_args
-
-# Run the build. Results of the build will be available as /tmp/clang.tar.gz.
+# Run the build. Results of the build will be available at /tmp/clang-install/.
RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
+
+
+# Stage 2. Produce a minimal release image with build results.
+FROM nvidia/cuda:8.0-devel
+LABEL maintainer "LLVM Developers"
+# Copy clang installation into this container.
+COPY --from=builder /tmp/clang-install/ /usr/local/
+# C++ standard library and binutils are already included in the base package.
Index: utils/docker/example/release/Dockerfile
===================================================================
--- utils/docker/example/release/Dockerfile
+++ /dev/null
@@ -1,24 +0,0 @@
-#===- llvm/utils/docker/example/release/Dockerfile -----------------------===//
-#
-# The LLVM Compiler Infrastructure
-#
-# This file is distributed under the University of Illinois Open Source
-# License. See LICENSE.TXT for details.
-#
-#===----------------------------------------------------------------------===//
-# An image that unpacks a clang installation, compiled by the 'build/'
-# container.
-# Replace FIXMEs to prepare your own image.
-
-# FIXME: Replace 'ubuntu' with your base image.
-FROM ubuntu
-
-# FIXME: Change maintainer name.
-LABEL maintainer "Maintainer <maintainer@email>"
-
-# FIXME: Install all packages you want to have in your release container.
-# A minimal useful installation must include libstdc++ and binutils.
-
-# Unpack clang installation into this container.
-# It is copied to this directory by build_docker_image.sh script.
-ADD clang.tar.gz /usr/local/
Index: utils/docker/example/Dockerfile
===================================================================
--- utils/docker/example/Dockerfile
+++ utils/docker/example/Dockerfile
@@ -9,20 +9,29 @@
# This is an example Dockerfile to build an image that compiles clang.
# Replace FIXMEs to prepare your own image.
+# Stage 1. Check out LLVM source code and run the build.
# FIXME: Replace 'ubuntu' with your base image
-FROM ubuntu
-
+FROM ubuntu as builder
# FIXME: Change maintainer name
LABEL maintainer "Maintainer <maintainer@email>"
-
-# FIXME: Install llvm/clang build dependencies. Including compiler to
+# FIXME: Install llvm/clang build dependencies here. Including compiler to
# build stage1, cmake, subversion, ninja, etc.
ADD checksums /tmp/checksums
ADD scripts /tmp/scripts
-
# Arguments passed to build_install_clang.sh.
ARG buildscript_args
-
-# Run the build. Results of the build will be available as /tmp/clang.tar.gz.
+# Run the build. Results of the build will be available as /tmp/clang-install.
RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
+
+
+# Stage 2. Produce a minimal release image with build results.
+# FIXME: Replace 'ubuntu' with your base image.
+FROM ubuntu
+# FIXME: Change maintainer name.
+LABEL maintainer "Maintainer <maintainer@email>"
+# FIXME: Install all packages you want to have in your release container.
+# A minimal useful installation should include at least libstdc++ and binutils.
+
+# Copy build results of stage 1 to /usr/local.
+COPY --from=builder /tmp/clang-install/ /usr/local/
Index: utils/docker/debian8/release/Dockerfile
===================================================================
--- utils/docker/debian8/release/Dockerfile
+++ /dev/null
@@ -1,21 +0,0 @@
-#===- llvm/utils/docker/debian8/release/Dockerfile -----------------------===//
-#
-# The LLVM Compiler Infrastructure
-#
-# This file is distributed under the University of Illinois Open Source
-# License. See LICENSE.TXT for details.
-#
-#===----------------------------------------------------------------------===//
-# A release image, containing clang installation, produced by the 'build/' image
-# and adding libstdc++ and binutils.
-FROM launcher.gcr.io/google/debian8:latest
-
-LABEL maintainer "LLVM Developers"
-
-# Install packages for minimal useful image.
-RUN apt-get update && \
- apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \
- rm -rf /var/lib/apt/lists/*
-
-# Unpack clang installation into this image.
-ADD clang.tar.gz /usr/local/
Index: utils/docker/debian8/Dockerfile
===================================================================
--- utils/docker/debian8/Dockerfile
+++ utils/docker/debian8/Dockerfile
@@ -6,33 +6,27 @@
# License. See LICENSE.TXT for details.
#
#===----------------------------------------------------------------------===//
-# Produces an image that compiles and archives clang, based on debian8.
-FROM launcher.gcr.io/google/debian8:latest
-
+# Stage 1. Check out LLVM source code and run the build.
+FROM launcher.gcr.io/google/debian8:latest as builder
LABEL maintainer "LLVM Developers"
-
# Install build dependencies of llvm.
# First, Update the apt's source list and include the sources of the packages.
RUN grep deb /etc/apt/sources.list | \
sed 's/^deb/deb-src /g' >> /etc/apt/sources.list
-
# Install compiler, python and subversion.
RUN apt-get update && \
apt-get install -y --no-install-recommends ca-certificates gnupg \
build-essential python wget subversion unzip && \
rm -rf /var/lib/apt/lists/*
-
# Install a newer ninja release. It seems the older version in the debian repos
# randomly crashes when compiling llvm.
RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip"; && \
echo "d2fea9ff33b3ef353161ed906f260d565ca55b8ca0568fa07b1d2cab90a84a07 ninja-linux.zip" \
| sha256sum -c && \
unzip ninja-linux.zip -d /usr/local/bin && \
rm ninja-linux.zip
-
# Import public key required for verifying signature of cmake download.
RUN gpg --keyserver hkp://pgp.mit.edu --recv 0x2D2CEF1034921684
-
# Download, verify and install cmake version that can compile clang into /usr/local.
# (Version in debian8 repos is is too old)
RUN mkdir /tmp/cmake-install && cd /tmp/cmake-install && \
@@ -47,9 +41,18 @@
ADD checksums /tmp/checksums
ADD scripts /tmp/scripts
-
# Arguments passed to build_install_clang.sh.
ARG buildscript_args
-
-# Run the build. Results of the build will be available as /tmp/clang.tar.gz.
+# Run the build. Results of the build will be available at /tmp/clang-install/.
RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
+
+
+# Stage 2. Produce a minimal release image with build results.
+FROM launcher.gcr.io/google/debian8:latest
+LABEL maintainer "LLVM Developers"
+# Install packages for minimal useful image.
+RUN apt-get update && \
+ apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \
+ rm -rf /var/lib/apt/lists/*
+# Copy build results of stage 1 to /usr/local.
+COPY --from=builder /tmp/clang-install/ /usr/local/
Index: utils/docker/build_docker_image.sh
===================================================================
--- utils/docker/build_docker_image.sh
+++ utils/docker/build_docker_image.sh
@@ -163,19 +163,9 @@
DOCKER_TAG=":$DOCKER_TAG"
fi
-echo "Building from $IMAGE_SOURCE"
-echo "Building $DOCKER_REPOSITORY-build$DOCKER_TAG"
-docker build -t "$DOCKER_REPOSITORY-build$DOCKER_TAG" \
+echo "Building ${DOCKER_REPOSITORY}${DOCKER_TAG} from $IMAGE_SOURCE"
+docker build -t "${DOCKER_REPOSITORY}${DOCKER_TAG}" \
--build-arg "buildscript_args=$BUILDSCRIPT_ARGS" \
- -f "$BUILD_DIR/$IMAGE_SOURCE/build/Dockerfile" \
+ -f "$BUILD_DIR/$IMAGE_SOURCE/Dockerfile" \
"$BUILD_DIR"
-
-echo "Copying clang installation to release image sources"
-docker run -v "$BUILD_DIR/$IMAGE_SOURCE:/workspace" "$DOCKER_REPOSITORY-build$DOCKER_TAG" \
- cp /tmp/clang.tar.gz /workspace/release
-
-echo "Building release image"
-docker build -t "${DOCKER_REPOSITORY}${DOCKER_TAG}" \
- "$BUILD_DIR/$IMAGE_SOURCE/release"
-
echo "Done"
Index: docs/Docker.rst
===================================================================
--- docs/Docker.rst
+++ docs/Docker.rst
@@ -53,24 +53,15 @@
LLVM components, compiled from sources. The sources are checked out from the
upstream svn repository when building the image.
-Inside each subfolder we host Dockerfiles for two images:
-
-- ``build/`` image is used to compile LLVM, it installs a system compiler and all
- build dependencies of LLVM. After the build process is finished, the build
- image will have an archive with compiled components at ``/tmp/clang.tar.gz``.
-- ``release/`` image usually only contains LLVM components, compiled by the
- ``build/`` image, and also libstdc++ and binutils to make image minimally
- useful for C++ development. The assumption is that you usually want clang to
- be one of the provided components.
-
-To build both of those images, use ``build_docker_image.sh`` script.
-It will checkout LLVM sources and build clang in the ``build`` container, copy results
-of the build to the local filesystem and then build the ``release`` container using
-those. The ``build_docker_image.sh`` accepts a list of LLVM repositories to
-checkout, and arguments for CMake invocation.
+The resulting image contains only the requested LLVM components and a few extra
+packages to make the image minimally useful for C++ development, e.g. libstdc++
+and binutils.
+
+The interface to run the build is ``build_docker_image.sh`` script. It accepts a
+list of LLVM repositories to checkout and arguments for CMake invocation.
If you want to write your own docker image, start with an ``example/`` subfolder.
-It provides incomplete Dockerfiles with (very few) FIXMEs explaining the steps
+It provides incomplete Dockerfile with (very few) FIXMEs explaining the steps
you need to take in order to make your Dockerfiles functional.
Usage
@@ -110,10 +101,10 @@
-DBOOTSTRAP_CMAKE_BUILD_TYPE=Release \
-DCLANG_ENABLE_BOOTSTRAP=ON -DCLANG_BOOTSTRAP_TARGETS="install-clang;install-clang-headers"
-This will produce two images, a release image ``clang-debian8:staging`` and a
-build image ``clang-debian8-build:staging`` from the latest upstream revision.
-After the image is built you can run bash inside a container based on your
-image like this:
+This will produce a new image ``clang-debian8:staging`` from the latest
+upstream revision.
+After the image is built you can run bash inside a container based on your image
+like this:
.. code-block:: bash
@@ -181,19 +172,14 @@
Minimizing docker image size
============================
-Due to Docker restrictions we use two images (i.e., build and release folders)
-for the release image to be as small as possible. It's much easier to achieve
-that using two images, because Docker would store a filesystem layer for each
-command in the Dockerfile, i.e. if you install some packages in one command,
-then remove those in a separate command, the size of the resulting image will
-still be proportinal to the size of an image with installed packages.
-Therefore, we strive to provide a very simple release image which only copies
-compiled clang and does not do anything else.
-
-Docker 1.13 added a ``--squash`` flag that allows to flatten the layers of the
-image, i.e. remove the parts that were actually deleted. That is an easier way
-to produce the smallest images possible by using just a single image. We do not
-use it because as of today the flag is in experimental stage and not everyone
-may have the latest docker version available. When the flag is out of
-experimental stage, we should investigate replacing two images approach with
-just a single image, built using ``--squash`` flag.
+Due to how Docker's filesystem works, all intermediate writes are persisted in
+the resulting image, even if they are removed in the following commands.
+To minimize the resulting image size we use `multi-stage Docker builds
+<https://docs.docker.com/develop/develop-images/multistage-build/>`_.
+Internally Docker builds two images. The first image does all the work: installs
+build dependencies, checks out LLVM source code, compiles LLVM, etc.
+The first image is only used during build and does not have a descriptive name,
+i.e. it is only accessible via the hash value after the build is finished.
+The second image is our resulting image. It contains only the built binaries
+and not any build dependencies. It is also accessible via a descriptive name
+(specified by -d and -t flags).
_______________________________________________
cfe-commits mailing list
cfe-commits@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
