Good one Zach. Yeah, I also played with building separately and combining the resulting image (`regctl` is a great tool that might help with it).
But I found it quite brittle and "manual" - I really like the buildx approach when you can have multiple builder instances running in parallel. And ssh-forwarding of docker sockets is a cool thing :). I even used reverse ssh port forwarding to docker on my Mac to build images in parallel when I ssh-ed from my Mac M1 to my home desktop linux and run the buildx command on Linux to build both images (this triggered the build back on my Mac). That was fun :) J. On Tue, Dec 6, 2022 at 5:48 PM Zach Hoffman <zrhoff...@apache.org> wrote: > Never mind, the first link was better > > -Zach > > On Tue, Dec 6, 2022 at 9:46 AM Zach Hoffman <zrhoff...@apache.org> wrote: > > > Oops, the container link got mangled, it was supposed to be < > > > https://github.com/apache/trafficcontrol/pkgs/container/trafficcontrol/ci/trafficserver-alpine > > >. > > > > On Tue, Dec 6, 2022 at 9:38 AM Zach Hoffman <zrhoff...@apache.org> > wrote: > > > >> in Traffic Control, we build an Alpine image < > >> > https://github.com/apache/trafficcontrol/pkgs/container/trafficcontrol%2Fci%2Ftrafficserver-alpine > >, > >> with Traffic Server baked into it, for amd64 and arm64 . Rather than > >> building them both from the same machine, the amd64 and arm64 images in > >> separate GitHub Actions jobs, then combined using `docker manifest` in a > >> third GitHub Actions job < > >> > https://github.com/apache/trafficcontrol/blob/master/.github/workflows/container-trafficserver-alpine.yml > >> >. > >> > >> The arm64 part still takes much longer (over 2 hours, compare to 10 > >> minutes for amd64), but this approach eliminates the need to build for > more > >> than one platform at a time, so if the arm64 job were to run on an arm64 > >> runner in the future, that wouldn't need to further complicate the other > >> jobs in order to take advantage of that speedup. > >> > >> -Zach > >> > >> On Tue, Dec 6, 2022 at 9:02 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >> > >>> In Airflow we have a bit more complex setting (we are building 2x5x2 > >>> different images and they are different sets of them for different > >>> branches), Building images for Airflow takes quite some time > (installing > >>> many dependencies) so qemu was out of the question (several hours to > >>> build > >>> single image). Unfortunately qemu is ~ 15 times (!) slower than > hardware > >>> builds from our experience. Good enough for small images, but > >>> unacceptable > >>> if your image normally takes time > >>> > >>> The built-in actions like the one mentioned by Jacob have the > limitations > >>> that they depende on qemu - I have not yet found an easy "Action" that > >>> could utilise AMD and ARM hardware together. But maybe there are other > >>> ways. > >>> > >>> We developed our own scripting and tooling: > >>> > >>> * we have self-hosted runners in GitHub and we only use those to build > >>> images (with Astronomer money and Amazon-sponsored credits on AWS) > >>> * for the time when we build multi-platform build we start ARM > instances > >>> (they have built-in timed auto-kill-switch so that they are not run > >>> indefinitely and take resources) > >>> * we configured ssh-forwarded docker port from our AMD instance to > >>> forward > >>> docker socket > >>> * we configured two builder with buildx (local for AMD and the > forwarded > >>> one for ARM) > >>> * we run multi-platform buildx build to use both builders. > >>> * we have our own Python build/development environment (breeze) that > >>> wraps > >>> the docker commands that are executed so the "code" doing it is not > easy > >>> to > >>> copy elsewhere sometimes, but I copied below the actual "meat" what > >>> happens > >>> under the hood. > >>> * we run it as part of our Github Action workflows > >>> > >>> Examples: > >>> > >>> * Starting ARM instance and configuring builders: > >>> > https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260757 > >>> > >>> This is the relevant part of establishing the worker for buildx (I omit > >>> starting the instance as it is amazon-specific): > >>> > >>> export AUTOSSH_LOGFILE="${WORKING_DIR}/autossh.log" > >>> autossh -f "-L12357:/var/run/docker.sock" \ > >>> -N -o "IdentitiesOnly=yes" -o "StrictHostKeyChecking=no" \ > >>> -i "${WORKING_DIR}/my_key" "${EC2_USER}@ > >>> ${INSTANCE_PRIVATE_DNS_NAME}" > >>> > >>> bash -c 'echo -n "Waiting port 12357 .."; for _ in `seq 1 40`; do echo > -n > >>> .; sleep 0.25; nc -z localhost 12357 && echo " Open." && exit ; done; > >>> echo > >>> " Timeout!" >&2; exit 1' > >>> > >>> docker buildx rm --force airflow_cache || true > >>> docker buildx create --name airflow_cache > >>> docker buildx create --name airflow_cache --append localhost:12357 > >>> > >>> * Releasing image: > >>> > https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260776 > >>> > >>> This is the command that is run under-the-hood > >>> > >>> docker buildx build --builder airflow_cache --build-arg > >>> PYTHON_BASE_IMAGE=python:3.10-slim-bullseye --build-arg > >>> AIRFLOW_VERSION=2.5.0 --platform linux/amd64,linux/arm64 . -t > >>> apache/airflow:2.5.0-python3.10 --push > >>> > >>> J. > >>> > >>> > >>> > >>> On Tue, Dec 6, 2022 at 2:10 PM Jacob Wujciak > >>> <ja...@voltrondata.com.invalid> > >>> wrote: > >>> > >>> > Hello Robert, > >>> > > >>> > I would suggest using GItHub Actions, there you can use the official > >>> suite > >>> > of docker actions to build multiplatform images with little need for > >>> custom > >>> > scripting [1]. > >>> > Feel free to ping me in the ASF slack. > >>> > > >>> > Best > >>> > Jacob > >>> > > >>> > [1]: https://github.com/docker/build-push-action > >>> > > >>> > On Tue, Dec 6, 2022 at 1:43 PM Robert Munteanu <romb...@apache.org> > >>> wrote: > >>> > > >>> > > Hi, > >>> > > > >>> > > We had a user report that our official Docker image does not > support > >>> > > architectures other than AMD64 [1]. M1 Macs and Raspberry Pis can't > >>> run > >>> > > the image with our current setup. > >>> > > > >>> > > On our side, we set up automated builds on Docker Hub using > automated > >>> > > builds. Unforunately, Docker Hub autobuilds don't support `docker > >>> > > buildx` or another form of multi-arch builds. It is not the roadmap > >>> > > though [2], but here is no guarantee on when (or if) it will become > >>> > > available. > >>> > > > >>> > > I see two alternatives so far: > >>> > > > >>> > > 1. Moving to GitHub actions > >>> > > 2. Use hooks to install qemu and 'fake' a multi-arch build on > Docker > >>> > > Hub > >>> > > > >>> > > To be honest, none is too appealing to me, we have a simple process > >>> > > that works. > >>> > > > >>> > > How are other projects handling this? Or does anyone have any ideas > >>> > > that they can share? > >>> > > > >>> > > Thanks, > >>> > > Robert > >>> > > > >>> > > [1]: https://issues.apache.org/jira/browse/SLING-11714 > >>> > > [2]: https://github.com/docker/roadmap/issues/109 > >>> > > > >>> > > >>> > >> >
