Hi Jarek, Zach, and Allen, That is a lot of good material, thank you. Looks like most people are using GH actions, so I'll take this to our dev list, we will probably do the same.
Thanks, Robert On Tue, 2022-12-06 at 18:56 +0100, Jarek Potiuk wrote: > Good one Zach. Yeah, I also played with building separately and > combining > the resulting image (`regctl` is a great tool that might help with > it). > > But I found it quite brittle and "manual" - I really like the buildx > approach when you can have multiple builder instances running in > parallel. > And ssh-forwarding of docker sockets is a cool thing :). I even used > reverse ssh port forwarding to docker on my Mac to build images in > parallel > when I ssh-ed from my Mac M1 to my home desktop linux and run the > buildx > command on Linux to build both images (this triggered the build back > on my > Mac). > > That was fun :) > > J. > > > > On Tue, Dec 6, 2022 at 5:48 PM Zach Hoffman <zrhoff...@apache.org> > wrote: > > > Never mind, the first link was better > > > > -Zach > > > > On Tue, Dec 6, 2022 at 9:46 AM Zach Hoffman <zrhoff...@apache.org> > > wrote: > > > > > Oops, the container link got mangled, it was supposed to be < > > > > > https://github.com/apache/trafficcontrol/pkgs/container/trafficcontrol/ci/trafficserver-alpine > > > > . > > > > > > On Tue, Dec 6, 2022 at 9:38 AM Zach Hoffman > > > <zrhoff...@apache.org> > > wrote: > > > > > > > in Traffic Control, we build an Alpine image < > > > > > > https://github.com/apache/trafficcontrol/pkgs/container/trafficcontrol%2Fci%2Ftrafficserver-alpine > > > , > > > > with Traffic Server baked into it, for amd64 and arm64 . Rather > > > > than > > > > building them both from the same machine, the amd64 and arm64 > > > > images in > > > > separate GitHub Actions jobs, then combined using `docker > > > > manifest` in a > > > > third GitHub Actions job < > > > > > > https://github.com/apache/trafficcontrol/blob/master/.github/workflows/container-trafficserver-alpine.yml > > > > > . > > > > > > > > The arm64 part still takes much longer (over 2 hours, compare > > > > to 10 > > > > minutes for amd64), but this approach eliminates the need to > > > > build for > > more > > > > than one platform at a time, so if the arm64 job were to run on > > > > an arm64 > > > > runner in the future, that wouldn't need to further complicate > > > > the other > > > > jobs in order to take advantage of that speedup. > > > > > > > > -Zach > > > > > > > > On Tue, Dec 6, 2022 at 9:02 AM Jarek Potiuk <ja...@potiuk.com> > > > > wrote: > > > > > > > > > In Airflow we have a bit more complex setting (we are > > > > > building 2x5x2 > > > > > different images and they are different sets of them for > > > > > different > > > > > branches), Building images for Airflow takes quite some time > > (installing > > > > > many dependencies) so qemu was out of the question (several > > > > > hours to > > > > > build > > > > > single image). Unfortunately qemu is ~ 15 times (!) slower > > > > > than > > hardware > > > > > builds from our experience. Good enough for small images, but > > > > > unacceptable > > > > > if your image normally takes time > > > > > > > > > > The built-in actions like the one mentioned by Jacob have the > > limitations > > > > > that they depende on qemu - I have not yet found an easy > > > > > "Action" that > > > > > could utilise AMD and ARM hardware together. But maybe there > > > > > are other > > > > > ways. > > > > > > > > > > We developed our own scripting and tooling: > > > > > > > > > > * we have self-hosted runners in GitHub and we only use those > > > > > to build > > > > > images (with Astronomer money and Amazon-sponsored credits on > > > > > AWS) > > > > > * for the time when we build multi-platform build we start > > > > > ARM > > instances > > > > > (they have built-in timed auto-kill-switch so that they are > > > > > not run > > > > > indefinitely and take resources) > > > > > * we configured ssh-forwarded docker port from our AMD > > > > > instance to > > > > > forward > > > > > docker socket > > > > > * we configured two builder with buildx (local for AMD and > > > > > the > > forwarded > > > > > one for ARM) > > > > > * we run multi-platform buildx build to use both builders. > > > > > * we have our own Python build/development environment > > > > > (breeze) that > > > > > wraps > > > > > the docker commands that are executed so the "code" doing it > > > > > is not > > easy > > > > > to > > > > > copy elsewhere sometimes, but I copied below the actual > > > > > "meat" what > > > > > happens > > > > > under the hood. > > > > > * we run it as part of our Github Action workflows > > > > > > > > > > Examples: > > > > > > > > > > * Starting ARM instance and configuring builders: > > > > > > > https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260757 > > > > > > > > > > This is the relevant part of establishing the worker for > > > > > buildx (I omit > > > > > starting the instance as it is amazon-specific): > > > > > > > > > > export AUTOSSH_LOGFILE="${WORKING_DIR}/autossh.log" > > > > > autossh -f "-L12357:/var/run/docker.sock" \ > > > > > -N -o "IdentitiesOnly=yes" -o "StrictHostKeyChecking=no" > > > > > \ > > > > > -i "${WORKING_DIR}/my_key" "${EC2_USER}@ > > > > > ${INSTANCE_PRIVATE_DNS_NAME}" > > > > > > > > > > bash -c 'echo -n "Waiting port 12357 .."; for _ in `seq 1 > > > > > 40`; do echo > > -n > > > > > .; sleep 0.25; nc -z localhost 12357 && echo " Open." && exit > > > > > ; done; > > > > > echo > > > > > " Timeout!" >&2; exit 1' > > > > > > > > > > docker buildx rm --force airflow_cache || true > > > > > docker buildx create --name airflow_cache > > > > > docker buildx create --name airflow_cache --append > > > > > localhost:12357 > > > > > > > > > > * Releasing image: > > > > > > > https://github.com/apache/airflow/actions/runs/3603432288/jobs/6080260776 > > > > > > > > > > This is the command that is run under-the-hood > > > > > > > > > > docker buildx build --builder airflow_cache --build-arg > > > > > PYTHON_BASE_IMAGE=python:3.10-slim-bullseye --build-arg > > > > > AIRFLOW_VERSION=2.5.0 --platform linux/amd64,linux/arm64 . -t > > > > > apache/airflow:2.5.0-python3.10 --push > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > On Tue, Dec 6, 2022 at 2:10 PM Jacob Wujciak > > > > > <ja...@voltrondata.com.invalid> > > > > > wrote: > > > > > > > > > > > Hello Robert, > > > > > > > > > > > > I would suggest using GItHub Actions, there you can use the > > > > > > official > > > > > suite > > > > > > of docker actions to build multiplatform images with little > > > > > > need for > > > > > custom > > > > > > scripting [1]. > > > > > > Feel free to ping me in the ASF slack. > > > > > > > > > > > > Best > > > > > > Jacob > > > > > > > > > > > > [1]: https://github.com/docker/build-push-action > > > > > > > > > > > > On Tue, Dec 6, 2022 at 1:43 PM Robert Munteanu > > > > > > <romb...@apache.org> > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > We had a user report that our official Docker image does > > > > > > > not > > support > > > > > > > architectures other than AMD64 [1]. M1 Macs and Raspberry > > > > > > > Pis can't > > > > > run > > > > > > > the image with our current setup. > > > > > > > > > > > > > > On our side, we set up automated builds on Docker Hub > > > > > > > using > > automated > > > > > > > builds. Unforunately, Docker Hub autobuilds don't support > > > > > > > `docker > > > > > > > buildx` or another form of multi-arch builds. It is not > > > > > > > the roadmap > > > > > > > though [2], but here is no guarantee on when (or if) it > > > > > > > will become > > > > > > > available. > > > > > > > > > > > > > > I see two alternatives so far: > > > > > > > > > > > > > > 1. Moving to GitHub actions > > > > > > > 2. Use hooks to install qemu and 'fake' a multi-arch > > > > > > > build on > > Docker > > > > > > > Hub > > > > > > > > > > > > > > To be honest, none is too appealing to me, we have a > > > > > > > simple process > > > > > > > that works. > > > > > > > > > > > > > > How are other projects handling this? Or does anyone have > > > > > > > any ideas > > > > > > > that they can share? > > > > > > > > > > > > > > Thanks, > > > > > > > Robert > > > > > > > > > > > > > > [1]: https://issues.apache.org/jira/browse/SLING-11714 > > > > > > > [2]: https://github.com/docker/roadmap/issues/109 > > > > > > > > > > > > > > > > > > > > > > > >
