Hello Everyone, TL;DR; I have a proposal regarding using the Airflow CI/Prod images generation for our daily builds/PR/local development. This is mostly internal detail on the code/image registry - the development workflow will remain practically the same with my proposal.
Context: Co far we have used DockerHub as a temporary cache for images (CI and Prod) that were build during the PR/master CI runs. This caused us some problems as the number of builds we had exceeded Docker's capacity and I had to implement some workarounds (which I am not happy with - skipping builds on DockerHub when they are < 10 hours old. However, with the new GIthubActions approach, we can use the internal registry of GitHub for build image caching (I tested it and it works as expected in https://github.com/apache/airflow/pull/8393 ) Unfortunately, GithubRegistry requires authentication (please comment on that thread https://github.community/t5/GitHub-Actions/docker-pull-from-public-GitHub-Package-Registry-fail-with-quot/m-p/52505/highlight/false#M8536) so we cannot use it for local development/breeze easily (everyone will have to authenticate with the registry ???). So we have to keep DockerHub for regular snapshots of images for local development. Proposal: I think that opens us an interesting possibility once we move everything to GA: 1) We use Github Registry for all CI image caches - that will make them always "fresh" 2) Use DockerHub to keep released images (1.10.10, 2.0.0 in the future ...) 3) We have daily TAGs applied automatically by CRON jobs in github that will be daily "snapshots" of Airflow code and images and we use those daily snapshots for local development/Breeze. This way we can trigger the builds in Dockerhub based on TAGs rather than on every master merge as we have now. This will add number of tags we have in our repo (but they will be well organised - they will follow for example 'daily-master-ci-2020-04-21' for CI image and `daily-master-2020-04-21` and `daily-master-build-2020-04-21` Such daily tags/snapshots are usual approach for many projects. Plus they will give us the opportunity to track down history a bit easier. WDYT? J. -- Jarek Potiuk Polidea | Principal Software Engineer M: +48 660 796 129
