And also another comment, If this is a concern only for DockerHub as a "CI" workflow, for now, I can actually recommend the Github Docker Registry (even if it is technically deprecated). It does a great job for us at Airflow.
We build our images during CI only once and push them to the Github Registry and use it from them for the rest of our CI workflow. This saved us a lot of build time already and gave full reproducibility of CI failures by downloading the images (we even print the exact command how to pull and use the very image that was used on CI - with all the dependencies, so it takes a minute or two to be able to reproduce the CI failure locally. I even prepared this nice document describing it: https://cwiki.apache.org/confluence/display/INFRA/Github+Actions+to+DockerHub in the INFRA wiki. In this scenario, all the "busy" stuff is done in the private per-project Github Docker Registry. All the CI builds are pulling cache images from there, we only periodically (nightly) push tags in our repository that triggers DockerHub rebuilding the images. So from the CI perspective, we are completely safe at Airflow. A refreshed version of the whole CI approach of Airflow is here: including some sequence diagrams: https://github.com/apache/airflow/blob/master/CI.rst . I am going to update it shortly with the "matrix builds" optimization we are testing now (information about it follows shortly in the separate thread). J. On Thu, Oct 29, 2020 at 10:10 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > >> As I mentioned in the referenced ticket, the Github Container Registry >> (“GCR”) is not something Infra can immediately support due to Github’s >> limitations. The Board collectively with Infra needs to determine how or if >> the GCR fits our needs from both operational and fiscal perspectives. >> Additionally, Infra does not have the ability to enable this beta feature >> at a repository level, it is an org-wide setting. Infra appreciates that >> this is of interest to some projects, and will continue to review it. >> >> > Of course. Just wanted to mention that as a viable option. > > >> >> -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
