> I realize dockerHub is the center of the docker ecosystem at this point being the primary registry. Would providing alternative registries help us in any way? I believe our jFrog artifactory instance can act as a docker registry.
I think for now we can get by with 2 users (I will let you know in a JIRA issue) We already have a free, extremely reliable, public GithubRegistry for that and we already use it for our development. It has no limitations of DockerHub for pulls (which do not affect ASF luckily) and public access with automated API, management, retention and great integration with the GitHub repos. So I think not having an alternative is not our problem. But I think nothing can really rival the "user facing" `docker pull apache/airflow`. Or when we get official dockerhub status (which I planned to) `docker pull apache-airflow`. I think `docker pull ghcr.io/apache/airlflow` that we could do is as close to that as it can get and we could always use it as an alternative, but I think keeping the "registry-less" version is strongly preferred. However indeed an apache registry with "independence" from external vendors (unless we depend on JFrog as service) might be appealing. Last two years we've seen very disturbing moves from Docker: * increasing rate limits for unregistered users (which I understand ASF is exempt from) https://www.docker.com/increase-rate-limits - November 20, 2020 * disabling autobuilds for free accounts (which also I understand ASF is exempt from) https://www.docker.com/blog/changes-to-docker-hub-autobuilds/, June 18, 2021 * limiting "used to be free" Docker Desktop usage to small companies only - https://www.docker.com/blog/updating-product-subscriptions/ August 2021 What would be the next move? Will the ASF be exempt? Who knows. Relying on it too heavily is a bad idea. I can raise the question to our PMC and later to the community. Important question - what would be the "URL" we can use, is there a possibility for a really short name like: `cr.apache.org/airlfow` ? Anything convoluted and long (and especially requiring long paths) would be a strong disadvantage. There are some problems if the image names contain "/" and needs to be sometimes %-encoded (we see that in ghcr.io) so if we could simply get <DOMAIN-IN_APACHE>/airflow - that would be absolutely doable to switch to it. J. On Fri, Jan 21, 2022 at 6:36 PM Chris Lambertus <c...@apache.org> wrote: > > Unfortunately, if most projects follow a similar workflow, we will be > unable to sustain this given the limitations of dockerHub. We have a 200 > user max, and 300+ projects. > > I've not been impressed with their support either, they cannot even tell > us a 'last accessed' date for the users in our Org. > > I realize dockerHub is the center of the docker ecosystem at this point > being the primary registry. Would providing alternative registries help us > in any way? I believe our jFrog artifactory instance can act as a docker > registry. > > -Chris > > > > > > On Jan 21, 2022, at 9:30 AM, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > We are pushing DockerHub images using personal accounts in Airflow. I > > believe we have three "active" users - Kaxil, Jarek, Ash, likely some > more > > seasoned PMC members (those can be removed), Basically everyone who is a > > member of the "airflow" team in DockerHub. > > But we also would like to add one or two more people as we have several > > people who can assume the release manager role. > > > > Is there a way we can keep the workflow where several release managers > > prepare (and sign) images manually and push the images there? > > > > I also have a comment about autobuilds - I heartily recommend not to use > > them and especially not to rely on them if you rely on fast and reliable > > providing images to your users. > > > > We used autobuilds in the past but they are terribly slow and unreliable > > (we had multiple cases where the autobuilds were hanging and actually > > blocking our release). Sometimes the queue was not cleared for days > without > > help or response from DockerHub. > > > > * GitHub issue with 48 comments about stuck/pending builds : > > https://github.com/docker/hub-feedback/issues/1935 (last one Dec 2021) > > * Builds stuck in pending: > > https://forums.docker.com/t/build-stuck-in-pending-on-docker-hub/1360 > > * Slow builds: > > > https://forums.docker.com/t/why-does-it-take-so-long-for-the-docker-hub-automated-builds-to-upload-the-built-image/1653/29 > > > > * Duplicated and slow builds: > > > https://forums.docker.com/t/in-docker-auto-build-images-building-duplicates-please-fix-this-images-building-very-long-it-was-taking-more-than-1-hour/87705/2 > > * Duplicated builds (I raised this) > > https://forums.docker.com/t/duplicate-builds-again/93127 > > > > I raised and commented and followed up about it in a number of issues > > (follow the links) a number of times, None of the issues raised for ~ 1 > > year (with follow-ups) got any reasonable response from DockerHub (I was > > following it very closely). After those experiences I do not trust > > autobuilds at all. > > Autobuilds caused us major pain and long delays in providing reference > > images to our users. > > > > This was a major problem for us therefore eventually we switched to > Github > > Registry for CI and we only use DockerHub to do manual pushing of the > > images (we experienced pretty much no problems since). > > > > We only use DockerHub to push "release images" and it's part of the > release > > process of ours - we treat the images the same way as "source releases" - > > we prepare it on the "hardware" owned by the release manager > > (semi-automatically using the same scripts as are used on CI) and push > them > > from there (I actually plan to sign those images very soon too with > private > > keys). > > > > > > J > > > > > > On Fri, Jan 21, 2022 at 5:57 PM Chris Lambertus <c...@apache.org> wrote: > > > >> Hi folks, > >> > >> We have reached our 200 seat maximum for dockerhub users in the Apache > >> organization, and Docker Hub is unable to provide any activity records > for > >> those users, so we do not know who to cull. > >> > >> Do we have any internal contacts amongst the list at DockerHub who might > >> be able to assist? > >> > >> If not, we'll need to start removing users at more-or-less random as > >> needed. > >> > >> Infra will be moving to a more conservative process for adding dockerhub > >> users in the future. We will request that: > >> > >> - autobuilds be used where possible instead of manual creation > >> - user access to dockerhub/teams be limited to a max of 2 per project > >> - all use of dockerhub for pushing artifacts from build servers be > limited > >> to the Infra-provided accounts for this purpose (e.g don't use personal > >> accounts to push artifacts) > >> > >> If your project falls into any of the above categories, please let us > know > >> ASAP so we can trim excess users. > >> > >> Thanks, > >> Chris > >> ASF Infra > >> > >> > >> > >> > >
