Hey folks, I want to add a few references here to close the loop.
The docker image is available on docker hub under `apache/iceberg-rest-fixture`, https://hub.docker.com/r/apache/iceberg-rest-fixture And a few iceberg subprojects are already using the image * iceberg-rust <https://github.com/apache/iceberg-rust/blob/2e0b64646fcfbd909788236a251a3a374a193542/crates/integration_tests/testdata/docker-compose.yaml#L23> * iceberg-go <https://github.com/apache/iceberg-go/blob/88bbae37af6b24998fc334831f4d63cd444aac1e/dev/docker-compose.yml#L42> * iceberg-python <https://github.com/apache/iceberg-python/blob/a97d13c17cd03f86252b9df2c65532ec45fb05da/dev/docker-compose-integration.yml#L44> Thanks everyone for making this happen! Best, Kevin Liu On Mon, Dec 9, 2024 at 12:15 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Piotr > > That's a good point. > > As DockerHub is managed by The ASF, I think it's worth it to have > docker images hosted there at least. That said, I don't see a problem > with publishing on GH Packages. > > Regards > JB > > On Thu, Dec 5, 2024 at 2:57 PM Piotr Findeisen > <piotr.findei...@gmail.com> wrote: > > > > Hi, > > > > Sorry for coming late here. > > Did we consider GitHub packages as a home of the Apache docker images? > > We already use GitHub for development and GitHub packages are better > integrated with GitHub. > > In my personal opinion github packages are also less likely to be rate > limited. > > > > Best > > Piotr > > > > > > > > > > On Fri, 22 Nov 2024 at 19:03, Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> > >> Hi > >> > >> That's correct: in Sung's PR, I can see the secret.DOCKERHUB_USER and > >> secret.DOCKERHUB_TOKEN. > >> So, we should be able to publish docker images via this GitHub action ;) > >> > >> Regards > >> JB > >> > >> On Fri, Nov 22, 2024 at 6:16 PM Fokko Driesprong <fo...@apache.org> > wrote: > >> > > >> > I think Sung beat you to it: > https://github.com/apache/iceberg/pull/11632 > >> > > >> > As mentioned earlier it would be awesome if we could have a nightly > build so we can test all the different languages against the nightly. In > this case, when there are changes or new features, we can test/implement > them right away. > >> > > >> > Kind regards, > >> > Fokko > >> > > >> > Op vr 22 nov 2024 om 18:11 schreef Kevin Liu <kevin.jq....@gmail.com > >: > >> >> > >> >> Thanks for setting this up, JB! It looks like PR #11283 is close to > being merged. > >> >> > >> >> What is the deployment strategy for the Docker image? Ideally, this > process could be fully automated using GitHub and GitHub Actions. > >> >> > >> >> I’d love to hear everyone’s thoughts on this! > >> >> > >> >> Best regards, > >> >> Kevin Liu > >> >> > >> >> > >> >> On Fri, Nov 22, 2024 at 6:06 AM Jean-Baptiste Onofré < > j...@nanthrax.net> wrote: > >> >>> > >> >>> Hi folks, > >> >>> > >> >>> I created the iceberg repo on DockerHub (in the Apache org): > >> >>> > >> >>> https://hub.docker.com/r/apache/iceberg > >> >>> > >> >>> I created an "Iceberg team" on DockerHub. > >> >>> > >> >>> I created DOCKERHUB_USER and DOCKERHUB_TOKEN credentials for the > >> >>> Iceberg repo. That will allow us to directly push on DockerHub repo > >> >>> from GitHub Action. > >> >>> I also added Fokko to the repo. > >> >>> > >> >>> If you are a committer and you want to get permission on the Iceberg > >> >>> DockerHub repo, please let me know, I will add your DockerHub > account > >> >>> to the "iceberg team". > >> >>> > >> >>> Thanks ! > >> >>> > >> >>> Regards > >> >>> JB > >> >>> > >> >>> On Fri, Nov 15, 2024 at 7:39 PM Kevin Liu <kevin.jq....@gmail.com> > wrote: > >> >>> > > >> >>> > +1 to Iceberg REST TCK docker image. Thanks, JB for driving this > and Ajantha for setting up the docker image. > >> >>> > We already found a bug in PyIceberg [1] from integrating with the > TCK docker image. It would be great to have a nightly build, perhaps we can > set up a Github Action to automate the docker image publishing. > >> >>> > > >> >>> > Best, > >> >>> > Kevin Liu > >> >>> > > >> >>> > > >> >>> > [1] https://github.com/apache/iceberg-python/pull/1321 > >> >>> > > >> >>> > On Fri, Nov 15, 2024 at 1:36 AM Fokko Driesprong < > fo...@apache.org> wrote: > >> >>> >> > >> >>> >> +1 — excited to see this happen! > >> >>> >> > >> >>> >> For the TCK, I think we can release this with the Java together, > and have a nightly build (tag the container with nightly Dockerhub). This > way we can already test out (and start implementing) the new features in > the related projects. Thoughts on that? > >> >>> >> > >> >>> >>> Regarding the Kafka Connect Docker image, I believe that if we > maintain it, we could also manage other integration images, such as those > for Spark and Trino with Iceberg. We should have a separate discussion on > which integration images Iceberg should officially support. > >> >>> >> > >> >>> >> > >> >>> >> Let's split out that discussion. My take on that is that we want > to defer that to the query engines. In an ideal situation, the Iceberg > integration should be part of the project itself (e.g. with Hive 4 where it > is maintained by Hive itself). For Spark itself, it only requires a runtime > to be added through the packages argument, and would love to see if we can > avoid maintaining images for that. > >> >>> >> > >> >>> >> Kind regards, > >> >>> >> Fokko > >> >>> >> > >> >>> >> > >> >>> >> Op do 14 nov 2024 om 18:16 schreef Christian Thiel < > christ...@hansetag.com.invalid>: > >> >>> >>> > >> >>> >>> +1 for this as well – for us especially the REST TCK image > would be nice. > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> From: Bryan Keller <brya...@gmail.com> > >> >>> >>> Date: Thursday, 14. November 2024 at 17:13 > >> >>> >>> To: dev@iceberg.apache.org <dev@iceberg.apache.org> > >> >>> >>> Subject: Re: [PROPOSAL] Create Iceberg DockerHub repository > >> >>> >>> > >> >>> >>> +1 this would be great! Thanks JB. > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> -Bryan > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> On Nov 14, 2024, at 8:30 AM, Ajantha Bhat < > ajanthab...@gmail.com> wrote: > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> +1 for setting up the DockerHub repo, > >> >>> >>> > >> >>> >>> We discussed about this already in > https://www.mail-archive.com/dev@iceberg.apache.org/msg07888.html > >> >>> >>> > >> >>> >>> Now that the Docker image PR is ready for the REST catalog > adapter, we can proceed with setting up the DockerHub repository. > >> >>> >>> > >> >>> >>> Regarding the Kafka Connect Docker image, I believe that if we > maintain it, we could also manage other integration images, such as those > for Spark and Trino with Iceberg. We should have a separate discussion on > which integration images Iceberg should officially support. > >> >>> >>> > >> >>> >>> For now, maintaining the REST catalog adapter image has already > been approved in earlier discussions, so let’s start with that. > >> >>> >>> > >> >>> >>> - Ajantha > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> On Thu, Nov 14, 2024 at 9:45 PM Sung Yun <sun...@apache.org> > wrote: > >> >>> >>> > >> >>> >>> Hi JB, > >> >>> >>> > >> >>> >>> That sounds great!! > >> >>> >>> > >> >>> >>> The REST TCK /adapter docker image will be super useful for the > Iceberg subprojects as it will ensure that they have access to a > light-weight REST Catalog Server image with the latest features to run > integration tests against. > >> >>> >>> > >> >>> >>> Sung > >> >>> >>> > >> >>> >>> On 2024/11/14 15:41:04 Jean-Baptiste Onofré wrote: > >> >>> >>> > Hi folks, > >> >>> >>> > > >> >>> >>> > While reviewing https://github.com/apache/iceberg/pull/11283, > we > >> >>> >>> > discussed having a DockerHub repository for Iceberg. > >> >>> >>> > > >> >>> >>> > I can create this repository, similar to other Apache > projects (like > >> >>> >>> > for example https://hub.docker.com/r/apache/activemq-classic, > >> >>> >>> > https://hub.docker.com/r/apache/airflow, etc). > >> >>> >>> > I can create an iceberg group (on DockerHub), and committers > can ask > >> >>> >>> > to join (in order to be able to push docker images). > >> >>> >>> > > >> >>> >>> > For now, the purpose of this DockerHub repo is to host: > >> >>> >>> > - Iceberg REST TCK docker images > >> >>> >>> > - Iceberg Kafka Connect docker images > >> >>> >>> > > >> >>> >>> > Thoughts ? > >> >>> >>> > > >> >>> >>> > Regards > >> >>> >>> > JB > >> >>> >>> > > >> >>> >>> > >> >>> >>> >
