+1 for TCK and a reference implementation. Once the reference implementation is ready, I suggest we also update the below quickstart to use it. https://iceberg.apache.org/spark-quickstart/?h=tabular#spark-and-iceberg-quickstart In my opinion, we should avoid directing open-source users to closed-source or vendor-specific implementations, as we currently do.
- Ajantha On Thu, Jun 20, 2024 at 2:23 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Dmitri > > Yes, it makes sense to me. > > As discussed today during the community meeting, where the "simple ref > impl"/TCK are located is not a big deal. But, I think it would be > great to actually have sref impl/TCK (in a separate repo or not). > > Regards > JB > > On Wed, Jun 19, 2024 at 7:08 PM Dmitri Bourlatchkov > <dmitri.bourlatch...@dremio.com.invalid> wrote: > > > > Re: TCK and ref impl in the "spec" repo. I believe the reference > implementation there does not have to be complete in the sense of actually > managing Iceberg metadata. It only needs to be complete enough to allow TCK > tests to pass. TCKs will be the normative checks for "real" catalog > implementations, not the reference "server" impl. We will have to be > careful about making TCKs match and validate the intention behind the > OpenAPI spec. Does that make sense? > > > > For that matter I think runtime dependencies on the reference server > implementation should not be an issue as they will not trickle down to any > other implementation. > > > > Cheers, > > Dmitri. > > > > On Wed, Jun 19, 2024 at 11:43 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> > >> Hi folks, > >> > >> I wanted to provide an update on this proposal and also extend > discussion. > >> > >> I started to rework on the REST Catalog proposal by splitting in > >> v1-improvements and v2-proposal (focusing on extensions), step by > >> step. > >> > >> I would like to emphasize a few proposals from the doc: > >> 1. In order to guarantee isolation of concerns and easier > >> contribution, I think it would be great to create iceberg-rest repo > >> and move openapi.yml in this repo. > >> 2. In the iceberg-rest repo, in order to verify the spec, we should > >> have a simple reference implementation. The purpose is NOT to have a > >> production grade implementation, but a simple ref implementation to > >> verify that all is working/integrate fine with engines and different > >> iceberg components. Weeks ago I started an implementation with Quarkus > >> but I stopped as we had a discussion saying it doesn't make sense to > >> have an "runtime" as part of Iceberg. I agree with that but I still > >> think the proposal of a simple ref implementation as part of > >> iceberg-rest makes sense. I think I was wrong to start with Quarkus > >> (too large), I will resume work this ref impl using pure Java (SPI > >> like, similar to what I did in Apache Karaf Minho > >> https://github.com/apache/karaf-minho). > >> 3. The iceberg-rest repo would also contain a TCK to verify REST > >> Catalog implementation compliance with the spec > >> > >> I propose to rework on the REST v2 proposal including these topics. > Thoughts ? > >> > >> Thanks > >> Regards > >> JB > >> > >> On Mon, Apr 1, 2024 at 4:36 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> > > >> > Hi folks, > >> > > >> > With several people from the community, we started to craft a proposal > >> > to design new REST Catalog Spec. > >> > > >> > I used the "proposal process" to track this: > >> > > >> > - The proposal "issue" is here: > https://github.com/apache/iceberg/issues/10075 > >> > - The proposal document is here: > >> > > https://docs.google.com/document/d/1JUtFpdEoa6IAKt1EzJi_re0PUbh56XnfUtRe5WAfl0s/edit?usp=sharing > >> > > >> > I propose the following next steps about this proposal: > >> > 1. Please review and comment in the doc > >> > 2. After ~ week, I would like to schedule a meeting to discuss the > >> > proposal and refine the document > >> > 3. Once the proposal is good enough (I would say a couple of weeks), I > >> > will call for a formal vote (according to the proposal process). > >> > > >> > Thanks ! > >> > Regards > >> > JB >
