Re: Proposal for Iceberg Spec Extensions for Data Access Decision Exchange

Hi Jack Thanks for sharing ! Let me take a look at the doc. I'm also interested in participating in REST Spec (I started to work on a simple REST catalog impl to evaluate some gaps). Regards JB On Fri, Mar 1, 2024 at 10:34 PM Jack Ye wrote: > > Hi everyone, > > Based on the discussion that we

Re: Gravitino an Iceberg REST catalog service

Hi Jack, I agree and it's my fault :) You are right, we need a dev ml thread on this topic (I sent a message a couple of weeks ago to start the discussion about Iceberg 2.0, the idea was to start a thread on each topic). Definitely it has to be discussed on the mailing list to include anyone in t

Re: [VOTE] Release Apache Iceberg 1.5.0 RC4

+1 (binding) Verified sigs/sums/licenses/build/test (Java 17) One thing I noticed while testing views is that there is a discrepancy between the spark catalog behavior of SHOW TABLES and what I see using an Iceberg catalog (jdbc or REST). The "SHOW TABLES" command in spark catalog shows both tab

Re: Materialized view integration with REST spec

The calendar on the site is currently broken https://iceberg.apache.org/community/#iceberg-community-events. Might help to fix it or share the meeting link here. On Fri, Mar 1, 2024 at 3:43 PM Jack Ye wrote: > Sounds good, let's discuss this in person! > > I am a bit worried that we have quite a

Re: [VOTE] Release Apache Iceberg 1.5.0 RC4

+1 (binding) - Verified signature - Verified checksum - RAT check - Compiled - Manually ran basic queries on Spark 3.5 On Fri, Mar 1, 2024 at 6:13 AM Fokko Driesprong wrote: > +1 (binding) > > - Checked checksum and signature > - Ran a modified version of dbt-spark to take advantage of the view

Re: Gravitino an Iceberg REST catalog service

Could we have a separate devlist thread dedicated for this discussion? It is a bit awkward to continue this critical Iceberg 2.0 catalog consolidation topic under this Gravitino thread, although I understand it is related. At least I have overlooked all these discussions until now, and I feel there

Re: Materialized view integration with REST spec

Sounds good, let's discuss this in person! I am a bit worried that we have quite a few critical topics going on right now on devlist, and this will take up a lot of time to discuss. If it ends up going for too long, l propose let us have a dedicated meeting, and I am more than happy to organize it

Re: Gravitino an Iceberg REST catalog service

To clarify, what I meant was that Iceberg has, so far, avoided providing runtime services that are ready to be deployed and used. I still think that is a good choice, and I agree with the reasons that Renjie listed. I disagree that this is inconsistent. We don't supply any of the other services th

Proposal for Iceberg Spec Extensions for Data Access Decision Exchange

Hi everyone, Based on the discussion that we had in the previous thread Support permission concepts in REST spec , I have worked with some other engineers in Amazon to come up with this design: https://docs.goog

RE: Re: Deprecate DynamodbCatalog

+1 for moving It to different repo On 2024/02/27 19:52:12 Jack Ye wrote: > Looks like Ajantha already reverted it now in > https://github.com/apache/iceberg/pull/9815 so let's just keep it as is for > 1.5. > > As I listed in the PR just now, I think there are 3 options: > 1. Salesforce can move i

Re: Materialized view integration with REST spec

Hey everyone, I think this thread has hit a point of diminishing returns and that we still don't have a common understanding of what the options under consideration actually are. Since we were already planning on discussing this at the next community sync, I suggest we pick this up there and use

Re: Inconsistency between REST spec and table/view spec

I'd like to make a slightly different point regarding metadata files. Currently the table spec does require that metadata be stored in a "file", however there is no way to discover that file outside of a Catalog. In other words, a client that is operating purely at the file level has no way of det

Re: Materialized view integration with REST spec

I am finding it hard to interpret the options concretely. I would also suggest breaking the expectation/outcome to milestones. Maybe it becomes easier if we agree to distinguish between an approach that is feasible in the near term and another in the long term, especially if the latter requires sig

Re: Inconsistency between REST spec and table/view spec

> > We may also choose to relax the requirement for metadata files in the > future — I see support for the idea and have considered proposing it also. > But for now, it’s a requirement, even if you don’t have to send the > location to the client (though note that the client has a hard dependency >

Re: Materialized view integration with REST spec

> All of these approaches are aligned in one, specific way: the storage table is an iceberg table. I do not think that is true. I think people are aligned that we would like to re-use the Iceberg table metadata defined in the Iceberg table spec to express the data in MV, but I don't think it goes

Re: Materialized view integration with REST spec

I feel I've been most vocal about pushing back against options 2+ (or Ryan's categories of combined table/view, or new metadata type), so I'll try to expand on my reasoning. I understand the appeal of creating a design where we encapsulate the view/storage from both a structural and performance st

Re: [VOTE] Release Apache Iceberg 1.5.0 RC4

+1 (binding) - Checked checksum and signature - Ran a modified version of dbt-spark to take advantage of the views, and it worked like a charm! 🥳 Cheers, Fokko Op vr 1 mrt 2024 om 06:43 schreef Ajantha Bhat : > Gentle reminder. > > On Wed, Feb 28, 2024 at 8:34 PM Eduard Tudenhoefner > wrote: >

Re: Gravitino an Iceberg REST catalog service

Hey Brian Thanks for the summary ! Good one ! I would just add the "REST ref impl" discussion. Regarding the anti patterns, I agree with the lists, imho, some are more "opinionated implementation", so definitely not in the API scope. +1 Thanks again ! Regards JB On Fri, Mar 1, 2024 at 1:15 P

Re: Gravitino an Iceberg REST catalog service

My attempt to consolidate a list of goals, anti patterns , and impl details mentioned since this discussion was brought up at the last Iceberg sync. Tried to roughly capture who mentioned these things so we can follow up if needed. Hopefully this can serve as a basis for the design discussion. Goa

Re: Gravitino an Iceberg REST catalog service

Hi Renjie, maybe I wasn't clear, sorry about that: the target is really both ref impl (where we can test different Iceberg parts like we do with the InMemoryCatalog, JdbcCatalog, etc) and ready to go service for users (simple but to start with). But we can't prevent the community from working on

Re: Gravitino an Iceberg REST catalog service

Hi: I think one thing missing in the discussion is that, if the iceberg community wants to maintain a rest catalog service, what's the target use case? Different target use cases may lead to different directions. If it's mainly designed for first time users to play or experience with rest catalog