Hi My understanding was last time it was still unresolved, and the action item was on Jack and/or/ Jan to make a shorter document. I think the debate now has boiled down to Ryan's three options:
1. separate table/view 2. combination of table/view tied together via commit 3. new metadata type with probably the first and third being the main contenders. My understanding was we wanted a table of pros/cons between (1) and (3), presumably giving folks a chance to address the cons, before the next meeting. Jack (main proponent of option (3) just went on paternity leave, so not sure if there was someone from Amazon with some context of Jack's thought to continue that train of thought though? Otherwise maybe Jan can give it a shot? Else I will be out and can't make the next iceberg sync, but can prepare one for the one after that, if needed. Re: 'new' proposal', not sure if we are ready for a formal one, given the deadlock between the two options, but Im open to that as well to make a proposal based on one of the options above. What do folks think? Thanks, Szehon On Fri, Mar 22, 2024 at 3:15 AM Renjie Liu <liurenjie2...@gmail.com> wrote: > +1 > > On Fri, Mar 22, 2024 at 16:42 Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> Hi Renjie, >> >> We discussed the MV proposal, without yet reaching any conclusion. >> >> I propose: >> - to use the "new" proposal process in place (creating an GH issue with >> proposal flag, with link to the document) >> - use the document and/or GH issue to add comments >> - finalize the document heading to a vote (to get consensus) >> >> Thoughts ? >> >> NB: I will follow up with "stale PR/proposal" PR to be sure we are moving >> forward ;) >> >> Regards >> JB >> >> On Fri, Mar 22, 2024 at 4:29 AM Renjie Liu <liurenjie2...@gmail.com> >> wrote: >> >>> Hi: >>> >>> Sorry I didn't make it to join the last community sync. Did we reach any >>> conclusion about mv spec? >>> >>> On Tue, Mar 5, 2024 at 11:28 PM himadri pal <meh...@gmail.com> wrote: >>> >>>> For me the calendar link did not work in mobile, but I was able to add >>>> the dev Google calendar from >>>> https://iceberg.apache.org/community/#iceberg-community-events by >>>> accessing it from laptop. >>>> >>>> Regards, >>>> Himadri Pal >>>> >>>> >>>> On Mon, Mar 4, 2024 at 4:43 PM Walaa Eldin Moustafa < >>>> wa.moust...@gmail.com> wrote: >>>> >>>>> Thanks Jack! I think the images are stripped from the message, but >>>>> they are there on the doc >>>>> <https://docs.google.com/spreadsheets/d/1a0tlyh8f2ft2SepE7H3bgoY2A0q5IILgzAsJMnwjTBs/edit#gid=0> >>>>> if >>>>> someone wants to check them out (I have left some comments while there). >>>>> >>>>> Also I no longer see the community sync calendar >>>>> https://iceberg.apache.org/community/#slack, so it is unclear when >>>>> the meeting is (and we do not have the link). >>>>> >>>>> Thanks, >>>>> Walaa. >>>>> >>>>> >>>>> On Mon, Mar 4, 2024 at 9:58 AM Jack Ye <yezhao...@gmail.com> wrote: >>>>> >>>>>> Thanks Jan! +1 for everyone to take a look before the discussion, and >>>>>> see if there are any missing options or major arguments. >>>>>> >>>>>> I have also added the images regarding all the options, it might be >>>>>> easier to parse than the big sheet. I will also put it here for people >>>>>> that >>>>>> do not have time to read through it: >>>>>> >>>>>> >>>>>> *Option 1: Add storage table identifier in view metadata content* >>>>>> >>>>>> [image: MV option 1.png] >>>>>> *Option 2: Add storage table metadata file pointer in view object* >>>>>> >>>>>> [image: MV option 2.png] >>>>>> *Option 3: Add storage table metadata file pointer in view metadata >>>>>> content* >>>>>> >>>>>> [image: MV option 3.png] >>>>>> >>>>>> *Option 4: Embed table metadata in view metadata content* >>>>>> >>>>>> [image: MV option 4.png] >>>>>> *Option 5: New MV spec, MV object has table and view metadata file >>>>>> pointers* >>>>>> >>>>>> [image: MV option 5.png] >>>>>> *Option 6: New MV spec, MV metadata content embeds table and view >>>>>> metadata* >>>>>> >>>>>> [image: MV option 6.png] >>>>>> *Option 7: New MV spec, completely new MV metadata content* >>>>>> >>>>>> [image: MV option 7.png] >>>>>> >>>>>> -Jack >>>>>> >>>>>> >>>>>> On Sun, Mar 3, 2024 at 11:45 PM Jan Kaul <jank...@mailbox.org.invalid> >>>>>> wrote: >>>>>> >>>>>>> I think it's great to have a face to face discussion about this. >>>>>>> Additionally, I would propose to use Jacks' document >>>>>>> <https://docs.google.com/spreadsheets/d/1a0tlyh8f2ft2SepE7H3bgoY2A0q5IILgzAsJMnwjTBs/edit#gid=0> >>>>>>> as a common ground for the discussion and that everyone has a quick look >>>>>>> before the next community sync. If you think the document is still >>>>>>> missing >>>>>>> some arguments, please make suggestions to add them. This way we have to >>>>>>> spend less time to get everyone up to speed and have a more common >>>>>>> terminology. >>>>>>> >>>>>>> Looking forward to the discussion, best wishes >>>>>>> >>>>>>> Jan >>>>>>> On 02.03.24 02:06, Walaa Eldin Moustafa wrote: >>>>>>> >>>>>>> The calendar on the site is currently broken >>>>>>> https://iceberg.apache.org/community/#iceberg-community-events. >>>>>>> Might help to fix it or share the meeting link here. >>>>>>> >>>>>>> On Fri, Mar 1, 2024 at 3:43 PM Jack Ye <yezhao...@gmail.com> wrote: >>>>>>> >>>>>>>> Sounds good, let's discuss this in person! >>>>>>>> >>>>>>>> I am a bit worried that we have quite a few critical topics going >>>>>>>> on right now on devlist, and this will take up a lot of time to >>>>>>>> discuss. If >>>>>>>> it ends up going for too long, l propose let us have a dedicated >>>>>>>> meeting, >>>>>>>> and I am more than happy to organize it. >>>>>>>> >>>>>>>> Best, >>>>>>>> Jack Ye >>>>>>>> >>>>>>>> On Fri, Mar 1, 2024 at 12:48 PM Ryan Blue <b...@tabular.io> wrote: >>>>>>>> >>>>>>>>> Hey everyone, >>>>>>>>> >>>>>>>>> I think this thread has hit a point of diminishing returns and >>>>>>>>> that we still don't have a common understanding of what the options >>>>>>>>> under >>>>>>>>> consideration actually are. >>>>>>>>> >>>>>>>>> Since we were already planning on discussing this at the next >>>>>>>>> community sync, I suggest we pick this up there and use that time to >>>>>>>>> align >>>>>>>>> on what exactly we're considering. We can then start a new thread to >>>>>>>>> lay >>>>>>>>> out the designs under consideration in more detail and then have a >>>>>>>>> discussion about trade-offs. >>>>>>>>> >>>>>>>>> Does that sound reasonable? >>>>>>>>> >>>>>>>>> Ryan >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Mar 1, 2024 at 11:09 AM Walaa Eldin Moustafa < >>>>>>>>> wa.moust...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> I am finding it hard to interpret the options concretely. I would >>>>>>>>>> also suggest breaking the expectation/outcome to milestones. Maybe it >>>>>>>>>> becomes easier if we agree to distinguish between an approach that is >>>>>>>>>> feasible in the near term and another in the long term, especially >>>>>>>>>> if the >>>>>>>>>> latter requires significant engine-side changes. >>>>>>>>>> >>>>>>>>>> Further, maybe it helps if we start with an option that fully >>>>>>>>>> reuses the existing spec, and see how we view it in comparison with >>>>>>>>>> the >>>>>>>>>> options discussed previously. I am sharing one below. It reuses the >>>>>>>>>> current >>>>>>>>>> spec of Iceberg views and tables by leveraging table properties to >>>>>>>>>> capture >>>>>>>>>> materialized view metadata. What is common (and not common) between >>>>>>>>>> this >>>>>>>>>> and the desired representations? >>>>>>>>>> >>>>>>>>>> The new properties are: >>>>>>>>>> Properties on a View: >>>>>>>>>> >>>>>>>>>> 1. >>>>>>>>>> >>>>>>>>>> *iceberg.materialized.view*: >>>>>>>>>> - *Type*: View property >>>>>>>>>> - *Purpose*: This property is used to mark whether a view >>>>>>>>>> is a materialized view. If set to true, the view is >>>>>>>>>> treated as a materialized view. This helps in differentiating >>>>>>>>>> between >>>>>>>>>> virtual and materialized views within the catalog and dictates >>>>>>>>>> specific >>>>>>>>>> handling and validation logic for materialized views. >>>>>>>>>> 2. >>>>>>>>>> >>>>>>>>>> *iceberg.materialized.view.storage.location*: >>>>>>>>>> - *Type*: View property >>>>>>>>>> - *Purpose*: Specifies the location of the storage table >>>>>>>>>> associated with the materialized view. This property is used >>>>>>>>>> for linking a >>>>>>>>>> materialized view with its corresponding storage table, >>>>>>>>>> enabling data >>>>>>>>>> management and query execution based on the stored data >>>>>>>>>> freshness. >>>>>>>>>> >>>>>>>>>> Properties on a Table: >>>>>>>>>> >>>>>>>>>> 1. *base.snapshot.[UUID]*: >>>>>>>>>> - *Type*: Table property >>>>>>>>>> - *Purpose*: These properties store the snapshot IDs of >>>>>>>>>> the base tables at the time the materialized view's data was >>>>>>>>>> last updated. >>>>>>>>>> Each property is prefixed with base.snapshot. followed by >>>>>>>>>> the UUID of the base table. They are used to track whether the >>>>>>>>>> materialized >>>>>>>>>> view's data is up to date with the base tables by comparing >>>>>>>>>> these snapshot >>>>>>>>>> IDs with the current snapshot IDs of the base tables. If all >>>>>>>>>> the base >>>>>>>>>> tables' current snapshot IDs match the ones stored in these >>>>>>>>>> properties, the >>>>>>>>>> materialized view's data is considered fresh. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Walaa. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Mar 1, 2024 at 9:15 AM Jack Ye <yezhao...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> > All of these approaches are aligned in one, specific way: the >>>>>>>>>>> storage table is an iceberg table. >>>>>>>>>>> >>>>>>>>>>> I do not think that is true. I think people are aligned that we >>>>>>>>>>> would like to re-use the Iceberg table metadata defined in the >>>>>>>>>>> Iceberg >>>>>>>>>>> table spec to express the data in MV, but I don't think it goes >>>>>>>>>>> that far to >>>>>>>>>>> say it must be an Iceberg table. Once you have that mindset, then >>>>>>>>>>> of course >>>>>>>>>>> option 1 (separate table and view) is the only option. >>>>>>>>>>> >>>>>>>>>>> > I don't think that is necessary and it significantly increases >>>>>>>>>>> the complexity. >>>>>>>>>>> >>>>>>>>>>> And can you quantify what you mean by "significantly increases >>>>>>>>>>> the complexity"? Seems like a lot of concerns are coming from the >>>>>>>>>>> tradeoff >>>>>>>>>>> with complexity. We probably all agree that using option 7 (a >>>>>>>>>>> completely >>>>>>>>>>> new metadata type) is a lot of work from scratch, that is why it is >>>>>>>>>>> not >>>>>>>>>>> favored. However, my understanding is that as long as we re-use the >>>>>>>>>>> view >>>>>>>>>>> and table metadata, then the majority of the existing logic can be >>>>>>>>>>> reused. >>>>>>>>>>> I think what we have gone through in Slack to draft the rough Java >>>>>>>>>>> API >>>>>>>>>>> shape helps here, because people can estimate the amount of effort >>>>>>>>>>> required >>>>>>>>>>> to implement it. And I don't think they are **significantly** more >>>>>>>>>>> complex >>>>>>>>>>> to implement. Could you elaborate more about the complexity that you >>>>>>>>>>> imagine? >>>>>>>>>>> >>>>>>>>>>> -Jack >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Mar 1, 2024 at 8:57 AM Daniel Weeks < >>>>>>>>>>> daniel.c.we...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> I feel I've been most vocal about pushing back against options >>>>>>>>>>>> 2+ (or Ryan's categories of combined table/view, or new metadata >>>>>>>>>>>> type), so >>>>>>>>>>>> I'll try to expand on my reasoning. >>>>>>>>>>>> >>>>>>>>>>>> I understand the appeal of creating a design where we >>>>>>>>>>>> encapsulate the view/storage from both a structural and performance >>>>>>>>>>>> standpoint, but I don't think that is necessary and it >>>>>>>>>>>> significantly increases the complexity. >>>>>>>>>>>> >>>>>>>>>>>> All of these approaches are aligned in one, specific way: the >>>>>>>>>>>> storage table is an iceberg table. >>>>>>>>>>>> >>>>>>>>>>>> Because of this, all the behaviors and requirements still apply >>>>>>>>>>>> to these tables. They need to be maintained (snapshot cleanup, >>>>>>>>>>>> orphan >>>>>>>>>>>> files), in cases need to be optimized (compaction, manifest >>>>>>>>>>>> rewrites), they >>>>>>>>>>>> need to be able to be inspected (this will be even more important >>>>>>>>>>>> with MV >>>>>>>>>>>> since staleness can produce different results and questions will >>>>>>>>>>>> arise >>>>>>>>>>>> about what state the storage table was in). There may be cases >>>>>>>>>>>> where the >>>>>>>>>>>> tables need to be managed directly. >>>>>>>>>>>> >>>>>>>>>>>> Anywhere we deviate from the existing constructs/commit/access >>>>>>>>>>>> for tables, we will ultimately have to then unwrap to re-expose the >>>>>>>>>>>> underlying Iceberg behavior. This creates unnecessary complexity >>>>>>>>>>>> in the >>>>>>>>>>>> library/API layer, which are not the primary interface users will >>>>>>>>>>>> have with >>>>>>>>>>>> materialized views where an engine is almost entirely necessary to >>>>>>>>>>>> interact >>>>>>>>>>>> with the dataset. >>>>>>>>>>>> >>>>>>>>>>>> As to the performance concerns around option 1, I think we're >>>>>>>>>>>> overstating the downsides. It really comes down to how many >>>>>>>>>>>> metadata loads >>>>>>>>>>>> are necessary and evaluating freshness would likely be the real >>>>>>>>>>>> bottleneck >>>>>>>>>>>> as it involves potentially loading many tables. All of the >>>>>>>>>>>> options are on >>>>>>>>>>>> the same order of performance for the metadata and table loads. >>>>>>>>>>>> >>>>>>>>>>>> As to the visibility of tables and whether they're registered >>>>>>>>>>>> in the catalog, I think registering in the catalog is the right >>>>>>>>>>>> approach so >>>>>>>>>>>> that the tables are still addressable for maintenance/etc. The >>>>>>>>>>>> visibility >>>>>>>>>>>> of the storage table is a catalog implementation decision and >>>>>>>>>>>> shouldn't be >>>>>>>>>>>> a requirement of the MV spec (I can see cases for both and it isn't >>>>>>>>>>>> necessary to dictate a behavior). >>>>>>>>>>>> >>>>>>>>>>>> I'm still strongly in favor of Option 1 (separate table and >>>>>>>>>>>> view) for these reasons. >>>>>>>>>>>> >>>>>>>>>>>> -Dan >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Feb 29, 2024 at 11:07 PM Jack Ye <yezhao...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> > Jack, it sounds like you’re the proponent of a combined >>>>>>>>>>>>> table and view (rather than a new metadata spec for a >>>>>>>>>>>>> materialized view). >>>>>>>>>>>>> What is the main motivation? It seems like you’re convinced of >>>>>>>>>>>>> that >>>>>>>>>>>>> approach, but I don’t understand the advantage it brings. >>>>>>>>>>>>> >>>>>>>>>>>>> Sorry I have to make a Google Sheet to capture all the options >>>>>>>>>>>>> we have discussed so far, I wanted to use the existing Google >>>>>>>>>>>>> Doc, but it >>>>>>>>>>>>> has really bad table/sheet support... >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> https://docs.google.com/spreadsheets/d/1a0tlyh8f2ft2SepE7H3bgoY2A0q5IILgzAsJMnwjTBs/edit#gid=0 >>>>>>>>>>>>> >>>>>>>>>>>>> I have listed all the options, with how they are implemented >>>>>>>>>>>>> and some important considerations we have discussed so far. Note >>>>>>>>>>>>> that: >>>>>>>>>>>>> 1. This sheet currently excludes the lineage information, >>>>>>>>>>>>> which we can discuss more later after the current topic is >>>>>>>>>>>>> resolved. >>>>>>>>>>>>> 2. I removed the considerations for REST integration since >>>>>>>>>>>>> from the other thread we have clarified that they should be >>>>>>>>>>>>> considered >>>>>>>>>>>>> completely separately. >>>>>>>>>>>>> >>>>>>>>>>>>> *Why I come as a proponent of having a new MV object with >>>>>>>>>>>>> table and view metadata file pointer* >>>>>>>>>>>>> >>>>>>>>>>>>> In my sheet, there are 3 options that do not have major >>>>>>>>>>>>> problems: >>>>>>>>>>>>> Option 2: Add storage table metadata file pointer in view >>>>>>>>>>>>> object >>>>>>>>>>>>> Option 5: New MV object with table and view metadata file >>>>>>>>>>>>> pointer >>>>>>>>>>>>> Option 6: New MV spec with table and view metadata >>>>>>>>>>>>> >>>>>>>>>>>>> I originally excluded option 2 because I think it does not >>>>>>>>>>>>> align with the REST spec, but after the other discussion thread >>>>>>>>>>>>> about "Inconsistency >>>>>>>>>>>>> between REST spec and table/view spec", I think my original >>>>>>>>>>>>> concern no >>>>>>>>>>>>> longer holds true so now I put it back. And based on my >>>>>>>>>>>>> personal preference that MV is an independent object that should >>>>>>>>>>>>> be >>>>>>>>>>>>> separated from view and table, plus the fact that option 5 is >>>>>>>>>>>>> probably less >>>>>>>>>>>>> work than option 6 for implementation, that is how I come as a >>>>>>>>>>>>> proponent of >>>>>>>>>>>>> option 5 at this moment. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> *Regarding Ryan's evaluation framework * >>>>>>>>>>>>> >>>>>>>>>>>>> I think we need to reconcile this sheet with Ryan's evaluation >>>>>>>>>>>>> framework. That framework categorization puts option 2, 3, 4, 5, >>>>>>>>>>>>> 6 all >>>>>>>>>>>>> under the same category of "A combination of a view and a >>>>>>>>>>>>> table" and concludes that they don't have any advantage for the >>>>>>>>>>>>> same set of >>>>>>>>>>>>> reasons. But those reasons are not really convincing to me so >>>>>>>>>>>>> let's talk >>>>>>>>>>>>> about them in more detail. >>>>>>>>>>>>> >>>>>>>>>>>>> (1) You said "I don’t see a reason why a combined view and >>>>>>>>>>>>> table is advantageous" as "this would cause unnecessary >>>>>>>>>>>>> dependence between >>>>>>>>>>>>> the view and table in catalogs." What dependency exactly do you >>>>>>>>>>>>> mean here? >>>>>>>>>>>>> And why is that unnecessary, given there has to be some sort of >>>>>>>>>>>>> dependency >>>>>>>>>>>>> anyway unless we go with option 5 or 6? >>>>>>>>>>>>> >>>>>>>>>>>>> (2) You said "I guess there’s an argument that you could load >>>>>>>>>>>>> both table and view metadata locations at the same time. That >>>>>>>>>>>>> hardly seems >>>>>>>>>>>>> worth the trouble". I disagree with that. Catalog interaction >>>>>>>>>>>>> performance >>>>>>>>>>>>> is critical to at least everyone working in EMR and Athena, and >>>>>>>>>>>>> MV itself >>>>>>>>>>>>> as an acceleration approach needs to be as fast as possible. >>>>>>>>>>>>> >>>>>>>>>>>>> I have put 3 key operations in the doc that I think matters >>>>>>>>>>>>> for MV during interactions with engine: >>>>>>>>>>>>> 1. refreshes storage table >>>>>>>>>>>>> 2. get the storage table of the MV >>>>>>>>>>>>> 3. if stale, get the view SQL >>>>>>>>>>>>> >>>>>>>>>>>>> And option 1 clearly falls short with 4 sequential steps >>>>>>>>>>>>> required to load a storage table. You mentioned "recent issues >>>>>>>>>>>>> with adding >>>>>>>>>>>>> views to the JDBC catalog" in this topic, could you explain a bit >>>>>>>>>>>>> more? >>>>>>>>>>>>> >>>>>>>>>>>>> (3) You said "I also think that once we decide on structure, >>>>>>>>>>>>> we can make it possible for REST catalog implementations to do >>>>>>>>>>>>> smart >>>>>>>>>>>>> things, in a way that doesn’t put additional requirements on the >>>>>>>>>>>>> underlying >>>>>>>>>>>>> catalog store." If REST is fully compatible with Iceberg spec >>>>>>>>>>>>> then I have >>>>>>>>>>>>> no problem with this statement. However, as we discussed in the >>>>>>>>>>>>> other >>>>>>>>>>>>> thread, it is not the case. In the current state, I think the >>>>>>>>>>>>> sequence of >>>>>>>>>>>>> action should be to evolve the Iceberg table/view spec (or add a >>>>>>>>>>>>> MV spec) >>>>>>>>>>>>> first, and then think about how REST can incorporate it or do >>>>>>>>>>>>> smart things >>>>>>>>>>>>> that are not Iceberg spec compliant. Do you agree with that? >>>>>>>>>>>>> >>>>>>>>>>>>> (4) You said the table identifier pointer "is a problem we >>>>>>>>>>>>> need to solve generally because a materialized table needs to be >>>>>>>>>>>>> able to >>>>>>>>>>>>> track the upstream state of tables that were used". I don't think >>>>>>>>>>>>> that is a >>>>>>>>>>>>> reason to choose to use a table identifier pointer for a storage >>>>>>>>>>>>> table. The >>>>>>>>>>>>> issue is not about using a table identifier pointer. It is about >>>>>>>>>>>>> exposing >>>>>>>>>>>>> the storage table as a separate entity in the catalog, which is >>>>>>>>>>>>> what people >>>>>>>>>>>>> do not like and is already discussed in length in Jan's question >>>>>>>>>>>>> 3 (also >>>>>>>>>>>>> linked in the sheet). I agree with that statement, because >>>>>>>>>>>>> without a REST >>>>>>>>>>>>> implementation that can magically hide the storage table, this >>>>>>>>>>>>> model adds >>>>>>>>>>>>> additional burden regarding compliance and data governance for >>>>>>>>>>>>> any other >>>>>>>>>>>>> non-REST catalog implementations that are compliant to the >>>>>>>>>>>>> Iceberg spec. >>>>>>>>>>>>> Many mechanisms need to be built in a catalog to hide, protect, >>>>>>>>>>>>> maintain, >>>>>>>>>>>>> recycle the storage table, that can be avoided by using other >>>>>>>>>>>>> approaches. I >>>>>>>>>>>>> think we should reach a consensus about that and discuss further >>>>>>>>>>>>> if you do >>>>>>>>>>>>> not agree. >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> Jack Ye >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Feb 29, 2024 at 10:53 PM Jan Kaul >>>>>>>>>>>>> <jank...@mailbox.org.invalid> <jank...@mailbox.org.invalid> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Ryan, we actually discussed your categories in this >>>>>>>>>>>>>> question >>>>>>>>>>>>>> <https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit?pli=1#heading=h.y70rtfhi9qxi>. >>>>>>>>>>>>>> Where your categories correspond to the following designs: >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Separate table and view => Design 1 >>>>>>>>>>>>>> - Combination of view and table => Design 2 >>>>>>>>>>>>>> - A new metadata type => Design 4 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jan >>>>>>>>>>>>>> On 01.03.24 00:03, Ryan Blue wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Looks like it wasn’t clear what I meant for the 3 categories, >>>>>>>>>>>>>> so I’ll be more specific: >>>>>>>>>>>>>> >>>>>>>>>>>>>> - *Separate table and view*: this option is to have the >>>>>>>>>>>>>> objects that we have today, with extra metadata. Commit >>>>>>>>>>>>>> processes are >>>>>>>>>>>>>> separate: committing to the table doesn’t alter the view and >>>>>>>>>>>>>> committing to >>>>>>>>>>>>>> the view doesn’t change the table. However, changing the view >>>>>>>>>>>>>> can make it >>>>>>>>>>>>>> so the table is no longer useful as a materialization. >>>>>>>>>>>>>> - *A combination of a view and a table*: in this option, >>>>>>>>>>>>>> the table metadata and view metadata are the same as the >>>>>>>>>>>>>> first option. The >>>>>>>>>>>>>> difference is that the commit process combines them, either >>>>>>>>>>>>>> by embedding a >>>>>>>>>>>>>> table metadata location in view metadata or by tracking both >>>>>>>>>>>>>> in the same >>>>>>>>>>>>>> catalog reference. >>>>>>>>>>>>>> - *A new metadata type*: this option is where we define a >>>>>>>>>>>>>> new metadata object that has view attributes, like SQL >>>>>>>>>>>>>> representations, >>>>>>>>>>>>>> along with table attributes, like partition specs and >>>>>>>>>>>>>> snapshots. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hopefully this is clear because I think much of the confusion >>>>>>>>>>>>>> is caused by different definitions. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The LoadTableResponse having optional metadata-location field >>>>>>>>>>>>>> implies that the object in the catalog no longer needs to hold a >>>>>>>>>>>>>> metadata >>>>>>>>>>>>>> file pointer >>>>>>>>>>>>>> >>>>>>>>>>>>>> The REST protocol has not removed the requirement for a >>>>>>>>>>>>>> metadata file, so I’m going to keep focused on the MV design >>>>>>>>>>>>>> options. >>>>>>>>>>>>>> >>>>>>>>>>>>>> When we say a MV can be a “new metadata type”, it does not >>>>>>>>>>>>>> mean it needs to define a completely brand new structure of the >>>>>>>>>>>>>> metadata >>>>>>>>>>>>>> content >>>>>>>>>>>>>> >>>>>>>>>>>>>> I’m making a distinction between separate metadata files for >>>>>>>>>>>>>> the table and the view and a combined metadata object, as above. >>>>>>>>>>>>>> >>>>>>>>>>>>>> We can define an “Iceberg MV” to be an object in a catalog, >>>>>>>>>>>>>> which has 1 table metadata file pointer, and 1 view metadata >>>>>>>>>>>>>> file pointer >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is the option I am referring to as a “combination of a >>>>>>>>>>>>>> view and a table”. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So to review my initial email, I don’t see a reason why a >>>>>>>>>>>>>> combined view and table is advantageous, either implemented by >>>>>>>>>>>>>> having a >>>>>>>>>>>>>> catalog reference with two metadata locations or embedding a >>>>>>>>>>>>>> table metadata >>>>>>>>>>>>>> location in view metadata. This would cause unnecessary >>>>>>>>>>>>>> dependence between >>>>>>>>>>>>>> the view and table in catalogs. I guess there’s an argument that >>>>>>>>>>>>>> you could >>>>>>>>>>>>>> load both table and view metadata locations at the same time. >>>>>>>>>>>>>> That hardly >>>>>>>>>>>>>> seems worth the trouble given the recent issues with adding >>>>>>>>>>>>>> views to the >>>>>>>>>>>>>> JDBC catalog. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I also think that once we decide on structure, we can make it >>>>>>>>>>>>>> possible for REST catalog implementations to do smart things, in >>>>>>>>>>>>>> a way that >>>>>>>>>>>>>> doesn’t put additional requirements on the underlying catalog >>>>>>>>>>>>>> store. For >>>>>>>>>>>>>> instance, we could specify how to send additional objects in a >>>>>>>>>>>>>> LoadViewResult, in case the catalog wants to pre-fetch table >>>>>>>>>>>>>> metadata. I >>>>>>>>>>>>>> think these optimizations are a later addition, after we define >>>>>>>>>>>>>> the >>>>>>>>>>>>>> relationship between views and tables. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jack, it sounds like you’re the proponent of a combined table >>>>>>>>>>>>>> and view (rather than a new metadata spec for a materialized >>>>>>>>>>>>>> view). What is >>>>>>>>>>>>>> the main motivation? It seems like you’re convinced of that >>>>>>>>>>>>>> approach, but I >>>>>>>>>>>>>> don’t understand the advantage it brings. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ryan >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Feb 29, 2024 at 12:26 PM Szehon Ho < >>>>>>>>>>>>>> szehon.apa...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes I mostly agree with the assessment. To clarify a few >>>>>>>>>>>>>>> minor points. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> is a materialized view a view and a separate table, a >>>>>>>>>>>>>>>> combination of the two (i.e. commits are combined), or a new >>>>>>>>>>>>>>>> metadata type? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For 'new metadata type', I consider mostly Jack's initial >>>>>>>>>>>>>>> proposal of a new Catalog MV object that has two references >>>>>>>>>>>>>>> (ViewMetadata + >>>>>>>>>>>>>>> TableMetadata). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The arguments that I see for a combined materialized view >>>>>>>>>>>>>>>> object are: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Regular views are separate, rather than being tables >>>>>>>>>>>>>>>> with SQL and no data so it would be inconsistent (“Iceberg >>>>>>>>>>>>>>>> view is just a >>>>>>>>>>>>>>>> table with no data but with representations defined. But we >>>>>>>>>>>>>>>> did not do >>>>>>>>>>>>>>>> that.”) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Materialized views are different objects in DDL >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Tables may be a superset of functionality needed for >>>>>>>>>>>>>>>> materialized views >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Tables are not typically exposed to end users — but >>>>>>>>>>>>>>>> this isn’t required by the separate view and table option >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> For completeness, there seem to be a few additional ones >>>>>>>>>>>>>>> (mentioned in the Slack and above messages). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Lack of spec change (to ViewMetadata). But as Jack >>>>>>>>>>>>>>> says it is a spec change (ie, to catalogs) >>>>>>>>>>>>>>> - A single call to get the View's StorageTable (versus >>>>>>>>>>>>>>> two calls) >>>>>>>>>>>>>>> - A more natural API, no opportunity for user to call >>>>>>>>>>>>>>> Catalog.dropTable() and renameTable() on storage table >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Thoughts: *I think the long discussion sessions we had on >>>>>>>>>>>>>>> Slack was fruitful for me, as seeing the API clarified some >>>>>>>>>>>>>>> things. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I was initially more in favor of MV being a new metadata >>>>>>>>>>>>>>> type (TableMetadata + ViewMetadata). But seeing most of the MV >>>>>>>>>>>>>>> operations >>>>>>>>>>>>>>> end up being ViewCatalog or Catalog operations, I am starting >>>>>>>>>>>>>>> to think >>>>>>>>>>>>>>> API-wise that it may not align with the new metadata type >>>>>>>>>>>>>>> (unless we define >>>>>>>>>>>>>>> MVCatalog and /MV REST endpoints, which then are boilerplate >>>>>>>>>>>>>>> wrappers). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Initially one question I had for option 'a view and a >>>>>>>>>>>>>>> separate table', was how to make this table reference >>>>>>>>>>>>>>> (metadata.json or >>>>>>>>>>>>>>> catalog reference). In the previous option, we had a precedent >>>>>>>>>>>>>>> of Catalog >>>>>>>>>>>>>>> references to Metadata, but not pointers between Metadatas. I >>>>>>>>>>>>>>> initially >>>>>>>>>>>>>>> saw the proposed Catalog's TableIdentifier pointer as >>>>>>>>>>>>>>> 'polluting' catalog >>>>>>>>>>>>>>> concerns in ViewMetadata. (I saw Catalog and ViewCatalog as a >>>>>>>>>>>>>>> layer above >>>>>>>>>>>>>>> TableMetadata and ViewMetadata). But I think Dan in the Slack >>>>>>>>>>>>>>> made a fair >>>>>>>>>>>>>>> point that ViewMetadata already is tightly bound with a >>>>>>>>>>>>>>> Catalog. In this >>>>>>>>>>>>>>> case, I think this approach does have its merits as well in >>>>>>>>>>>>>>> aligning >>>>>>>>>>>>>>> Catalog API's with the metadata. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> Szehon >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Feb 29, 2024 at 5:45 AM Jan Kaul >>>>>>>>>>>>>>> <jank...@mailbox.org.invalid> <jank...@mailbox.org.invalid> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I would like to provide my perspective on the question of >>>>>>>>>>>>>>>> what a materialized view is and elaborate on Jack's recent >>>>>>>>>>>>>>>> proposal to view >>>>>>>>>>>>>>>> a materialized view as a catalog concept. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Firstly, let's look at the role of the catalog. Every >>>>>>>>>>>>>>>> entity in the catalog has a *unique identifier*, and the >>>>>>>>>>>>>>>> catalog provides methods to create, load, and update these >>>>>>>>>>>>>>>> entities. An >>>>>>>>>>>>>>>> important thing to note is that the catalog methods exhibit >>>>>>>>>>>>>>>> two different >>>>>>>>>>>>>>>> behaviors: the *create and load methods deal with the >>>>>>>>>>>>>>>> entire entity*, while the *update(commit) method only >>>>>>>>>>>>>>>> deals with partial changes* to the entities. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In the context of our current discussion, materialized view >>>>>>>>>>>>>>>> (MV) metadata is a union of view and table metadata. The fact >>>>>>>>>>>>>>>> that the >>>>>>>>>>>>>>>> update method deals only with partial changes, enables us to >>>>>>>>>>>>>>>> *reuse >>>>>>>>>>>>>>>> the existing methods for updating tables and views*. For >>>>>>>>>>>>>>>> updates we don't have to define what constitutes an entire >>>>>>>>>>>>>>>> materialized >>>>>>>>>>>>>>>> view. Changes to a materialized view targeting the properties >>>>>>>>>>>>>>>> related to >>>>>>>>>>>>>>>> the view metadata could use the update(commit) view method. >>>>>>>>>>>>>>>> Similarly, >>>>>>>>>>>>>>>> changes targeting the properties related to the table metadata >>>>>>>>>>>>>>>> could use >>>>>>>>>>>>>>>> the update(commit) table method. This is great news because we >>>>>>>>>>>>>>>> don't have >>>>>>>>>>>>>>>> to redefine view and table commits (requirements, updates). >>>>>>>>>>>>>>>> This is shown in the fact that Jack uses the same operation >>>>>>>>>>>>>>>> to update the storage table for Option 1 and 3: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> // REST: POST >>>>>>>>>>>>>>>> /namespaces/db1/tables/mv1?materializedView=true >>>>>>>>>>>>>>>> // non-REST: update JSON files at table_metadata_location >>>>>>>>>>>>>>>> storageTable.newAppend().appendFile(...).commit(); >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The open question is *whether the create and load methods >>>>>>>>>>>>>>>> should treat the properties that constitute the MV metadata as >>>>>>>>>>>>>>>> two entities >>>>>>>>>>>>>>>> (View + Table) or one entity (new MV object)*. This is all >>>>>>>>>>>>>>>> part of Jack's proposal, where Option 1 proposes a new MV >>>>>>>>>>>>>>>> object, and >>>>>>>>>>>>>>>> Option 3 proposes two separate entities. The advantage of >>>>>>>>>>>>>>>> Option 1 is that >>>>>>>>>>>>>>>> it doesn't require two operations to load the metadata. On the >>>>>>>>>>>>>>>> other hand, >>>>>>>>>>>>>>>> the advantage of Option 3 is that no new operations or >>>>>>>>>>>>>>>> catalogs have to be >>>>>>>>>>>>>>>> defined. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In my opinion, defining a new representation for >>>>>>>>>>>>>>>> materialized views (Option 1) is generally the cleaner >>>>>>>>>>>>>>>> solution. However, I >>>>>>>>>>>>>>>> see a path where we could first introduce Option 3 and still >>>>>>>>>>>>>>>> have the >>>>>>>>>>>>>>>> possibility to transition to Option 1 if needed. The great >>>>>>>>>>>>>>>> thing about >>>>>>>>>>>>>>>> Option 3 is that it only requires minor changes to the current >>>>>>>>>>>>>>>> spec and is >>>>>>>>>>>>>>>> mostly implementation detail. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Therefore I would propose small additions to Jacks Option 3 >>>>>>>>>>>>>>>> that only introduce changes to the spec that are not specific >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> materialized views. The idea is to introduce boolean >>>>>>>>>>>>>>>> properties to be set >>>>>>>>>>>>>>>> on the creation of the view and the storage table that >>>>>>>>>>>>>>>> indicate that they >>>>>>>>>>>>>>>> belong to a materialized view. The view property >>>>>>>>>>>>>>>> "materialized" is set to >>>>>>>>>>>>>>>> "true" for a MV and "false" for a regular view. And the table >>>>>>>>>>>>>>>> property >>>>>>>>>>>>>>>> "storage_table" is set to "true" for a storage table and >>>>>>>>>>>>>>>> "false" for a >>>>>>>>>>>>>>>> regular table. The absence of these properties indicates a >>>>>>>>>>>>>>>> regular view or >>>>>>>>>>>>>>>> table. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ViewCatalog viewCatalog = (ViewCatalog) catalog; >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> // REST: GET /namespaces/db1/views/mv1 >>>>>>>>>>>>>>>> // non-REST: load JSON file at metadata_location >>>>>>>>>>>>>>>> View mv = viewCatalog.loadView(TableIdentifier.of("db1", >>>>>>>>>>>>>>>> "mv1")); >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> // REST: GET /namespaces/db1/tables/mv1 >>>>>>>>>>>>>>>> // non-REST: load JSON file at table_metadata_location if >>>>>>>>>>>>>>>> present >>>>>>>>>>>>>>>> Table storageTable = view.storageTable(); >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> // REST: POST /namespaces/db1/tables/mv1 >>>>>>>>>>>>>>>> // non-REST: update JSON file at table_metadata_location >>>>>>>>>>>>>>>> storageTable.newAppend().appendFile(...).commit(); >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We could then introduce a new requirement for views and >>>>>>>>>>>>>>>> tables called "AssertProperty" which could make sure to only >>>>>>>>>>>>>>>> perform >>>>>>>>>>>>>>>> updates that are inline with materialized views. The >>>>>>>>>>>>>>>> additional requirement >>>>>>>>>>>>>>>> can be seen as a general extension which does not need to be >>>>>>>>>>>>>>>> changed if we >>>>>>>>>>>>>>>> decide to got with Option 1 in the future. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Let me know what you think. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best wishes, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Jan >>>>>>>>>>>>>>>> On 29.02.24 04:09, Walaa Eldin Moustafa wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks Ryan for the insights. I agree that reusing existing >>>>>>>>>>>>>>>> metadata definitions and minimizing spec changes are very >>>>>>>>>>>>>>>> important. This >>>>>>>>>>>>>>>> also minimizes spec drift (between materialized views and >>>>>>>>>>>>>>>> views spec, and >>>>>>>>>>>>>>>> between materialized views and tables spec), and simplifies the >>>>>>>>>>>>>>>> implementation. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In an effort to take the discussion forward with concrete >>>>>>>>>>>>>>>> design options based on an end-to-end implementation, I have >>>>>>>>>>>>>>>> prototyped the >>>>>>>>>>>>>>>> implementation (and added Spark support) in this PR >>>>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/9830. I hope it >>>>>>>>>>>>>>>> helps us reach convergence faster. More details about some of >>>>>>>>>>>>>>>> the design >>>>>>>>>>>>>>>> options are discussed in the description of the PR. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Walaa. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Feb 28, 2024 at 6:20 PM Ryan Blue <b...@tabular.io> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I mean separate table and view metadata that is somehow >>>>>>>>>>>>>>>>> combined through a commit process. For instance, keeping a >>>>>>>>>>>>>>>>> pointer to a >>>>>>>>>>>>>>>>> table metadata file in a view metadata file or combining >>>>>>>>>>>>>>>>> commits to >>>>>>>>>>>>>>>>> reference both. I don't see the value in either option. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, Feb 28, 2024 at 5:05 PM Jack Ye < >>>>>>>>>>>>>>>>> yezhao...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks Ryan for the help to trace back to the root >>>>>>>>>>>>>>>>>> question! Just a clarification question regarding your reply >>>>>>>>>>>>>>>>>> before I reply >>>>>>>>>>>>>>>>>> further: what exactly does the option "a combination of the >>>>>>>>>>>>>>>>>> two (i.e. >>>>>>>>>>>>>>>>>> commits are combined)" mean? How is that different from "a >>>>>>>>>>>>>>>>>> new metadata >>>>>>>>>>>>>>>>>> type"? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -Jack >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>
