Sounds good, let's discuss this in person! I am a bit worried that we have quite a few critical topics going on right now on devlist, and this will take up a lot of time to discuss. If it ends up going for too long, l propose let us have a dedicated meeting, and I am more than happy to organize it.
Best, Jack Ye On Fri, Mar 1, 2024 at 12:48 PM Ryan Blue <b...@tabular.io> wrote: > Hey everyone, > > I think this thread has hit a point of diminishing returns and that we > still don't have a common understanding of what the options under > consideration actually are. > > Since we were already planning on discussing this at the next community > sync, I suggest we pick this up there and use that time to align on what > exactly we're considering. We can then start a new thread to lay out the > designs under consideration in more detail and then have a discussion about > trade-offs. > > Does that sound reasonable? > > Ryan > > > On Fri, Mar 1, 2024 at 11:09 AM Walaa Eldin Moustafa < > wa.moust...@gmail.com> wrote: > >> I am finding it hard to interpret the options concretely. I would also >> suggest breaking the expectation/outcome to milestones. Maybe it becomes >> easier if we agree to distinguish between an approach that is feasible in >> the near term and another in the long term, especially if the latter >> requires significant engine-side changes. >> >> Further, maybe it helps if we start with an option that fully reuses the >> existing spec, and see how we view it in comparison with the options >> discussed previously. I am sharing one below. It reuses the current spec of >> Iceberg views and tables by leveraging table properties to capture >> materialized view metadata. What is common (and not common) between this >> and the desired representations? >> >> The new properties are: >> Properties on a View: >> >> 1. >> >> *iceberg.materialized.view*: >> - *Type*: View property >> - *Purpose*: This property is used to mark whether a view is a >> materialized view. If set to true, the view is treated as a >> materialized view. This helps in differentiating between virtual and >> materialized views within the catalog and dictates specific handling >> and >> validation logic for materialized views. >> 2. >> >> *iceberg.materialized.view.storage.location*: >> - *Type*: View property >> - *Purpose*: Specifies the location of the storage table >> associated with the materialized view. This property is used for >> linking a >> materialized view with its corresponding storage table, enabling data >> management and query execution based on the stored data freshness. >> >> Properties on a Table: >> >> 1. *base.snapshot.[UUID]*: >> - *Type*: Table property >> - *Purpose*: These properties store the snapshot IDs of the base >> tables at the time the materialized view's data was last updated. Each >> property is prefixed with base.snapshot. followed by the UUID of >> the base table. They are used to track whether the materialized view's >> data >> is up to date with the base tables by comparing these snapshot IDs >> with the >> current snapshot IDs of the base tables. If all the base tables' >> current >> snapshot IDs match the ones stored in these properties, the >> materialized >> view's data is considered fresh. >> >> >> Thanks, >> Walaa. >> >> >> On Fri, Mar 1, 2024 at 9:15 AM Jack Ye <yezhao...@gmail.com> wrote: >> >>> > All of these approaches are aligned in one, specific way: the storage >>> table is an iceberg table. >>> >>> I do not think that is true. I think people are aligned that we would >>> like to re-use the Iceberg table metadata defined in the Iceberg table spec >>> to express the data in MV, but I don't think it goes that far to say it >>> must be an Iceberg table. Once you have that mindset, then of course option >>> 1 (separate table and view) is the only option. >>> >>> > I don't think that is necessary and it significantly increases the >>> complexity. >>> >>> And can you quantify what you mean by "significantly increases the >>> complexity"? Seems like a lot of concerns are coming from the tradeoff with >>> complexity. We probably all agree that using option 7 (a completely new >>> metadata type) is a lot of work from scratch, that is why it is not >>> favored. However, my understanding is that as long as we re-use the view >>> and table metadata, then the majority of the existing logic can be reused. >>> I think what we have gone through in Slack to draft the rough Java API >>> shape helps here, because people can estimate the amount of effort required >>> to implement it. And I don't think they are **significantly** more complex >>> to implement. Could you elaborate more about the complexity that you >>> imagine? >>> >>> -Jack >>> >>> >>> >>> On Fri, Mar 1, 2024 at 8:57 AM Daniel Weeks <daniel.c.we...@gmail.com> >>> wrote: >>> >>>> I feel I've been most vocal about pushing back against options 2+ (or >>>> Ryan's categories of combined table/view, or new metadata type), so I'll >>>> try to expand on my reasoning. >>>> >>>> I understand the appeal of creating a design where we encapsulate the >>>> view/storage from both a structural and performance standpoint, but I don't >>>> think that is necessary and it significantly increases the complexity. >>>> >>>> All of these approaches are aligned in one, specific way: the storage >>>> table is an iceberg table. >>>> >>>> Because of this, all the behaviors and requirements still apply to >>>> these tables. They need to be maintained (snapshot cleanup, orphan files), >>>> in cases need to be optimized (compaction, manifest rewrites), they need to >>>> be able to be inspected (this will be even more important with MV since >>>> staleness can produce different results and questions will arise about what >>>> state the storage table was in). There may be cases where the tables need >>>> to be managed directly. >>>> >>>> Anywhere we deviate from the existing constructs/commit/access for >>>> tables, we will ultimately have to then unwrap to re-expose the underlying >>>> Iceberg behavior. This creates unnecessary complexity in the library/API >>>> layer, which are not the primary interface users will have with >>>> materialized views where an engine is almost entirely necessary to interact >>>> with the dataset. >>>> >>>> As to the performance concerns around option 1, I think we're >>>> overstating the downsides. It really comes down to how many metadata loads >>>> are necessary and evaluating freshness would likely be the real bottleneck >>>> as it involves potentially loading many tables. All of the options are on >>>> the same order of performance for the metadata and table loads. >>>> >>>> As to the visibility of tables and whether they're registered in the >>>> catalog, I think registering in the catalog is the right approach so that >>>> the tables are still addressable for maintenance/etc. The visibility of >>>> the storage table is a catalog implementation decision and shouldn't be a >>>> requirement of the MV spec (I can see cases for both and it isn't necessary >>>> to dictate a behavior). >>>> >>>> I'm still strongly in favor of Option 1 (separate table and view) for >>>> these reasons. >>>> >>>> -Dan >>>> >>>> >>>> >>>> On Thu, Feb 29, 2024 at 11:07 PM Jack Ye <yezhao...@gmail.com> wrote: >>>> >>>>> > Jack, it sounds like you’re the proponent of a combined table and >>>>> view (rather than a new metadata spec for a materialized view). What is >>>>> the >>>>> main motivation? It seems like you’re convinced of that approach, but I >>>>> don’t understand the advantage it brings. >>>>> >>>>> Sorry I have to make a Google Sheet to capture all the options we have >>>>> discussed so far, I wanted to use the existing Google Doc, but it has >>>>> really bad table/sheet support... >>>>> >>>>> >>>>> https://docs.google.com/spreadsheets/d/1a0tlyh8f2ft2SepE7H3bgoY2A0q5IILgzAsJMnwjTBs/edit#gid=0 >>>>> >>>>> I have listed all the options, with how they are implemented and some >>>>> important considerations we have discussed so far. Note that: >>>>> 1. This sheet currently excludes the lineage information, which we can >>>>> discuss more later after the current topic is resolved. >>>>> 2. I removed the considerations for REST integration since from the >>>>> other thread we have clarified that they should be considered completely >>>>> separately. >>>>> >>>>> *Why I come as a proponent of having a new MV object with table and >>>>> view metadata file pointer* >>>>> >>>>> In my sheet, there are 3 options that do not have major problems: >>>>> Option 2: Add storage table metadata file pointer in view object >>>>> Option 5: New MV object with table and view metadata file pointer >>>>> Option 6: New MV spec with table and view metadata >>>>> >>>>> I originally excluded option 2 because I think it does not align with >>>>> the REST spec, but after the other discussion thread about "Inconsistency >>>>> between REST spec and table/view spec", I think my original concern no >>>>> longer holds true so now I put it back. And based on my personal >>>>> preference that MV is an independent object that should be separated from >>>>> view and table, plus the fact that option 5 is probably less work than >>>>> option 6 for implementation, that is how I come as a proponent of option 5 >>>>> at this moment. >>>>> >>>>> >>>>> *Regarding Ryan's evaluation framework* >>>>> >>>>> I think we need to reconcile this sheet with Ryan's evaluation >>>>> framework. That framework categorization puts option 2, 3, 4, 5, 6 all >>>>> under the same category of "A combination of a view and a table" and >>>>> concludes that they don't have any advantage for the same set of reasons. >>>>> But those reasons are not really convincing to me so let's talk about them >>>>> in more detail. >>>>> >>>>> (1) You said "I don’t see a reason why a combined view and table is >>>>> advantageous" as "this would cause unnecessary dependence between the view >>>>> and table in catalogs." What dependency exactly do you mean here? And why >>>>> is that unnecessary, given there has to be some sort of dependency anyway >>>>> unless we go with option 5 or 6? >>>>> >>>>> (2) You said "I guess there’s an argument that you could load both >>>>> table and view metadata locations at the same time. That hardly seems >>>>> worth >>>>> the trouble". I disagree with that. Catalog interaction performance is >>>>> critical to at least everyone working in EMR and Athena, and MV itself as >>>>> an acceleration approach needs to be as fast as possible. >>>>> >>>>> I have put 3 key operations in the doc that I think matters for MV >>>>> during interactions with engine: >>>>> 1. refreshes storage table >>>>> 2. get the storage table of the MV >>>>> 3. if stale, get the view SQL >>>>> >>>>> And option 1 clearly falls short with 4 sequential steps required to >>>>> load a storage table. You mentioned "recent issues with adding views to >>>>> the >>>>> JDBC catalog" in this topic, could you explain a bit more? >>>>> >>>>> (3) You said "I also think that once we decide on structure, we can >>>>> make it possible for REST catalog implementations to do smart things, in a >>>>> way that doesn’t put additional requirements on the underlying catalog >>>>> store." If REST is fully compatible with Iceberg spec then I have no >>>>> problem with this statement. However, as we discussed in the other thread, >>>>> it is not the case. In the current state, I think the sequence of action >>>>> should be to evolve the Iceberg table/view spec (or add a MV spec) first, >>>>> and then think about how REST can incorporate it or do smart things that >>>>> are not Iceberg spec compliant. Do you agree with that? >>>>> >>>>> (4) You said the table identifier pointer "is a problem we need to >>>>> solve generally because a materialized table needs to be able to track the >>>>> upstream state of tables that were used". I don't think that is a reason >>>>> to >>>>> choose to use a table identifier pointer for a storage table. The issue is >>>>> not about using a table identifier pointer. It is about exposing the >>>>> storage table as a separate entity in the catalog, which is what people do >>>>> not like and is already discussed in length in Jan's question 3 (also >>>>> linked in the sheet). I agree with that statement, because without a REST >>>>> implementation that can magically hide the storage table, this model adds >>>>> additional burden regarding compliance and data governance for any other >>>>> non-REST catalog implementations that are compliant to the Iceberg spec. >>>>> Many mechanisms need to be built in a catalog to hide, protect, maintain, >>>>> recycle the storage table, that can be avoided by using other approaches. >>>>> I >>>>> think we should reach a consensus about that and discuss further if you do >>>>> not agree. >>>>> >>>>> Best, >>>>> Jack Ye >>>>> >>>>> On Thu, Feb 29, 2024 at 10:53 PM Jan Kaul <jank...@mailbox.org.invalid> >>>>> wrote: >>>>> >>>>>> Hi Ryan, we actually discussed your categories in this question >>>>>> <https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit?pli=1#heading=h.y70rtfhi9qxi>. >>>>>> Where your categories correspond to the following designs: >>>>>> >>>>>> - Separate table and view => Design 1 >>>>>> - Combination of view and table => Design 2 >>>>>> - A new metadata type => Design 4 >>>>>> >>>>>> Jan >>>>>> On 01.03.24 00:03, Ryan Blue wrote: >>>>>> >>>>>> Looks like it wasn’t clear what I meant for the 3 categories, so I’ll >>>>>> be more specific: >>>>>> >>>>>> - *Separate table and view*: this option is to have the objects >>>>>> that we have today, with extra metadata. Commit processes are >>>>>> separate: >>>>>> committing to the table doesn’t alter the view and committing to the >>>>>> view >>>>>> doesn’t change the table. However, changing the view can make it so >>>>>> the >>>>>> table is no longer useful as a materialization. >>>>>> - *A combination of a view and a table*: in this option, the >>>>>> table metadata and view metadata are the same as the first option. The >>>>>> difference is that the commit process combines them, either by >>>>>> embedding a >>>>>> table metadata location in view metadata or by tracking both in the >>>>>> same >>>>>> catalog reference. >>>>>> - *A new metadata type*: this option is where we define a new >>>>>> metadata object that has view attributes, like SQL representations, >>>>>> along >>>>>> with table attributes, like partition specs and snapshots. >>>>>> >>>>>> Hopefully this is clear because I think much of the confusion is >>>>>> caused by different definitions. >>>>>> >>>>>> The LoadTableResponse having optional metadata-location field implies >>>>>> that the object in the catalog no longer needs to hold a metadata file >>>>>> pointer >>>>>> >>>>>> The REST protocol has not removed the requirement for a metadata >>>>>> file, so I’m going to keep focused on the MV design options. >>>>>> >>>>>> When we say a MV can be a “new metadata type”, it does not mean it >>>>>> needs to define a completely brand new structure of the metadata content >>>>>> >>>>>> I’m making a distinction between separate metadata files for the >>>>>> table and the view and a combined metadata object, as above. >>>>>> >>>>>> We can define an “Iceberg MV” to be an object in a catalog, which has >>>>>> 1 table metadata file pointer, and 1 view metadata file pointer >>>>>> >>>>>> This is the option I am referring to as a “combination of a view and >>>>>> a table”. >>>>>> >>>>>> So to review my initial email, I don’t see a reason why a combined >>>>>> view and table is advantageous, either implemented by having a catalog >>>>>> reference with two metadata locations or embedding a table metadata >>>>>> location in view metadata. This would cause unnecessary dependence >>>>>> between >>>>>> the view and table in catalogs. I guess there’s an argument that you >>>>>> could >>>>>> load both table and view metadata locations at the same time. That hardly >>>>>> seems worth the trouble given the recent issues with adding views to the >>>>>> JDBC catalog. >>>>>> >>>>>> I also think that once we decide on structure, we can make it >>>>>> possible for REST catalog implementations to do smart things, in a way >>>>>> that >>>>>> doesn’t put additional requirements on the underlying catalog store. For >>>>>> instance, we could specify how to send additional objects in a >>>>>> LoadViewResult, in case the catalog wants to pre-fetch table metadata. I >>>>>> think these optimizations are a later addition, after we define the >>>>>> relationship between views and tables. >>>>>> >>>>>> Jack, it sounds like you’re the proponent of a combined table and >>>>>> view (rather than a new metadata spec for a materialized view). What is >>>>>> the >>>>>> main motivation? It seems like you’re convinced of that approach, but I >>>>>> don’t understand the advantage it brings. >>>>>> >>>>>> Ryan >>>>>> >>>>>> On Thu, Feb 29, 2024 at 12:26 PM Szehon Ho <szehon.apa...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi >>>>>>> >>>>>>> Yes I mostly agree with the assessment. To clarify a few minor >>>>>>> points. >>>>>>> >>>>>>> is a materialized view a view and a separate table, a combination of >>>>>>>> the two (i.e. commits are combined), or a new metadata type? >>>>>>> >>>>>>> >>>>>>> For 'new metadata type', I consider mostly Jack's initial proposal >>>>>>> of a new Catalog MV object that has two references (ViewMetadata + >>>>>>> TableMetadata). >>>>>>> >>>>>>> The arguments that I see for a combined materialized view object >>>>>>>> are: >>>>>>>> >>>>>>>> - Regular views are separate, rather than being tables with SQL >>>>>>>> and no data so it would be inconsistent (“Iceberg view is just a >>>>>>>> table with >>>>>>>> no data but with representations defined. But we did not do that.”) >>>>>>>> >>>>>>>> >>>>>>>> - Materialized views are different objects in DDL >>>>>>>> >>>>>>>> >>>>>>>> - Tables may be a superset of functionality needed for >>>>>>>> materialized views >>>>>>>> >>>>>>>> >>>>>>>> - Tables are not typically exposed to end users — but this >>>>>>>> isn’t required by the separate view and table option >>>>>>>> >>>>>>>> For completeness, there seem to be a few additional ones (mentioned >>>>>>> in the Slack and above messages). >>>>>>> >>>>>>> - Lack of spec change (to ViewMetadata). But as Jack says it is >>>>>>> a spec change (ie, to catalogs) >>>>>>> - A single call to get the View's StorageTable (versus two calls) >>>>>>> - A more natural API, no opportunity for user to call >>>>>>> Catalog.dropTable() and renameTable() on storage table >>>>>>> >>>>>>> >>>>>>> *Thoughts: *I think the long discussion sessions we had on Slack >>>>>>> was fruitful for me, as seeing the API clarified some things. >>>>>>> >>>>>>> I was initially more in favor of MV being a new metadata type >>>>>>> (TableMetadata + ViewMetadata). But seeing most of the MV operations >>>>>>> end >>>>>>> up being ViewCatalog or Catalog operations, I am starting to think >>>>>>> API-wise >>>>>>> that it may not align with the new metadata type (unless we define >>>>>>> MVCatalog and /MV REST endpoints, which then are boilerplate wrappers). >>>>>>> >>>>>>> Initially one question I had for option 'a view and a separate >>>>>>> table', was how to make this table reference (metadata.json or catalog >>>>>>> reference). In the previous option, we had a precedent of Catalog >>>>>>> references to Metadata, but not pointers between Metadatas. I initially >>>>>>> saw the proposed Catalog's TableIdentifier pointer as 'polluting' >>>>>>> catalog >>>>>>> concerns in ViewMetadata. (I saw Catalog and ViewCatalog as a layer >>>>>>> above >>>>>>> TableMetadata and ViewMetadata). But I think Dan in the Slack made a >>>>>>> fair >>>>>>> point that ViewMetadata already is tightly bound with a Catalog. In >>>>>>> this >>>>>>> case, I think this approach does have its merits as well in aligning >>>>>>> Catalog API's with the metadata. >>>>>>> >>>>>>> Thanks >>>>>>> Szehon >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Feb 29, 2024 at 5:45 AM Jan Kaul >>>>>>> <jank...@mailbox.org.invalid> <jank...@mailbox.org.invalid> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I would like to provide my perspective on the question of what a >>>>>>>> materialized view is and elaborate on Jack's recent proposal to view a >>>>>>>> materialized view as a catalog concept. >>>>>>>> >>>>>>>> Firstly, let's look at the role of the catalog. Every entity in the >>>>>>>> catalog has a *unique identifier*, and the catalog provides >>>>>>>> methods to create, load, and update these entities. An important thing >>>>>>>> to >>>>>>>> note is that the catalog methods exhibit two different behaviors: the >>>>>>>> *create >>>>>>>> and load methods deal with the entire entity*, while the >>>>>>>> *update(commit) >>>>>>>> method only deals with partial changes* to the entities. >>>>>>>> >>>>>>>> In the context of our current discussion, materialized view (MV) >>>>>>>> metadata is a union of view and table metadata. The fact that the >>>>>>>> update >>>>>>>> method deals only with partial changes, enables us to *reuse the >>>>>>>> existing methods for updating tables and views*. For updates we >>>>>>>> don't have to define what constitutes an entire materialized view. >>>>>>>> Changes >>>>>>>> to a materialized view targeting the properties related to the view >>>>>>>> metadata could use the update(commit) view method. Similarly, changes >>>>>>>> targeting the properties related to the table metadata could use the >>>>>>>> update(commit) table method. This is great news because we don't have >>>>>>>> to >>>>>>>> redefine view and table commits (requirements, updates). >>>>>>>> This is shown in the fact that Jack uses the same operation to >>>>>>>> update the storage table for Option 1 and 3: >>>>>>>> >>>>>>>> // REST: POST /namespaces/db1/tables/mv1?materializedView=true >>>>>>>> // non-REST: update JSON files at table_metadata_location >>>>>>>> storageTable.newAppend().appendFile(...).commit(); >>>>>>>> >>>>>>>> The open question is *whether the create and load methods should >>>>>>>> treat the properties that constitute the MV metadata as two entities >>>>>>>> (View >>>>>>>> + Table) or one entity (new MV object)*. This is all part of >>>>>>>> Jack's proposal, where Option 1 proposes a new MV object, and Option 3 >>>>>>>> proposes two separate entities. The advantage of Option 1 is that it >>>>>>>> doesn't require two operations to load the metadata. On the other >>>>>>>> hand, the >>>>>>>> advantage of Option 3 is that no new operations or catalogs have to be >>>>>>>> defined. >>>>>>>> >>>>>>>> In my opinion, defining a new representation for materialized views >>>>>>>> (Option 1) is generally the cleaner solution. However, I see a path >>>>>>>> where >>>>>>>> we could first introduce Option 3 and still have the possibility to >>>>>>>> transition to Option 1 if needed. The great thing about Option 3 is >>>>>>>> that it >>>>>>>> only requires minor changes to the current spec and is mostly >>>>>>>> implementation detail. >>>>>>>> >>>>>>>> Therefore I would propose small additions to Jacks Option 3 that >>>>>>>> only introduce changes to the spec that are not specific to >>>>>>>> materialized >>>>>>>> views. The idea is to introduce boolean properties to be set on the >>>>>>>> creation of the view and the storage table that indicate that they >>>>>>>> belong >>>>>>>> to a materialized view. The view property "materialized" is set to >>>>>>>> "true" >>>>>>>> for a MV and "false" for a regular view. And the table property >>>>>>>> "storage_table" is set to "true" for a storage table and "false" for a >>>>>>>> regular table. The absence of these properties indicates a regular >>>>>>>> view or >>>>>>>> table. >>>>>>>> >>>>>>>> ViewCatalog viewCatalog = (ViewCatalog) catalog; >>>>>>>> >>>>>>>> // REST: GET /namespaces/db1/views/mv1 >>>>>>>> // non-REST: load JSON file at metadata_location >>>>>>>> View mv = viewCatalog.loadView(TableIdentifier.of("db1", "mv1")); >>>>>>>> >>>>>>>> // REST: GET /namespaces/db1/tables/mv1 >>>>>>>> // non-REST: load JSON file at table_metadata_location if present >>>>>>>> Table storageTable = view.storageTable(); >>>>>>>> >>>>>>>> // REST: POST /namespaces/db1/tables/mv1 >>>>>>>> // non-REST: update JSON file at table_metadata_location >>>>>>>> storageTable.newAppend().appendFile(...).commit(); >>>>>>>> >>>>>>>> We could then introduce a new requirement for views and tables >>>>>>>> called "AssertProperty" which could make sure to only perform updates >>>>>>>> that >>>>>>>> are inline with materialized views. The additional requirement can be >>>>>>>> seen >>>>>>>> as a general extension which does not need to be changed if we decide >>>>>>>> to >>>>>>>> got with Option 1 in the future. >>>>>>>> >>>>>>>> Let me know what you think. >>>>>>>> >>>>>>>> Best wishes, >>>>>>>> >>>>>>>> Jan >>>>>>>> On 29.02.24 04:09, Walaa Eldin Moustafa wrote: >>>>>>>> >>>>>>>> Thanks Ryan for the insights. I agree that reusing existing >>>>>>>> metadata definitions and minimizing spec changes are very important. >>>>>>>> This >>>>>>>> also minimizes spec drift (between materialized views and views spec, >>>>>>>> and >>>>>>>> between materialized views and tables spec), and simplifies the >>>>>>>> implementation. >>>>>>>> >>>>>>>> In an effort to take the discussion forward with concrete design >>>>>>>> options based on an end-to-end implementation, I have prototyped the >>>>>>>> implementation (and added Spark support) in this PR >>>>>>>> https://github.com/apache/iceberg/pull/9830. I hope it helps us >>>>>>>> reach convergence faster. More details about some of the design >>>>>>>> options are >>>>>>>> discussed in the description of the PR. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Walaa. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Feb 28, 2024 at 6:20 PM Ryan Blue <b...@tabular.io> wrote: >>>>>>>> >>>>>>>>> I mean separate table and view metadata that is somehow combined >>>>>>>>> through a commit process. For instance, keeping a pointer to a table >>>>>>>>> metadata file in a view metadata file or combining commits to >>>>>>>>> reference >>>>>>>>> both. I don't see the value in either option. >>>>>>>>> >>>>>>>>> On Wed, Feb 28, 2024 at 5:05 PM Jack Ye <yezhao...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks Ryan for the help to trace back to the root question! Just >>>>>>>>>> a clarification question regarding your reply before I reply >>>>>>>>>> further: what >>>>>>>>>> exactly does the option "a combination of the two (i.e. commits are >>>>>>>>>> combined)" mean? How is that different from "a new metadata type"? >>>>>>>>>> >>>>>>>>>> -Jack >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Feb 28, 2024 at 2:10 PM Ryan Blue <b...@tabular.io> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> I’m catching up on this conversation, so hopefully I can bring a >>>>>>>>>>> fresh perspective. >>>>>>>>>>> >>>>>>>>>>> Jack already pointed out that we need to start from the basics >>>>>>>>>>> and I agree with that. Let’s remove voting at this point. Right now >>>>>>>>>>> is the >>>>>>>>>>> time for discussing trade-offs, not lining up and taking sides. I >>>>>>>>>>> realize >>>>>>>>>>> that wasn’t the intent with adding a vote, but that’s almost always >>>>>>>>>>> the >>>>>>>>>>> result. It’s too easy to use it as a stand-in for consensus and >>>>>>>>>>> move on >>>>>>>>>>> prematurely. I get the impression from the swirl in Slack that >>>>>>>>>>> discussion >>>>>>>>>>> has moved ahead of agreement. >>>>>>>>>>> >>>>>>>>>>> We’re still at the most basic question: is a materialized view a >>>>>>>>>>> view and a separate table, a combination of the two (i.e. commits >>>>>>>>>>> are >>>>>>>>>>> combined), or a new metadata type? >>>>>>>>>>> >>>>>>>>>>> For now, I’m ignoring whether the “separate table” is some kind >>>>>>>>>>> of “system table” (meaning hidden?) or if it is exposed in the >>>>>>>>>>> catalog. >>>>>>>>>>> That’s a later choice (already pointed out) and, I suspect, it >>>>>>>>>>> should be >>>>>>>>>>> delegated to catalog implementations. >>>>>>>>>>> >>>>>>>>>>> To simplify this a little, I think that we can eliminate the >>>>>>>>>>> option to combine table and view commits. I don’t think there is a >>>>>>>>>>> reason >>>>>>>>>>> to combine the two. If separate, a table would track the view >>>>>>>>>>> version used >>>>>>>>>>> along with freshness information for referenced tables. If the >>>>>>>>>>> table is >>>>>>>>>>> automatically skipped when the version no longer matches the view, >>>>>>>>>>> then no >>>>>>>>>>> action needs to happen when a view definition changes. Similarly, >>>>>>>>>>> the table >>>>>>>>>>> can be updated independently without needing to also swap view >>>>>>>>>>> metadata. >>>>>>>>>>> This also aligns with the idea from the original doc that there can >>>>>>>>>>> be >>>>>>>>>>> multiple materialization tables for a view. Each should operate >>>>>>>>>>> independently unless I’m missing something >>>>>>>>>>> >>>>>>>>>>> I don’t think the last paragraph’s conclusion is contentious so >>>>>>>>>>> I’ll move on, but please stop here and reply if you disagree! >>>>>>>>>>> >>>>>>>>>>> That leaves the main two options, a view and a separate table >>>>>>>>>>> linked by metadata, or, combined materialized view metadata. >>>>>>>>>>> >>>>>>>>>>> As the doc notes, the separate view and table option is simpler >>>>>>>>>>> because it reuses existing metadata definitions and falls back to >>>>>>>>>>> simple >>>>>>>>>>> views. That is a significantly smaller spec and small is very, very >>>>>>>>>>> important when it comes to specs. I think that the argument for a >>>>>>>>>>> new >>>>>>>>>>> definition of a materialized view needs to overcome this >>>>>>>>>>> disadvantage. >>>>>>>>>>> >>>>>>>>>>> The arguments that I see for a combined materialized view object >>>>>>>>>>> are: >>>>>>>>>>> >>>>>>>>>>> - Regular views are separate, rather than being tables with >>>>>>>>>>> SQL and no data so it would be inconsistent (“Iceberg view is >>>>>>>>>>> just a table >>>>>>>>>>> with no data but with representations defined. But we did not do >>>>>>>>>>> that.”) >>>>>>>>>>> - Materialized views are different objects in DDL >>>>>>>>>>> - Tables may be a superset of functionality needed for >>>>>>>>>>> materialized views >>>>>>>>>>> - Tables are not typically exposed to end users — but this >>>>>>>>>>> isn’t required by the separate view and table option >>>>>>>>>>> >>>>>>>>>>> Am I missing any arguments for combined metadata? >>>>>>>>>>> >>>>>>>>>>> Ryan >>>>>>>>>>> -- >>>>>>>>>>> Ryan Blue >>>>>>>>>>> Tabular >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Ryan Blue >>>>>>>>> Tabular >>>>>>>>> >>>>>>>> >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> Tabular >>>>>> >>>>>> > > -- > Ryan Blue > Tabular >
