Ok since the option "A combination of a view and a table" also has some
sort of a pointer (from the view to the table in the view metadata), and it
is rejected, I think the key distinction with the first option "Separate
table and view", is that in the second option, a specific version of the
table is tracked, e.g., "/path/v_*N*.metadata.json", however, in the first
option, we track the table, not a particular version of it. Is this
understanding correct?
On Thu, Feb 29, 2024 at 4:38 PM Ryan Blue <b...@tabular.io> wrote:
> > Ryan, in the option "Separate table and view", will there be a reference
> (or pointer) to the table from the view metadata?
>
> Yes. And this is a problem we need to solve generally because a
> materialized table needs to be able to track the upstream state of tables
> that were used. I think it would be one or more identifiers stored in a
> view metadata field, one for each materialization. But there's a lot of
> assumptions about how we come out on these questions before we get to how
> to store metadata.
>
> On Thu, Feb 29, 2024 at 4:35 PM Walaa Eldin Moustafa <
> wa.moust...@gmail.com> wrote:
>
>> Ryan, in the option "Separate table and view", will there be a reference
>> (or pointer) to the table from the view metadata? Since the option of
>> "embedding a table metadata location in view metadata" is not preferred, it
>> is not clear how to associate the table with the view in the "Separate
>> table and view" option without such a pointer.
>>
>> Thanks,
>> Walaa.
>>
>>
>> On Thu, Feb 29, 2024 at 3:04 PM Ryan Blue <b...@tabular.io> wrote:
>>
>>> Looks like it wasn’t clear what I meant for the 3 categories, so I’ll be
>>> more specific:
>>>
>>> - *Separate table and view*: this option is to have the objects that
>>> we have today, with extra metadata. Commit processes are separate:
>>> committing to the table doesn’t alter the view and committing to the view
>>> doesn’t change the table. However, changing the view can make it so the
>>> table is no longer useful as a materialization.
>>> - *A combination of a view and a table*: in this option, the table
>>> metadata and view metadata are the same as the first option. The
>>> difference
>>> is that the commit process combines them, either by embedding a table
>>> metadata location in view metadata or by tracking both in the same
>>> catalog
>>> reference.
>>> - *A new metadata type*: this option is where we define a new
>>> metadata object that has view attributes, like SQL representations, along
>>> with table attributes, like partition specs and snapshots.
>>>
>>> Hopefully this is clear because I think much of the confusion is caused
>>> by different definitions.
>>>
>>> The LoadTableResponse having optional metadata-location field implies
>>> that the object in the catalog no longer needs to hold a metadata file
>>> pointer
>>>
>>> The REST protocol has not removed the requirement for a metadata file,
>>> so I’m going to keep focused on the MV design options.
>>>
>>> When we say a MV can be a “new metadata type”, it does not mean it needs
>>> to define a completely brand new structure of the metadata content
>>>
>>> I’m making a distinction between separate metadata files for the table
>>> and the view and a combined metadata object, as above.
>>>
>>> We can define an “Iceberg MV” to be an object in a catalog, which has 1
>>> table metadata file pointer, and 1 view metadata file pointer
>>>
>>> This is the option I am referring to as a “combination of a view and a
>>> table”.
>>>
>>> So to review my initial email, I don’t see a reason why a combined view
>>> and table is advantageous, either implemented by having a catalog reference
>>> with two metadata locations or embedding a table metadata location in view
>>> metadata. This would cause unnecessary dependence between the view and
>>> table in catalogs. I guess there’s an argument that you could load both
>>> table and view metadata locations at the same time. That hardly seems worth
>>> the trouble given the recent issues with adding views to the JDBC catalog.
>>>
>>> I also think that once we decide on structure, we can make it possible
>>> for REST catalog implementations to do smart things, in a way that doesn’t
>>> put additional requirements on the underlying catalog store. For instance,
>>> we could specify how to send additional objects in a LoadViewResult, in
>>> case the catalog wants to pre-fetch table metadata. I think these
>>> optimizations are a later addition, after we define the relationship
>>> between views and tables.
>>>
>>> Jack, it sounds like you’re the proponent of a combined table and view
>>> (rather than a new metadata spec for a materialized view). What is the main
>>> motivation? It seems like you’re convinced of that approach, but I don’t
>>> understand the advantage it brings.
>>>
>>> Ryan
>>>
>>> On Thu, Feb 29, 2024 at 12:26 PM Szehon Ho <szehon.apa...@gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> Yes I mostly agree with the assessment. To clarify a few minor points.
>>>>
>>>> is a materialized view a view and a separate table, a combination of
>>>>> the two (i.e. commits are combined), or a new metadata type?
>>>>
>>>>
>>>> For 'new metadata type', I consider mostly Jack's initial proposal of a
>>>> new Catalog MV object that has two references (ViewMetadata +
>>>> TableMetadata).
>>>>
>>>> The arguments that I see for a combined materialized view object are:
>>>>>
>>>>> - Regular views are separate, rather than being tables with SQL
>>>>> and no data so it would be inconsistent (“Iceberg view is just a table
>>>>> with
>>>>> no data but with representations defined. But we did not do that.”)
>>>>>
>>>>>
>>>>> - Materialized views are different objects in DDL
>>>>>
>>>>>
>>>>> - Tables may be a superset of functionality needed for
>>>>> materialized views
>>>>>
>>>>>
>>>>> - Tables are not typically exposed to end users — but this isn’t
>>>>> required by the separate view and table option
>>>>>
>>>>> For completeness, there seem to be a few additional ones (mentioned in
>>>> the Slack and above messages).
>>>>
>>>> - Lack of spec change (to ViewMetadata). But as Jack says it is a
>>>> spec change (ie, to catalogs)
>>>> - A single call to get the View's StorageTable (versus two calls)
>>>> - A more natural API, no opportunity for user to call
>>>> Catalog.dropTable() and renameTable() on storage table
>>>>
>>>>
>>>> *Thoughts: *I think the long discussion sessions we had on Slack
>>>> was fruitful for me, as seeing the API clarified some things.
>>>>
>>>> I was initially more in favor of MV being a new metadata type
>>>> (TableMetadata + ViewMetadata). But seeing most of the MV operations end
>>>> up being ViewCatalog or Catalog operations, I am starting to think API-wise
>>>> that it may not align with the new metadata type (unless we define
>>>> MVCatalog and /MV REST endpoints, which then are boilerplate wrappers).
>>>>
>>>> Initially one question I had for option 'a view and a separate table',
>>>> was how to make this table reference (metadata.json or catalog reference).
>>>> In the previous option, we had a precedent of Catalog references to
>>>> Metadata, but not pointers between Metadatas. I initially saw the proposed
>>>> Catalog's TableIdentifier pointer as 'polluting' catalog concerns in
>>>> ViewMetadata. (I saw Catalog and ViewCatalog as a layer above
>>>> TableMetadata and ViewMetadata). But I think Dan in the Slack made a fair
>>>> point that ViewMetadata already is tightly bound with a Catalog. In this
>>>> case, I think this approach does have its merits as well in aligning
>>>> Catalog API's with the metadata.
>>>>
>>>> Thanks
>>>> Szehon
>>>>
>>>>
>>>>
>>>> On Thu, Feb 29, 2024 at 5:45 AM Jan Kaul <jank...@mailbox.org.invalid>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I would like to provide my perspective on the question of what a
>>>>> materialized view is and elaborate on Jack's recent proposal to view a
>>>>> materialized view as a catalog concept.
>>>>>
>>>>> Firstly, let's look at the role of the catalog. Every entity in the
>>>>> catalog has a *unique identifier*, and the catalog provides methods
>>>>> to create, load, and update these entities. An important thing to note is
>>>>> that the catalog methods exhibit two different behaviors: the *create
>>>>> and load methods deal with the entire entity*, while the *update(commit)
>>>>> method only deals with partial changes* to the entities.
>>>>>
>>>>> In the context of our current discussion, materialized view (MV)
>>>>> metadata is a union of view and table metadata. The fact that the update
>>>>> method deals only with partial changes, enables us to *reuse the
>>>>> existing methods for updating tables and views*. For updates we don't
>>>>> have to define what constitutes an entire materialized view. Changes to a
>>>>> materialized view targeting the properties related to the view metadata
>>>>> could use the update(commit) view method. Similarly, changes targeting the
>>>>> properties related to the table metadata could use the update(commit)
>>>>> table
>>>>> method. This is great news because we don't have to redefine view and
>>>>> table
>>>>> commits (requirements, updates).
>>>>> This is shown in the fact that Jack uses the same operation to update
>>>>> the storage table for Option 1 and 3:
>>>>>
>>>>> // REST: POST /namespaces/db1/tables/mv1?materializedView=true
>>>>> // non-REST: update JSON files at table_metadata_location
>>>>> storageTable.newAppend().appendFile(...).commit();
>>>>>
>>>>> The open question is *whether the create and load methods should
>>>>> treat the properties that constitute the MV metadata as two entities (View
>>>>> + Table) or one entity (new MV object)*. This is all part of Jack's
>>>>> proposal, where Option 1 proposes a new MV object, and Option 3 proposes
>>>>> two separate entities. The advantage of Option 1 is that it doesn't
>>>>> require
>>>>> two operations to load the metadata. On the other hand, the advantage of
>>>>> Option 3 is that no new operations or catalogs have to be defined.
>>>>>
>>>>> In my opinion, defining a new representation for materialized views
>>>>> (Option 1) is generally the cleaner solution. However, I see a path where
>>>>> we could first introduce Option 3 and still have the possibility to
>>>>> transition to Option 1 if needed. The great thing about Option 3 is that
>>>>> it
>>>>> only requires minor changes to the current spec and is mostly
>>>>> implementation detail.
>>>>>
>>>>> Therefore I would propose small additions to Jacks Option 3 that only
>>>>> introduce changes to the spec that are not specific to materialized views.
>>>>> The idea is to introduce boolean properties to be set on the creation of
>>>>> the view and the storage table that indicate that they belong to a
>>>>> materialized view. The view property "materialized" is set to "true" for a
>>>>> MV and "false" for a regular view. And the table property "storage_table"
>>>>> is set to "true" for a storage table and "false" for a regular table. The
>>>>> absence of these properties indicates a regular view or table.
>>>>>
>>>>> ViewCatalog viewCatalog = (ViewCatalog) catalog;
>>>>>
>>>>> // REST: GET /namespaces/db1/views/mv1
>>>>> // non-REST: load JSON file at metadata_location
>>>>> View mv = viewCatalog.loadView(TableIdentifier.of("db1", "mv1"));
>>>>>
>>>>> // REST: GET /namespaces/db1/tables/mv1
>>>>> // non-REST: load JSON file at table_metadata_location if present
>>>>> Table storageTable = view.storageTable();
>>>>>
>>>>> // REST: POST /namespaces/db1/tables/mv1
>>>>> // non-REST: update JSON file at table_metadata_location
>>>>> storageTable.newAppend().appendFile(...).commit();
>>>>>
>>>>> We could then introduce a new requirement for views and tables called
>>>>> "AssertProperty" which could make sure to only perform updates that are
>>>>> inline with materialized views. The additional requirement can be seen as
>>>>> a
>>>>> general extension which does not need to be changed if we decide to got
>>>>> with Option 1 in the future.
>>>>>
>>>>> Let me know what you think.
>>>>>
>>>>> Best wishes,
>>>>>
>>>>> Jan
>>>>>
>>>>> On 29.02.24 04:09, Walaa Eldin Moustafa wrote:
>>>>>
>>>>> Thanks Ryan for the insights. I agree that reusing existing metadata
>>>>> definitions and minimizing spec changes are very important. This also
>>>>> minimizes spec drift (between materialized views and views spec, and
>>>>> between materialized views and tables spec), and simplifies the
>>>>> implementation.
>>>>>
>>>>> In an effort to take the discussion forward with concrete design
>>>>> options based on an end-to-end implementation, I have prototyped the
>>>>> implementation (and added Spark support) in this PR
>>>>> https://github.com/apache/iceberg/pull/9830. I hope it helps us reach
>>>>> convergence faster. More details about some of the design options are
>>>>> discussed in the description of the PR.
>>>>>
>>>>> Thanks,
>>>>> Walaa.
>>>>>
>>>>>
>>>>> On Wed, Feb 28, 2024 at 6:20 PM Ryan Blue <b...@tabular.io> wrote:
>>>>>
>>>>>> I mean separate table and view metadata that is somehow combined
>>>>>> through a commit process. For instance, keeping a pointer to a table
>>>>>> metadata file in a view metadata file or combining commits to reference
>>>>>> both. I don't see the value in either option.
>>>>>>
>>>>>> On Wed, Feb 28, 2024 at 5:05 PM Jack Ye <yezhao...@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks Ryan for the help to trace back to the root question! Just a
>>>>>>> clarification question regarding your reply before I reply further: what
>>>>>>> exactly does the option "a combination of the two (i.e. commits are
>>>>>>> combined)" mean? How is that different from "a new metadata type"?
>>>>>>>
>>>>>>> -Jack
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 28, 2024 at 2:10 PM Ryan Blue <b...@tabular.io> wrote:
>>>>>>>
>>>>>>>> I’m catching up on this conversation, so hopefully I can bring a
>>>>>>>> fresh perspective.
>>>>>>>>
>>>>>>>> Jack already pointed out that we need to start from the basics and
>>>>>>>> I agree with that. Let’s remove voting at this point. Right now is the
>>>>>>>> time
>>>>>>>> for discussing trade-offs, not lining up and taking sides. I realize
>>>>>>>> that
>>>>>>>> wasn’t the intent with adding a vote, but that’s almost always the
>>>>>>>> result.
>>>>>>>> It’s too easy to use it as a stand-in for consensus and move on
>>>>>>>> prematurely. I get the impression from the swirl in Slack that
>>>>>>>> discussion
>>>>>>>> has moved ahead of agreement.
>>>>>>>>
>>>>>>>> We’re still at the most basic question: is a materialized view a
>>>>>>>> view and a separate table, a combination of the two (i.e. commits are
>>>>>>>> combined), or a new metadata type?
>>>>>>>>
>>>>>>>> For now, I’m ignoring whether the “separate table” is some kind of
>>>>>>>> “system table” (meaning hidden?) or if it is exposed in the catalog.
>>>>>>>> That’s
>>>>>>>> a later choice (already pointed out) and, I suspect, it should be
>>>>>>>> delegated
>>>>>>>> to catalog implementations.
>>>>>>>>
>>>>>>>> To simplify this a little, I think that we can eliminate the option
>>>>>>>> to combine table and view commits. I don’t think there is a reason to
>>>>>>>> combine the two. If separate, a table would track the view version used
>>>>>>>> along with freshness information for referenced tables. If the table is
>>>>>>>> automatically skipped when the version no longer matches the view,
>>>>>>>> then no
>>>>>>>> action needs to happen when a view definition changes. Similarly, the
>>>>>>>> table
>>>>>>>> can be updated independently without needing to also swap view
>>>>>>>> metadata.
>>>>>>>> This also aligns with the idea from the original doc that there can be
>>>>>>>> multiple materialization tables for a view. Each should operate
>>>>>>>> independently unless I’m missing something
>>>>>>>>
>>>>>>>> I don’t think the last paragraph’s conclusion is contentious so
>>>>>>>> I’ll move on, but please stop here and reply if you disagree!
>>>>>>>>
>>>>>>>> That leaves the main two options, a view and a separate table
>>>>>>>> linked by metadata, or, combined materialized view metadata.
>>>>>>>>
>>>>>>>> As the doc notes, the separate view and table option is simpler
>>>>>>>> because it reuses existing metadata definitions and falls back to
>>>>>>>> simple
>>>>>>>> views. That is a significantly smaller spec and small is very, very
>>>>>>>> important when it comes to specs. I think that the argument for a new
>>>>>>>> definition of a materialized view needs to overcome this disadvantage.
>>>>>>>>
>>>>>>>> The arguments that I see for a combined materialized view object
>>>>>>>> are:
>>>>>>>>
>>>>>>>> - Regular views are separate, rather than being tables with SQL
>>>>>>>> and no data so it would be inconsistent (“Iceberg view is just a
>>>>>>>> table with
>>>>>>>> no data but with representations defined. But we did not do that.”)
>>>>>>>> - Materialized views are different objects in DDL
>>>>>>>> - Tables may be a superset of functionality needed for
>>>>>>>> materialized views
>>>>>>>> - Tables are not typically exposed to end users — but this
>>>>>>>> isn’t required by the separate view and table option
>>>>>>>>
>>>>>>>> Am I missing any arguments for combined metadata?
>>>>>>>>
>>>>>>>> Ryan
>>>>>>>> --
>>>>>>>> Ryan Blue
>>>>>>>> Tabular
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Tabular
>>>>>>
>>>>>
>>>
>>> --
>>> Ryan Blue
>>> Tabular
>>>
>>
>
> --
> Ryan Blue
> Tabular
>
