* lineage state JSON structure On Thu, Jun 6, 2024 at 11:31 PM Walaa Eldin Moustafa <wa.moust...@gmail.com> wrote:
> Hi Benny, > > Your understanding is correct. > > Another point that we discussed was the type of APIs engines can use to > conveniently update the storage table with view query results as well as > set the snapshot summary on the output snapshot (one that was produced by > the update). We will follow up on that separately. > > Jan, do you want to reflect the lineage + state discussion in the doc > so we can iterate on the lineage JSON structure? > > Thanks, > Walaa. > > > On Thu, Jun 6, 2024 at 9:40 PM Benny Chow <btc...@gmail.com> wrote: > >> I really enjoyed listening to the replay and hearing everyone's >> feedback! I'm in agreement with all 3 consensus items, especially around >> Dan's idea to separate the view's query tree lineage vs >> materialization's lineage state. >> >> I'll summarize my understanding about the distinction and add a few >> comments: >> >> Materialized View's Query Tree Lineage >> - It's basically the SQL representation converted to a distinct list of >> tables and views. >> - Stored inside view versions so if you change the view SQL, you can >> include the lineage with that change. >> - Tables support time travel so they can optionally include a ref type >> and name/timestamp >> - Views would NOT include the version (that's part of the materialization >> lineage state below) >> - I think we should use fully qualified identifiers here instead of >> UUIDs. Dropping and re-creating a referenced table or view doesn't break >> the view SQL so the lineage should not be broken either. I also don't >> think we can support time travel if we used table UUIDs here. >> - Each table or view can be assigned a unique sequence number. This >> sequence number is scoped to a single view version. >> >> Materialization Lineage State >> - It's basically a lookup table for the above sequence number to either a >> table snapshot id or view version that was used at the time of >> creating/refreshing the storage table. For views, these are nested views >> within the MV's query tree - not the MV itself. >> - Stored inside the table's snapshot summary >> - Additional property "refresh-version-id" to identify the MV's version. >> >> In order to validate the freshness of a materialization, everything above >> has to be checked against the latest tables and views. This should cover >> all data and query tree changes (that I can think of) such as the "limit >> 100" example I gave in Slack >> <https://apache-iceberg.slack.com/archives/C06LPRD60EL/p1717476837288479?thread_ts=1717173133.294819&cid=C06LPRD60EL> >> . >> >> Please let me know your thoughts. >> >> Thanks >> >> On Thu, Jun 6, 2024 at 7:53 AM <russell.spit...@gmail.com> wrote: >> >>> Thanks for hosting it was a very helpful meeting. I really hope we can >>> do more in the future to accelerate consensus on other proposals. >>> >>> >>> I do encourage anyone on the mailing list to add your comments offline >>> as well, especially if you have strong feelings. Iceberg is an open project >>> and we realize not everyone can attend virtual meetings and want you to >>> know you are welcome. >>> >>> >>> >>> On Jun 6, 2024, at 7:11 AM, Jan Kaul <jank...@mailbox.org.invalid> >>> wrote: >>> >>> >>> >>> Hi all, >>> >>> thanks to all of you who attended the meeting yesterday! It was great to >>> talk to you and I think we made great progress. For those of you who >>> weren't able to attend the meeting, I summarized the main points below: >>> >>> * Question 1*: Should we store the "storage table pointer" as a view >>> property or as additional field in the view metadata? >>> >>> We reached consensus to add a *new metadata field* "storage-table" to >>> the view version <https://iceberg.apache.org/view-spec/#versions> >>> record that stores the identifier of the the storage table. The motivation >>> for introducing a new field is that this emphasizes that materialized views >>> are part of the standard and it enforces a common behavior. >>> >>> *Question 2*: Where should the lineage-state information be stored? >>> >>> We reached consensus on storing the lineage-state information in the >>> *snapshot >>> summary* of the storage table. The motivation behind this is that the >>> table spec should not be concerned with defining view constructs. >>> >>> *Question 3*: How should the lineage-state information be represented? >>> >>> We reached consensus on representing the lineage-state in the form of >>> nested objects and storing these as a *JSON-encoded string* inside the >>> storage table snapshot summary. >>> >>> Additionally, Dan proposed to introduce a new lineage construct as part >>> of the view definition in addition to the lineage-state that is part of the >>> storage table. The idea is to separate the concerns. The lineage-state in >>> the storage table should only capture the state of the source tables at the >>> time of the last refresh, whereas the lineage information in the view >>> contains more information about the source tables and is responsible for >>> resolving the identifiers. We haven't really decided on how the new lineage >>> construct should be represented or integrated into the view metadata. >>> >>> One point that we didn't really have the time to discuss was Benny's >>> comment of also storing the version-id of views in the case that the >>> materialized view is referencing a view. I think we should also integrate >>> that into the spec. >>> >>> You can find the recording of the meeting here: >>> >>> >>> https://drive.google.com/file/d/1DE09tYS28L3xL_NgnM9g0Olbe6aHza5G/view?usp=sharing >>> >>> Best wishes, >>> >>> Jan >>> >>>
