Hi all,
thanks to all of you who attended the meeting yesterday! It was great to
talk to you and I think we made great progress. For those of you who
weren't able to attend the meeting, I summarized the main points below:
*
Question 1*: Should we store the "storage table pointer" as a view
property or as additional field in the view metadata?
We reached consensus to add a *new metadata field* "storage-table" to
the view version <https://iceberg.apache.org/view-spec/#versions> record
that stores the identifier of the the storage table. The motivation for
introducing a new field is that this emphasizes that materialized views
are part of the standard and it enforces a common behavior.
*Question 2*: Where should the lineage-state information be stored?
We reached consensus on storing the lineage-state information in the
*snapshot summary* of the storage table. The motivation behind this is
that the table spec should not be concerned with defining view constructs.
*Question 3*: How should the lineage-state information be represented?
We reached consensus on representing the lineage-state in the form of
nested objects and storing these as a *JSON-encoded string* inside the
storage table snapshot summary.
Additionally, Dan proposed to introduce a new lineage construct as part
of the view definition in addition to the lineage-state that is part of
the storage table. The idea is to separate the concerns. The
lineage-state in the storage table should only capture the state of the
source tables at the time of the last refresh, whereas the lineage
information in the view contains more information about the source
tables and is responsible for resolving the identifiers. We haven't
really decided on how the new lineage construct should be represented or
integrated into the view metadata.
One point that we didn't really have the time to discuss was Benny's
comment of also storing the version-id of views in the case that the
materialized view is referencing a view. I think we should also
integrate that into the spec.
You can find the recording of the meeting here:
https://drive.google.com/file/d/1DE09tYS28L3xL_NgnM9g0Olbe6aHza5G/view?usp=sharing
<https://drive.google.com/file/d/1DE09tYS28L3xL_NgnM9g0Olbe6aHza5G/view?usp=sharing>
Best wishes,
Jan
