Thanks Walaa for driving it forward, looking forward to thinking about implementation of Materialized Views.
I see Jan's point, the PR spec change is similar but does not seem to be completely aligned with the Draft Spec in the design doc: https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/ . I left my comments on PR of those sections with the links to the difference. I think most of those Draft Spec proposal is still applicable after the decision to have separate Table and View objects It will be interesting to at least see drill a bit further why we did not choose the approach in the Draft Spec and chose another way. Thanks Szehon On Wed, May 8, 2024 at 4:48 AM Jan Kaul <jank...@mailbox.org.invalid> wrote: > Well, everybody that actively contributed to the discussion on the > original google doc was in consensus. That's why I brought up the topic at > the Community Sync on the 2024-02-14 (https://youtu.be/uAQVGd5zV4I?t=890) > to raise the awareness of the broader community. After which the discussion > about the storage model started. I don't think that the discussion about a > single aspect of a proposal should invalidate all other aspects of the > proposal. > > Regardless, the state of the proposal from the original google doc > contains a lot of valuable contributions from Micah, Szehon, Jack, Dan, > yourself and others and it should at least provide the basis for any > further discussion. I don't think it's effective to start with a completely > different design because we are bound to have the same discussions all over > again. > > Thanks, Jan > On 08.05.24 12:11, Walaa Eldin Moustafa wrote: > > The only consensus the community had was on the object model through the > most recent voting thread [1]. This kind of consensus was not present > during the doc discussions, and this should be evident from the fact the > last doc state listed 5 alternatives with no particular conclusion. I am > not quite sure what type of consensus we are referring to here given all > the follow up discussions, alternatives, etc. > > Due to the separate object model, the PR is fundamentally different from > the doc in the sense it does not propose a new metadata model but rather > formalizes some new table and view properties related to MVs. That is also > one reason there are no repeated discussions. That said, if you feel there > is a repeated discussion (which I do not see so far), it would be best to > link the relevant discussion from the doc in a comment. > > Happy to move the discussion elsewhere if there is sufficient support for > this idea, but as things stand, I do not see this as an efficient way to > make progress. It sounds we have been re-emphasizing the same points in the > last two replies, so I will let others chime in at this point. > > [1] https://lists.apache.org/thread/rotmqzmwk5jrcsyxhzjhrvcjs5v3yjcc > > Thanks, > Walaa. > > > On Wed, May 8, 2024 at 2:31 AM Jan Kaul <jank...@mailbox.org.invalid> > <jank...@mailbox.org.invalid> wrote: > >> The original google doc >> <https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit?usp=sharing> >> discussed multiple aspects of the Materialized View spec. One was the >> storage model while others were related to the metadata. After we (Micah, >> Szehon, you, me) reached consensus in the google doc, Jack raised his >> concern about the storage model and the long discussion about the storage >> model started. Now we truly reached consensus about the storage model, >> which is now also reflected in the google doc. All other aspects from the >> google doc about the metadata weren't questioned and still represent the >> consensus. >> >> I would like to *avoid repeating the discussions* in your PR that we >> already had in the google doc. Especially since we reached consensus which >> took a considerable amount of time. >> >> Thanks, Jan >> On 08.05.24 10:21, Walaa Eldin Moustafa wrote: >> >> Thanks Jan. I think we moved on to more alignment steps beyond that doc a >> while ago. After that doc, we have discussed the topic further in 2 dev >> list threads and one more doc >> <https://docs.google.com/document/d/1zg0wQ5bVKTckf7-K_cdwF4mlRi6sixLcyEh6jErpGYY/edit?pli=1> >> (with strictly two options for the storage model to consider). Moreover, >> the original doc grew to 14 pages long with one section comparing 5 design >> alternatives, which made things harder to reach consensus. The lack of >> consensus is what partly led up to the subsequent discussions and call for >> a more focused approach to reach consensus. If we already have a consensus >> on the storage model (separate tables and views), I think we should take >> things further and have continued focused discussions on the specific >> metadata in the form of a PR. I have included all previous discussions >> including the original doc and issue as references in the PR description. >> Please let me know if this works. Happy to hear others' thoughts on the >> best way to move forward. >> >> Thanks, >> Walaa. >> >> >> On Wed, May 8, 2024 at 12:56 AM Jan Kaul <jank...@mailbox.org.invalid> >> <jank...@mailbox.org.invalid> wrote: >> >>> Thanks Walaa for trying to move things along. However I don't think it's >>> a good idea to start a separate discussion about the metadata for >>> materialized views because we already had this discussion and reached >>> consensus in this google doc: >>> >>> >>> https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit?usp=sharing >>> >>> Once the draft is finalized we can adopt the PR to reflect the consensus >>> from the google doc. >>> >>> Best wishes, >>> >>> Jan >>> On 07.05.24 19:11, Walaa Eldin Moustafa wrote: >>> >>> Thanks Steven. I feel it is needed so the MV spec is not scattered >>> across the table and view spec pages. We may add a reference in each >>> respective properties section. >>> >>> On Tue, May 7, 2024 at 10:04 AM Steven Wu <stevenz...@gmail.com> wrote: >>> >>>> Walaa, thanks for initiating the next step. >>>> >>>> With the agreed model of separate view and storage table, I am >>>> wondering if a separate materialized view spec page is needed. E.g., the >>>> new view metadata (view-materialized and view-storage-table) is probably >>>> good to be added to the view page directly to avoid information scattering. >>>> The same can be said about the storage table metadata. >>>> >>>> We may keep the separate materialized view page to document motivation, >>>> freshness semantics, etc.. >>>> >>>> On Mon, May 6, 2024 at 10:58 PM Walaa Eldin Moustafa < >>>> wa.moust...@gmail.com> wrote: >>>> >>>>> Hi Everyone, >>>>> >>>>> Thanks again for participating in the modeling discussion [1]. Since >>>>> the outcome of this discussion was to model materialized views as separate >>>>> objects, an Iceberg view and a table, I think the next step should be >>>>> discussing the metadata details for each object. I have created a PR >>>>> https://github.com/apache/iceberg/pull/10280 with an initial spec >>>>> improvement. Please feel free to review it and leave feedback there. >>>>> >>>>> [1] https://lists.apache.org/thread/rotmqzmwk5jrcsyxhzjhrvcjs5v3yjcc >>>>> >>>>> Thanks, >>>>> Walaa. >>>>> >>>>>
