Thanks everyone for the help in organizing the thoughts! I have moved the summary of everyone's comments here also to the doc that Jan linked under question 0. We can continue to have more discussions there and cast votes!
Best, Jack Ye On Wed, Feb 21, 2024 at 12:14 PM Jan Kaul <jank...@mailbox.org.invalid> wrote: > Thanks Micah, I think the voting chips are great. > > @Szehon, actually what I had in mind was not to have one thread per > question but rather have smaller threads that can be resolved more easily. > I have the fear that one thread for the current question would lead to a > very long and unmanageable discussion. > > I've added another row to the table where everyone could provide a summary > of their reason for choosing a certain design. This way we could move some > of the content from the comment threads to the main document. > On 21.02.24 19:58, Micah Kornfield wrote: > > Of course we also need threads that express our preferences (voting). I >> would suggest to keep these separate from discussions about single points >> so that they can be persisted in the document. > > > Not sure if it helpful, but I added voting chips Question 0, as maybe an > easier way to keep track of votes. If it is helpful, I can add them in > other places that still need a vote (I think one needs a paid Google Docs > account to insert them). > > Thanks, > Micah > > On Wed, Feb 21, 2024 at 10:23 AM Szehon Ho <szehon.apa...@gmail.com> > wrote: > >> Thanks Jan. +1 on having just one thread per question for >> vote/preference. Where do you suggest we have it, on the discussion >> question itself? It would be to keep the existing threads and move it >> there. >> >> Also, I think it makes sense with making a slack channel (for quick >> question, reply) , and also discuss unresolved questions in the next week's >> sync or a separate meeting. >> >> On Wed, Feb 21, 2024 at 12:40 AM Jan Kaul <jank...@mailbox.org.invalid> >> <jank...@mailbox.org.invalid> wrote: >> >>> Thank you Jack for driving the consensus for the MV spec and thank you >>> all for the discussion. >>> >>> I really like the idea about incremental consensus because we often >>> loose sight in detailed discussions. As Jack mentioned, the highest >>> priority question currently is: >>> >>> *Should the Iceberg MV be realized as a view + storage table or do we >>> define a new metadata format? *To have one place for the discussion, I >>> created another Question ( >>> https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit?pli=1#heading=h.y70rtfhi9qxi) >>> to the Materialized View Spec google document. >>> >>> To improve the visibility of the arguments I would like to propose a new >>> process. It would be great if all relevant information is stored in the >>> document itself. Therefore I would suggest to use the comment threads for >>> smaller, temporary discussions which can be resolved by adding the points >>> to the main document. Please close the threads if the information was added >>> to the document. Additionally, I gave you all permissions to edit the >>> documents, so you can add missing points yourselves. >>> >>> Of course we also need threads that express our preferences (voting). I >>> would suggest to keep these separate from discussions about single points >>> so that they can be persisted in the document. >>> >>> After a phase of collecting arguments for the different designs I think >>> it would make sense to have video call to have a face to face discussion. >>> >>> What do you think? >>> >>> Best wishes, >>> >>> Jan >>> On 20.02.24 21:32, Manish Malhotra wrote: >>> >>> Very excited for MV to be in Iceberg :) >>> Keeping in the same doc. would be helpful, to have the trail. >>> But also agreed, if there are too many directions/threads, then keep >>> closing the old one, if there are no more questions. >>> And put down the assumptions for the initial version to move forward. >>> >>> >>> On Tue, Feb 20, 2024 at 12:17 PM Walaa Eldin Moustafa < >>> wa.moust...@gmail.com> wrote: >>> >>>> I would vote to keep a log in the doc with open questions, and keep the >>>> doc updated with open questions as they arise/get resolved. >>>> >>>> On Tue, Feb 20, 2024 at 11:37 AM Jack Ye <yezhao...@gmail.com> wrote: >>>> >>>>> Thanks for the response from everyone! >>>>> >>>>> Before proceeding further, I see a few people referring back to the >>>>> current design from Jan. I specifically raised this thread based on the >>>>> information in the doc and a few latest discussions we had there. Because >>>>> there are many threads in the doc, and each thread points further to other >>>>> discussion threads in the same doc or other doc, it is now quite hard to >>>>> follow and continue discussing all different topics there. >>>>> >>>>> I hope we can make incremental consensus of the questions in the doc >>>>> through devlist, because it provides more visibility, and also a single >>>>> thread instead of multiple threads going on at the same time. If we think >>>>> this format is not effective, I propose that we create a new mv channel in >>>>> Iceberg Slack workspace, and people interested can join and discuss all >>>>> these points directly. What do we think? >>>>> >>>>> Best, >>>>> Jack Ye >>>>> >>>>> >>>>> >>>>> On Mon, Feb 19, 2024 at 6:03 PM Szehon Ho <szehon.apa...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Great to see more discussion on the MV spec. Actually, Jan's >>>>>> document "Iceberg Materialized View Spec" >>>>>> <https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A> >>>>>> has >>>>>> been organized , with a "Design Questions" section to track these >>>>>> debates, >>>>>> and it would be nice to centralize the debates there, as Micah mentions. >>>>>> >>>>>> For Dan's question, I think this debate was tracked in "Design Question >>>>>> 3: Should the storage table be registered in the catalog?". I think the >>>>>> general idea there was to not expose it directly via Catalog as it is >>>>>> then >>>>>> exposed to user modification. If the engine wants to access anything >>>>>> about >>>>>> the storage table (including audit and storage), it is of course there >>>>>> via >>>>>> the storage table pointer. I think Walaa's point is also good, we could >>>>>> expose it as we expose metadata tables, but I am still not sure if there >>>>>> is >>>>>> still some use-cases of engine access not covered? >>>>>> >>>>>> It is true that for Jack's initial question (Do we really want to go >>>>>> with the MV = view + storage table design approach for Iceberg MV?), >>>>>> unfortunately we did not capture it as a "Design Question" in Jan's doc, >>>>>> as >>>>>> it was an implicit assumption of 'yes', because it is the choice of Hive, >>>>>> Trino, and other engines , as others have pointed out. >>>>>> >>>>>> Jack's point about potential evolution of MV (like to add >>>>>> partitioning) is an interesting one, but definitely hard to grasp. I >>>>>> think >>>>>> it makes sense to add this as a separate Design Question in the doc, and >>>>>> add the options. This will allow us to flesh out this alternative >>>>>> option(s). Maybe Micah's point about modifying existing proposal to >>>>>> 'embed' the required table metadata fields in the existing view metadata, >>>>>> is one middle ground option. Or we add a totally new MV object spec for >>>>>> MV, separate than existing View spec? >>>>>> >>>>>> Also , as Jack pointed out, it may make sense to have the REST / >>>>>> Catalog API proposal in the doc to educate the above decision. >>>>>> >>>>>> Thanks >>>>>> Szehon >>>>>> >>>>>> On Mon, Feb 19, 2024 at 4:08 PM Walaa Eldin Moustafa < >>>>>> wa.moust...@gmail.com> wrote: >>>>>> >>>>>>> I think it would help if we answer the question of whether an MV is >>>>>>> a view + storage table (and degree of exposing this underlying >>>>>>> implementation) in the context of the user interfacing with those >>>>>>> concepts: >>>>>>> >>>>>>> For the end user, interfacing with the engine APIs (e.g., through >>>>>>> SQL), materialized view APIs should be almost the same as regular view >>>>>>> APIs >>>>>>> (except for operations specific to materialized views like REFRESH >>>>>>> command >>>>>>> etc). Typically, the end user interacts with the (materialized) view >>>>>>> object >>>>>>> as a view, and the engine performs the abstraction over the storage >>>>>>> table. >>>>>>> >>>>>>> For the engines interfacing with Iceberg, it sounds the correct >>>>>>> abstraction at this layer is indeed view + storage table, and engines >>>>>>> could >>>>>>> have access to both objects to optimize queries. >>>>>>> >>>>>>> So in a sense, the engine will ultimately hide most of the >>>>>>> storage detail from the end user (except for advanced users who want to >>>>>>> explicitly access the storage table with a modifier like >>>>>>> "db.view.storageTable" -- and they can only read it), while Iceberg will >>>>>>> expose the storage details to the engine catalog to use it in scans if >>>>>>> needed. So the storage table is hidden or exposed based on the >>>>>>> context/the >>>>>>> actual users. From Iceberg point of view (which interacts with the >>>>>>> engines), the storage table is exposed. Note that this does not >>>>>>> necessarily mean that the storage table is registered in the catalog >>>>>>> with >>>>>>> its own independent name (e.g., where we can drop the view but keep the >>>>>>> storage table and access it from the catalog). Addressing the storage >>>>>>> table >>>>>>> using a virtual namespace like "db.view.storageTable" sounds like a good >>>>>>> middle ground. Anyways, end users should not need to directly access the >>>>>>> storage table in most cases. >>>>>>> >>>>>>> Thanks, >>>>>>> Walaa. >>>>>>> >>>>>>> On Mon, Feb 19, 2024 at 3:38 PM Micah Kornfield < >>>>>>> emkornfi...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Jack, >>>>>>>> >>>>>>>> >>>>>>>>> In my mind, the first key point we all need to agree upon to move >>>>>>>>> this design forward is*: Do we really want to go with the MV = >>>>>>>>> view + storage table design approach for Iceberg MV?* >>>>>>>> >>>>>>>> >>>>>>>> I think we want this to the extent that we do not want to redefine >>>>>>>> the same concept with different representations/naming to the greatest >>>>>>>> degree possible. This is why borrowing the concepts from the view >>>>>>>> (e.g. >>>>>>>> multiple ways of expressing the same view logic in different dialects) >>>>>>>> and >>>>>>>> aspects of the materialized data (e.g. partitioning, ordering) feels >>>>>>>> most >>>>>>>> natural. IIUC your proposal, I think you are saying maybe two >>>>>>>> modifications to the existing proposals in the document: >>>>>>>> >>>>>>>> 1. No separate storage table link, instead embed most of the >>>>>>>> metadata of the materialized table into the MV document (the exception >>>>>>>> seems to be snapshot history) >>>>>>>> 2. For snapshot history, have one unified history specific to the >>>>>>>> MV. >>>>>>>> >>>>>>>> This seems fairly reasonable to me and I think I can solve some >>>>>>>> challenges with the existing proposal in an elegant way. If this is >>>>>>>> correct (or maybe if it isn't quite correct) perhaps you can make >>>>>>>> suggestions to the document so all of the trade-offs can be discussed >>>>>>>> in >>>>>>>> one place? >>>>>>>> >>>>>>>> I think the one thing the current draft of the materialized view >>>>>>>> ignores is how to store algebraic summaries (e.g. separate sum and >>>>>>>> count >>>>>>>> for averages, or other sketches), so that new data can be incrementally >>>>>>>> incorporated. But representing these structures feels like it >>>>>>>> potentially >>>>>>>> has value beyond just MVs (e.g. it can be a natural way to express >>>>>>>> summary >>>>>>>> statistics in table metadata), so I think it deserves at least a try in >>>>>>>> incorporating the concepts in the table specification, so the >>>>>>>> definitions >>>>>>>> can be shared. I was imagining this could come as part of the next >>>>>>>> revision of MV specification. >>>>>>>> >>>>>>>> The MV internal structure could evolve in a way that works more >>>>>>>>> efficiently with the reduced scope of functionalities, without >>>>>>>>> relying on >>>>>>>>> table to offer the same capabilities. I can at least say that is true >>>>>>>>> based >>>>>>>>> on my internal knowledge of how Redshift MVs work. >>>>>>>> >>>>>>>> >>>>>>>> I'm not sure I fully understand this point, but it seems the main >>>>>>>> question here is what would break if it started to evolve in this >>>>>>>> direction. Is it purely additive or do we suspect some elements would >>>>>>>> need >>>>>>>> to be removed? My gut feeling here is the main concerns here are >>>>>>>> getting >>>>>>>> the cardinatities correct (i.e. 1 MV should probably have 0, 1 or more >>>>>>>> materialized storage tables associated with it, to support more >>>>>>>> advanced >>>>>>>> algebraic structures listed above, and perhaps a second without them, >>>>>>>> and >>>>>>>> additional metadata to distinguish between these two different modes). >>>>>>>> >>>>>>>> If after the evaluation, we are confident that the MV = view + >>>>>>>>> storage table approach is the right way to go, then we can debate the >>>>>>>>> other >>>>>>>>> issues, and I think the next issue to reach consensus should be >>>>>>>>> "Should the >>>>>>>>> storage table be registered in the catalog?". >>>>>>>> >>>>>>>> >>>>>>>> I actually think there are actually more fundamental questions >>>>>>>> posed: >>>>>>>> 1. Should be considering how items should be modelled in the REST >>>>>>>> API concurrently with the Iceberg spec, as that potentially impacts >>>>>>>> design >>>>>>>> decision (I think the answer is yes, and we should update the doc with >>>>>>>> sketches on new endpoints and operations on the endpoints to ensure >>>>>>>> things >>>>>>>> align). >>>>>>>> 2. Going forward should new aspects of Iceberg artifacts rely on >>>>>>>> the fact that a catalog is present and we can rely on a naming >>>>>>>> convention >>>>>>>> for looking up other artifacts in a catalog as pointers (I lean yes on >>>>>>>> this, but I'm a little bit more ambivalent). >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Micah >>>>>>>> >>>>>>>> On Mon, Feb 19, 2024 at 12:52 PM Jack Ye <yezhao...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I suggest we need a step-by-step process to make incremental >>>>>>>>> consensus, otherwise we are constantly talking about many different >>>>>>>>> debates >>>>>>>>> at the same time. >>>>>>>>> >>>>>>>>> In my mind, the first key point we all need to agree upon to move >>>>>>>>> this design forward is*: Do we really want to go with the MV = >>>>>>>>> view + storage table design approach for Iceberg MV?* >>>>>>>>> >>>>>>>>> I think we (at least me) started with this assumption, mostly >>>>>>>>> because this is how Trino implements MV, and how Hive tables store MV >>>>>>>>> information today. But does it mean we should design it that way in >>>>>>>>> Iceberg? >>>>>>>>> >>>>>>>>> Now I look back at how we did the view spec design, we could also >>>>>>>>> say that we just add a representation field in the table spec to store >>>>>>>>> view, and an Iceberg view is just a table with no data but with >>>>>>>>> representations defined. But we did not do that. So it feels now quite >>>>>>>>> inconsistent to say we want to just add a few fields in the table and >>>>>>>>> view >>>>>>>>> spec to call it an Iceberg MV. >>>>>>>>> >>>>>>>>> If we look into most of the other database systems (e.g. Redshift, >>>>>>>>> BigQuery, Snowflake), they never expose such implementation details >>>>>>>>> like >>>>>>>>> storage table. Apart from being close-sourced systems, I think it is >>>>>>>>> also >>>>>>>>> for good technical reasons. There are many more things that a table >>>>>>>>> needs >>>>>>>>> to support, but does not really apply to MV. The MV internal structure >>>>>>>>> could evolve in a way that works more efficiently with the reduced >>>>>>>>> scope of >>>>>>>>> functionalities, without relying on table to offer the same >>>>>>>>> capabilities. I >>>>>>>>> can at least say that is true based on my internal knowledge of how >>>>>>>>> Redshift MVs work. >>>>>>>>> >>>>>>>>> I think we should fully evaluate both directions, and commit to >>>>>>>>> one first before debating more things. >>>>>>>>> >>>>>>>>> If we have a new and independent Iceberg MV spec, then an Iceberg >>>>>>>>> MV is under-the-hood a single object containing all MV information. >>>>>>>>> It has >>>>>>>>> its own name, snapshots, view representation, etc. I don't believe we >>>>>>>>> will >>>>>>>>> be blocked by Trino due to its MV SPIs currently requiring the >>>>>>>>> existence of >>>>>>>>> a storage table, as it will just be a different implementation from >>>>>>>>> the >>>>>>>>> existing one in Trino-Iceberg. In this direction, I don't think we >>>>>>>>> need to >>>>>>>>> have any further debate about pointers, metadata locations, storage >>>>>>>>> table, >>>>>>>>> etc. because everything will be new. >>>>>>>>> >>>>>>>>> If after the evaluation, we are confident that the MV = view + >>>>>>>>> storage table approach is the right way to go, then we can debate the >>>>>>>>> other >>>>>>>>> issues, and I think the next issue to reach consensus should be >>>>>>>>> "Should the >>>>>>>>> storage table be registered in the catalog?". >>>>>>>>> >>>>>>>>> What do we think? >>>>>>>>> >>>>>>>>> -Jack >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Feb 19, 2024 at 11:32 AM Daniel Weeks <dwe...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Jack, >>>>>>>>>> >>>>>>>>>> I think we should consider either allowing the storage table to >>>>>>>>>> be fully exposed/addressable via the catalog or allow access via >>>>>>>>>> namespacing like with metadata tables. E.g. >>>>>>>>>> <catalog>.<database>.<table>.<storage>, which would allow for full >>>>>>>>>> access >>>>>>>>>> to the underlying table. >>>>>>>>>> >>>>>>>>>> For other engines to interact with the storage table (e.g. to >>>>>>>>>> execute the query to materialize the table), it may be necessary >>>>>>>>>> that the >>>>>>>>>> table is fully addressable. Whether the storage table is returned >>>>>>>>>> as part >>>>>>>>>> of list operations may be something we leave up to the catalog >>>>>>>>>> implementation. >>>>>>>>>> >>>>>>>>>> I don't think the table should reference a physical location >>>>>>>>>> (only a logical reference) since things will be changing behind the >>>>>>>>>> view >>>>>>>>>> definition and I'm not confident we want to have to update the view >>>>>>>>>> representation everytime the storage table is updated. >>>>>>>>>> >>>>>>>>>> I think there's still some exploration as to whether we need to >>>>>>>>>> model this as separate from view endpoints, but there may be enough >>>>>>>>>> overlap >>>>>>>>>> that it's not necessary to have yet another set of endpoints for >>>>>>>>>> materialized views (maybe filter params if you need to distinguish?). >>>>>>>>>> >>>>>>>>>> -Dan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sun, Feb 18, 2024 at 6:57 PM Renjie Liu < >>>>>>>>>> liurenjie2...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, Jack: >>>>>>>>>>> >>>>>>>>>>> Thanks for raising this. >>>>>>>>>>> >>>>>>>>>>> In most database systems, MV, view and table are considered >>>>>>>>>>>> independent objects, at least at API level. It is very rare for a >>>>>>>>>>>> system to >>>>>>>>>>>> support operations like "materializing a logical view" or >>>>>>>>>>>> "upgrading a >>>>>>>>>>>> logical view to MV", because view and MV are very different in >>>>>>>>>>>> almost every >>>>>>>>>>>> aspect of user experience. Extending the existing view or table >>>>>>>>>>>> spec to >>>>>>>>>>>> accommodate MV might give us a MV implementation similar to the >>>>>>>>>>>> current >>>>>>>>>>>> Trino or Hive views, save us some effort and a few APIs in REST, >>>>>>>>>>>> but it >>>>>>>>>>>> binds us to a very specific design of MV, which we might regret in >>>>>>>>>>>> the >>>>>>>>>>>> future. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> When I reviewed the doc, I thought we were discussing the spec >>>>>>>>>>> of materialized view, just like the spec of table metadata, but >>>>>>>>>>> didn't not >>>>>>>>>>> the user facing api. I would definitely agree that we should >>>>>>>>>>> consider MV as >>>>>>>>>>> another kind of database object in user facing api, even though it's >>>>>>>>>>> internally modelled as a view + storage table pointer. >>>>>>>>>>> >>>>>>>>>>> If we want to make the REST experience good for MV, I think we >>>>>>>>>>>> should at least consider directly describing the full metadata of >>>>>>>>>>>> the >>>>>>>>>>>> storage table in Iceberg view, instead of pointing to a JSON file. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Do you mean we need to add components like >>>>>>>>>>> `LoadMaterializedViewResponse`, if so, I would +1 for this. >>>>>>>>>>> >>>>>>>>>>> *Q2: what REST APIs do we expect to use for interactions with >>>>>>>>>>>> MVs?* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> As I have mentioned above, I think we should consider MV as >>>>>>>>>>> another database object, so I think we should add a set of apis >>>>>>>>>>> specifically designed for MV, such as `loadMV`, `freshMV`. >>>>>>>>>>> >>>>>>>>>>> On Sat, Feb 17, 2024 at 11:14 AM Jack Ye <yezhao...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi everyone, >>>>>>>>>>>> >>>>>>>>>>>> As we are discussing the spec change for materialized view, >>>>>>>>>>>> there has been a missing aspect that is technically also related, >>>>>>>>>>>> and might >>>>>>>>>>>> affect the MV spec design: *how do we want to add MV support >>>>>>>>>>>> to the REST spec?* >>>>>>>>>>>> >>>>>>>>>>>> I would like to discuss this in a new thread to collect >>>>>>>>>>>> people's thoughts. This topic expands to the following 2 >>>>>>>>>>>> sub-questions: >>>>>>>>>>>> >>>>>>>>>>>> *Q1: how would the MV spec change affect the REST spec?* >>>>>>>>>>>> In the current proposal, it looks like we are using a design >>>>>>>>>>>> where a MV is modeled as an Iceberg view linking to an Iceberg >>>>>>>>>>>> storage >>>>>>>>>>>> table. At the same time, we do not want to expose this storage >>>>>>>>>>>> table in the >>>>>>>>>>>> catalog, thus the Iceberg view has a pointer to only a metadata >>>>>>>>>>>> JSON file >>>>>>>>>>>> of the Iceberg storage table. Each MV refresh updates the pointer >>>>>>>>>>>> to a new >>>>>>>>>>>> metadata JSON file. >>>>>>>>>>>> >>>>>>>>>>>> I feel this does not play very well with the direction that >>>>>>>>>>>> REST is going. The REST catalog is trying to remove the dependency >>>>>>>>>>>> to the >>>>>>>>>>>> metadata JSON file. For example, in LoadTableResponse the only >>>>>>>>>>>> required >>>>>>>>>>>> field is the metadata, and metadata-location is actually optional. >>>>>>>>>>>> >>>>>>>>>>>> If we want to make the REST experience good for MV, I think we >>>>>>>>>>>> should at least consider directly describing the full metadata of >>>>>>>>>>>> the >>>>>>>>>>>> storage table in Iceberg view, instead of pointing to a JSON file. >>>>>>>>>>>> >>>>>>>>>>>> *Q2: what REST APIs do we expect to use for interactions with >>>>>>>>>>>> MVs?* >>>>>>>>>>>> So far we have been thinking about amending the view spec to >>>>>>>>>>>> accommodate MV. This entails likely having MVs also being handled >>>>>>>>>>>> through >>>>>>>>>>>> the view APIs in REST spec. >>>>>>>>>>>> >>>>>>>>>>>> We need to agree with that first in the community, because this >>>>>>>>>>>> has various implications, and I am not really sure at this point >>>>>>>>>>>> if it is >>>>>>>>>>>> the best way to go. >>>>>>>>>>>> >>>>>>>>>>>> If MV interactions are through the view APIs, the view APIs >>>>>>>>>>>> need to be updated to accommodate MV constructs that are not >>>>>>>>>>>> really related >>>>>>>>>>>> to logical views. In fact, most actions performed on MVs are more >>>>>>>>>>>> similar >>>>>>>>>>>> to actions performed on table rather than view, which involve >>>>>>>>>>>> configuring >>>>>>>>>>>> data layout, read and write constructs. For example, users might >>>>>>>>>>>> run >>>>>>>>>>>> something like: >>>>>>>>>>>> >>>>>>>>>>>> CREATE MATERIALIZED VIEW mv >>>>>>>>>>>> PARTITION BY col1 >>>>>>>>>>>> CLUSTER BY col2 >>>>>>>>>>>> AS ( // some sql ) >>>>>>>>>>>> >>>>>>>>>>>> then the CreateView API needs to accept partition spec and sort >>>>>>>>>>>> order that are completely not relevant for logical views. >>>>>>>>>>>> >>>>>>>>>>>> When reading a MV, we might even want to have a >>>>>>>>>>>> PlanMaterializedView API similar to the PlanTable API we are >>>>>>>>>>>> adding. >>>>>>>>>>>> >>>>>>>>>>>> *My personal take* >>>>>>>>>>>> It feels like we need to reconsider the question of what is the >>>>>>>>>>>> best way to model MV in Iceberg. Should it be (1) a view linked to >>>>>>>>>>>> a >>>>>>>>>>>> storage table, or (2) a table with a view SQL associated with it, >>>>>>>>>>>> or (3) >>>>>>>>>>>> it's a completely independent thing. This topic was discussed in >>>>>>>>>>>> the past in >>>>>>>>>>>> this doc >>>>>>>>>>>> <https://docs.google.com/document/d/1QAuy-meSZ6Oy37iPym8sV_n7R2yKZOHunVR-ZWhhZ6Q/edit?pli=1>, >>>>>>>>>>>> but at that time we did not have much perspective about aspects >>>>>>>>>>>> like REST >>>>>>>>>>>> spec, and the view integration was also not fully completed yet. >>>>>>>>>>>> With the >>>>>>>>>>>> new knowledge, currently I am actually leaning a bit more towards >>>>>>>>>>>> (3). >>>>>>>>>>>> >>>>>>>>>>>> In most database systems, MV, view and table are considered >>>>>>>>>>>> independent objects, at least at API level. It is very rare for a >>>>>>>>>>>> system to >>>>>>>>>>>> support operations like "materializing a logical view" or >>>>>>>>>>>> "upgrading a >>>>>>>>>>>> logical view to MV", because view and MV are very different in >>>>>>>>>>>> almost every >>>>>>>>>>>> aspect of user experience. Extending the existing view or table >>>>>>>>>>>> spec to >>>>>>>>>>>> accommodate MV might give us a MV implementation similar to the >>>>>>>>>>>> current >>>>>>>>>>>> Trino or Hive views, save us some effort and a few APIs in REST, >>>>>>>>>>>> but it >>>>>>>>>>>> binds us to a very specific design of MV, which we might regret in >>>>>>>>>>>> the >>>>>>>>>>>> future. >>>>>>>>>>>> >>>>>>>>>>>> If we make a new MV spec, it can be made up of fields that >>>>>>>>>>>> already exist in the table and view specs, but it is a whole new >>>>>>>>>>>> spec. In >>>>>>>>>>>> this way, the spec can evolve independently to accommodate MV >>>>>>>>>>>> specific >>>>>>>>>>>> features, and we can also create MV-related REST endpoints that >>>>>>>>>>>> will evolve >>>>>>>>>>>> independently from table and view REST APIs. >>>>>>>>>>>> >>>>>>>>>>>> But on the other side it is definitely associated with more >>>>>>>>>>>> work to maintain a new spec, and potentially big refactoring in the >>>>>>>>>>>> codebase to make sure operations today that work on table or view >>>>>>>>>>>> can now >>>>>>>>>>>> support MV as a different object. And it definitely has other >>>>>>>>>>>> problems that >>>>>>>>>>>> I have overlooked. I would greatly appreciate any thoughts about >>>>>>>>>>>> this! >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Jack Ye >>>>>>>>>>>> >>>>>>>>>>>>
