Jack, I think we should consider either allowing the storage table to be fully exposed/addressable via the catalog or allow access via namespacing like with metadata tables. E.g. <catalog>.<database>.<table>.<storage>, which would allow for full access to the underlying table.
For other engines to interact with the storage table (e.g. to execute the query to materialize the table), it may be necessary that the table is fully addressable. Whether the storage table is returned as part of list operations may be something we leave up to the catalog implementation. I don't think the table should reference a physical location (only a logical reference) since things will be changing behind the view definition and I'm not confident we want to have to update the view representation everytime the storage table is updated. I think there's still some exploration as to whether we need to model this as separate from view endpoints, but there may be enough overlap that it's not necessary to have yet another set of endpoints for materialized views (maybe filter params if you need to distinguish?). -Dan On Sun, Feb 18, 2024 at 6:57 PM Renjie Liu <liurenjie2...@gmail.com> wrote: > Hi, Jack: > > Thanks for raising this. > > In most database systems, MV, view and table are considered independent >> objects, at least at API level. It is very rare for a system to support >> operations like "materializing a logical view" or "upgrading a logical view >> to MV", because view and MV are very different in almost every aspect of >> user experience. Extending the existing view or table spec to accommodate >> MV might give us a MV implementation similar to the current Trino or Hive >> views, save us some effort and a few APIs in REST, but it binds us to a >> very specific design of MV, which we might regret in the future. > > > When I reviewed the doc, I thought we were discussing the spec of > materialized view, just like the spec of table metadata, but didn't not the > user facing api. I would definitely agree that we should consider MV as > another kind of database object in user facing api, even though it's > internally modelled as a view + storage table pointer. > > If we want to make the REST experience good for MV, I think we should at >> least consider directly describing the full metadata of the storage table >> in Iceberg view, instead of pointing to a JSON file. > > > Do you mean we need to add components like `LoadMaterializedViewResponse`, > if so, I would +1 for this. > > *Q2: what REST APIs do we expect to use for interactions with MVs?* > > > As I have mentioned above, I think we should consider MV as another > database object, so I think we should add a set of apis specifically > designed for MV, such as `loadMV`, `freshMV`. > > On Sat, Feb 17, 2024 at 11:14 AM Jack Ye <yezhao...@gmail.com> wrote: > >> Hi everyone, >> >> As we are discussing the spec change for materialized view, there has >> been a missing aspect that is technically also related, and might affect >> the MV spec design: *how do we want to add MV support to the REST spec?* >> >> I would like to discuss this in a new thread to collect people's >> thoughts. This topic expands to the following 2 sub-questions: >> >> *Q1: how would the MV spec change affect the REST spec?* >> In the current proposal, it looks like we are using a design where a MV >> is modeled as an Iceberg view linking to an Iceberg storage table. At the >> same time, we do not want to expose this storage table in the catalog, thus >> the Iceberg view has a pointer to only a metadata JSON file of the Iceberg >> storage table. Each MV refresh updates the pointer to a new metadata JSON >> file. >> >> I feel this does not play very well with the direction that REST is >> going. The REST catalog is trying to remove the dependency to the metadata >> JSON file. For example, in LoadTableResponse the only required field is the >> metadata, and metadata-location is actually optional. >> >> If we want to make the REST experience good for MV, I think we should at >> least consider directly describing the full metadata of the storage table >> in Iceberg view, instead of pointing to a JSON file. >> >> *Q2: what REST APIs do we expect to use for interactions with MVs?* >> So far we have been thinking about amending the view spec to accommodate >> MV. This entails likely having MVs also being handled through the view APIs >> in REST spec. >> >> We need to agree with that first in the community, because this has >> various implications, and I am not really sure at this point if it is the >> best way to go. >> >> If MV interactions are through the view APIs, the view APIs need to be >> updated to accommodate MV constructs that are not really related to logical >> views. In fact, most actions performed on MVs are more similar to actions >> performed on table rather than view, which involve configuring data layout, >> read and write constructs. For example, users might run something like: >> >> CREATE MATERIALIZED VIEW mv >> PARTITION BY col1 >> CLUSTER BY col2 >> AS ( // some sql ) >> >> then the CreateView API needs to accept partition spec and sort order >> that are completely not relevant for logical views. >> >> When reading a MV, we might even want to have a PlanMaterializedView API >> similar to the PlanTable API we are adding. >> >> *My personal take* >> It feels like we need to reconsider the question of what is the best way >> to model MV in Iceberg. Should it be (1) a view linked to a storage table, >> or (2) a table with a view SQL associated with it, or (3) it's a completely >> independent thing. This topic was discussed in the past in this doc >> <https://docs.google.com/document/d/1QAuy-meSZ6Oy37iPym8sV_n7R2yKZOHunVR-ZWhhZ6Q/edit?pli=1>, >> but at that time we did not have much perspective about aspects like REST >> spec, and the view integration was also not fully completed yet. With the >> new knowledge, currently I am actually leaning a bit more towards (3). >> >> In most database systems, MV, view and table are considered independent >> objects, at least at API level. It is very rare for a system to support >> operations like "materializing a logical view" or "upgrading a logical view >> to MV", because view and MV are very different in almost every aspect of >> user experience. Extending the existing view or table spec to accommodate >> MV might give us a MV implementation similar to the current Trino or Hive >> views, save us some effort and a few APIs in REST, but it binds us to a >> very specific design of MV, which we might regret in the future. >> >> If we make a new MV spec, it can be made up of fields that already exist >> in the table and view specs, but it is a whole new spec. In this way, the >> spec can evolve independently to accommodate MV specific features, and we >> can also create MV-related REST endpoints that will evolve independently >> from table and view REST APIs. >> >> But on the other side it is definitely associated with more work to >> maintain a new spec, and potentially big refactoring in the codebase to >> make sure operations today that work on table or view can now support MV as >> a different object. And it definitely has other problems that I have >> overlooked. I would greatly appreciate any thoughts about this! >> >> Best, >> Jack Ye >> >>
