> I think that's the lesser evil compared to Iceberg specifying how engines should resolve identifiers
I think this is also similar to the previous point. It is the other way around. Right now the spec dictates how to resolve (through employing a view-specific `default-catalog` field). The proposal is suggesting to get out of this space and let engines handle it similar to how they handle all identifiers. On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa <wa.moust...@gmail.com> wrote: > > I thought "default-catalog" could be set via the USE command. > > Benny, I think this is a misconception or miscommunication. The USE > command has no impact on the `default-catalog` field. In fact, the > proposal's direction is exactly to establish that USE command should > influence how tables are resolved, same like everywhere else. Right now it > is not the case under the current spec. > > > On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <btc...@gmail.com> wrote: > >> > there is no SQL construct today to explicitly set default-catalog >> >> I thought "default-catalog" could be set via the USE command. >> >> I generally agree with Dan about requiring consistent catalog names. I >> think that's the lesser evil compared to Iceberg specifying how engines >> should resolve identifiers. Another thing to consider is that identifier >> resolution can be very expensive at query validation time if identifiers >> need to be looked up from a bunch of places. Hopefully, it should be >> possible to define a view in such a way that identifiers can be resolved on >> the first try. >> >> Benny >> >> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa < >> wa.moust...@gmail.com> wrote: >> >>> Hi Rishabh, >>> >>> You're right that the proposal touches on two aspects, and resolution >>> rules are one of them. The other aspect is the proposal's position that >>> table identifiers should be stored in metadata exactly as they appear in >>> the view text (e.g., even if they're two-part or partially qualified), >>> along with their corresponding UUIDs for validation. This applies both to >>> referenced input tables and the storage table identifier in materialized >>> views. >>> >>> We may be able to converge on this storage format even if we haven't yet >>> converged on the resolution fallback rules. I believe both resolution >>> strategies currently being discussed would still lead to storing >>> identifiers in this way. >>> >>> I'm supportive of moving forward with consensus on the identifier >>> storage format. That said, we may continue to run into questions related to >>> resolution during implementation. For example: Should the storage table >>> identifier follow the same default-catalog and default-namespace resolution >>> behavior as other table references? >>> >>> Thanks, >>> Walaa. >>> >>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia < >>> bhatiarishab...@gmail.com> wrote: >>> >>>> Hello Walaa, >>>> >>>> Thanks for starting this discussion. >>>> >>>> I think we should decouple at least the MV Spec from the proposal to >>>> change the current behavior of view resolution. >>>> >>>> We can continue having the discussion if the current view spec needs to >>>> be changed or not. Based on the decision at a later point if required we >>>> can update the view resolution rule. >>>> >>>> >>>> Thanks, >>>> Rishabh >>>> >>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa < >>>> wa.moust...@gmail.com> wrote: >>>> >>>>> Correction of typo: both engines seem to set default-catalog to the >>>>> view catalog if it is defined, or to null if the view catalog is not >>>>> defined. >>>>> >>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa < >>>>> wa.moust...@gmail.com> wrote: >>>>> >>>>>> Hi Dan, >>>>>> >>>>>> Thanks again for your response. >>>>>> >>>>>> I agree that catalog renaming is an environmental event, but it's a >>>>>> real one that happens frequently in practice. >>>>>> Saying that the Iceberg spec cannot accommodate something as common >>>>>> as catalog renaming feels very restrictive, and could make the spec less >>>>>> practical, even unusable, for real-world deployments. >>>>>> I’m sharing this from the perspective of a large data lake >>>>>> environment where views are heavily deployed and operationalized. >>>>>> >>>>>> Further, it's worth noting that the table spec is resilient to >>>>>> catalog renaming, but the view spec is not. If we have an opportunity to >>>>>> make the view spec similarly resilient, I wonder why not? >>>>>> Both specifications are deterministic in their definition, but one is >>>>>> more fragile to environmental changes than the other. Improving >>>>>> resilience >>>>>> does not sacrifice determinism. It simply makes views safer and more >>>>>> portable over time. >>>>>> >>>>>> Separately, given that there is no SQL construct today to explicitly >>>>>> set default-catalog at creation time, what is the intuition behind how >>>>>> engines like Spark and Trino currently assign default-catalog? >>>>>> Today, both engines seem to set default-catalog to null if the view >>>>>> catalog is defined, or to the view catalog if not. >>>>>> What was the intended thought process behind this behavior? >>>>>> >>>>>> Thanks, >>>>>> Walaa >>>>>> >>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <dwe...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> Walaa, >>>>>>> >>>>>>> > tables inside views remain reachable after a catalog rename >>>>>>> >>>>>>> This problem stems from the exact environmental/configuration issue >>>>>>> that we should not be trying to address. I don't think we would expect >>>>>>> references to survive a catalog rename. That's not something covered by >>>>>>> the spec and needs to be handled separately as a platform-level >>>>>>> migration >>>>>>> specific to the affected environment. >>>>>>> >>>>>>> The identifier resolution logic is clear and deterministic. It >>>>>>> should not matter whether an engine resolves and encodes the >>>>>>> default-catalog or leaves it to the resolution rules. >>>>>>> >>>>>>> The issue isn't with how the spec is defined, but rather view >>>>>>> behavior when you start altering the environment around it, which isn't >>>>>>> something we should be trying to define here. >>>>>>> >>>>>>> -Dan >>>>>>> >>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa < >>>>>>> wa.moust...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Dan, >>>>>>>> >>>>>>>> Thanks for chiming in. >>>>>>>> >>>>>>>> I believe the issues we’re seeing now go beyond just catalog naming >>>>>>>> consistency. The behavior around default-catalog itself introduces >>>>>>>> resolution inconsistencies even when catalog names are consistent. >>>>>>>> For example: >>>>>>>> >>>>>>>> * When default-catalog is set to null, tables inside views remain >>>>>>>> reachable after a catalog rename. But if it is set to a non-null value, >>>>>>>> table references will break. >>>>>>>> >>>>>>>> * default-catalog causes table references inside views to be early >>>>>>>> bound (i.e., bound at view creation time, especially when using a >>>>>>>> non-null >>>>>>>> value), while table references inside standalone queries are late bound >>>>>>>> (bound at query time). This creates inconsistencies when resolving the >>>>>>>> same >>>>>>>> table name inside and outside views, even within the same job. >>>>>>>> >>>>>>>> * It causes Spark's and Trino behavior to drift from the spec. >>>>>>>> There is no way to fully align Spark's behavior without making invasive >>>>>>>> changes to the Spark SQL grammar and the View DataSource API >>>>>>>> (specifically >>>>>>>> on the CREATE side). This challenge would extend to other engines too. >>>>>>>> Both >>>>>>>> Spark and Trino set this field based on a heuristic in today's >>>>>>>> implementation. >>>>>>>> >>>>>>>> * With view nesting (views depending on views), these >>>>>>>> inconsistencies amplify further, forcing users and engines to reason >>>>>>>> about >>>>>>>> catalog resolution at every level in the view tree. >>>>>>>> >>>>>>>> * It will be difficult to migrate Hive views to Iceberg with that >>>>>>>> model. Migrated Hive views will have to unfollow that spec. >>>>>>>> >>>>>>>> How would you suggest approaching the engine-level changes required >>>>>>>> to support the current default-catalog field? >>>>>>>> Also, do you believe the Spark and Trino communities would align >>>>>>>> around having table resolution behave inconsistently between queries >>>>>>>> and >>>>>>>> views, or inconsistency between Iceberg and other types of views? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Walaa >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks <dwe...@apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I would agree with Jan's summary of why 'default-catalog' was >>>>>>>>> introduced, but I think we need to step back and align on what we are >>>>>>>>> really attempting to support in the spec. >>>>>>>>> >>>>>>>>> The issues we're discussing largely stem from using multiple >>>>>>>>> engines with cross catalog references and configurations where catalog >>>>>>>>> names are not aligned. If we have multiple engines that all have the >>>>>>>>> same >>>>>>>>> catalog names/configurations, the current spec implementation is well >>>>>>>>> defined for table resolution even across catalogs. The >>>>>>>>> 'default-catalog' >>>>>>>>> (and namespace equivalent) was intended to address the resolution >>>>>>>>> within >>>>>>>>> the context of the sql text, not to address catalog/naming >>>>>>>>> inconsistencies. >>>>>>>>> >>>>>>>>> I feel like we're trying to adapt the original intent to address >>>>>>>>> the catalog naming/configuration and would argue that we shouldn't >>>>>>>>> attempt >>>>>>>>> to do that as part of the spec. Inconsistently named catalogs are a >>>>>>>>> reality, but we should consider that a configuration/environmental >>>>>>>>> issue, >>>>>>>>> not something to solve for in the spec. >>>>>>>>> >>>>>>>>> We should support and advocate for consistency in catalog naming >>>>>>>>> and define the spec along those lines. The fact is that with all of >>>>>>>>> the >>>>>>>>> recent work that's gone into making catalogs pluggable, it makes more >>>>>>>>> sense >>>>>>>>> to just register catalog configuration with consistent names (even if >>>>>>>>> you >>>>>>>>> have to duplicate the configuration for supporting existing >>>>>>>>> readers/writers). I think it's better to provide a path toward >>>>>>>>> consistency >>>>>>>>> than to normalize complicated schemes to workaround the issues caused >>>>>>>>> by >>>>>>>>> environmental/configuration inconsistencies. >>>>>>>>> >>>>>>>>> If the goal is to create clever ways to hack the late binding >>>>>>>>> resolution to swap in different catalogs or make references >>>>>>>>> contextual, I >>>>>>>>> feel like that is something we should strongly discourage as it leads >>>>>>>>> to >>>>>>>>> confusion about what is resolved as part of the query. >>>>>>>>> >>>>>>>>> At this point, I don't see a good argument to add >>>>>>>>> additional configuration or change the resolution behaviors. >>>>>>>>> >>>>>>>>> -Dan >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul >>>>>>>>> <jank...@mailbox.org.invalid> wrote: >>>>>>>>> >>>>>>>>>> I think the intention with the "default-catalog" was that every >>>>>>>>>> query engine uses it to store its session default catalog at the >>>>>>>>>> time of >>>>>>>>>> creating the view. This way the view could be reused in another >>>>>>>>>> session. >>>>>>>>>> The idea was not to introduce an additional SQL syntax to set the >>>>>>>>>> default-catalog. >>>>>>>>>> >>>>>>>>>> Generally we have different environments we want to support with >>>>>>>>>> the view spec: >>>>>>>>>> >>>>>>>>>> 1. Consistent catalog naming >>>>>>>>>> >>>>>>>>>> When the environment supports it, using consistent catalog names >>>>>>>>>> can have a great benefit for multi-catalog, multi-engine setups. With >>>>>>>>>> consistent catalog names, using the "default-catalog" field works >>>>>>>>>> without >>>>>>>>>> any issues. >>>>>>>>>> >>>>>>>>>> 2. Inconsistent catalog naming >>>>>>>>>> >>>>>>>>>> This can be the case when different query engines refer to the >>>>>>>>>> same physical catalog by different names. This often happens because >>>>>>>>>> different query engines use different strategies to setup the >>>>>>>>>> catalogs. If >>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" field >>>>>>>>>> does >>>>>>>>>> not work because it is not guaranteed that the catalog name can be >>>>>>>>>> resolved >>>>>>>>>> with another engine. Using the "view catalog" as a fallback is a >>>>>>>>>> better >>>>>>>>>> solution for this use case, as it avoids catalog names altogether. >>>>>>>>>> It is >>>>>>>>>> however limited to table references in the same catalog. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> What do you think of introducing a view property that specifies >>>>>>>>>> if the "default-catalog" or the "view catalog" should be used? This >>>>>>>>>> way, >>>>>>>>>> you could use the "default-catalog" in environments where you can >>>>>>>>>> guarantee >>>>>>>>>> consistent naming, but you would be able to directly fallback to the >>>>>>>>>> "view-catalog" when you don't have consistent naming. The query >>>>>>>>>> engines >>>>>>>>>> could set the default for this view property at creation time. Spark >>>>>>>>>> for >>>>>>>>>> example could set it to automatically use the "view catalog". >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> Jan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote: >>>>>>>>>> >>>>>>>>>> To help folks catch up on the latest discussions and >>>>>>>>>> interpretation of the spec, I have summarized everything we >>>>>>>>>> discussed so >>>>>>>>>> far at the top of the proposal document (here >>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>). >>>>>>>>>> I have slightly updated the proposal to be in sync with the new >>>>>>>>>> interpretation to avoid confusion. In summary: >>>>>>>>>> >>>>>>>>>> * Remove default-catalog and default-namespace fields from the >>>>>>>>>> view spec completely. >>>>>>>>>> >>>>>>>>>> * Hence, we do not attempt to define separate view-level default >>>>>>>>>> catalogs or namespaces. >>>>>>>>>> >>>>>>>>>> Instead: >>>>>>>>>> >>>>>>>>>> * If a table identifier inside a view lacks a catalog qualifier, >>>>>>>>>> engines should resolve it using the current engine catalog at query >>>>>>>>>> time. >>>>>>>>>> >>>>>>>>>> * Reference table identifiers in the metadata exactly as they >>>>>>>>>> appear in the view SQL text. >>>>>>>>>> >>>>>>>>>> * If an identifier lacks the catalog part at creation, it should >>>>>>>>>> still lack a catalog in the stored metadata. >>>>>>>>>> >>>>>>>>>> * Store UUIDs alongside table identifiers whenever possible. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Walaa. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa < >>>>>>>>>> wa.moust...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion the >>>>>>>>>>> fallback creates. Also just to be clear, at this point and after >>>>>>>>>>> clarifying >>>>>>>>>>> the current spec intentions, I am convinced that we should remove >>>>>>>>>>> the >>>>>>>>>>> default catalog and default namespace fields altogether. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Walaa. >>>>>>>>>>> >>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <btc...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> I'd like to contribute my opinions on this: >>>>>>>>>>>> >>>>>>>>>>>> - I don't particularly like the current behavior of "default to >>>>>>>>>>>> the view's catalog when default-catalog is not set". >>>>>>>>>>>> Fundamentally, I >>>>>>>>>>>> believe the intent of default-catalog and default-namespace is >>>>>>>>>>>> there to >>>>>>>>>>>> help users write more concise SQL. >>>>>>>>>>>> - spark session catalog is engine specific and I don't think we >>>>>>>>>>>> should design something that says first use this catalog, then that >>>>>>>>>>>> catalog.. or that catalog. For example, resolving identifiers >>>>>>>>>>>> using >>>>>>>>>>>> default-catalog -> view's catalog -> session catalog is not good. >>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see no value >>>>>>>>>>>> in putting views in the catalog to share with other engines >>>>>>>>>>>> - Interoperability between different engine types is very hard >>>>>>>>>>>> due to dialect issues... so I think we should focus on supporting >>>>>>>>>>>> different >>>>>>>>>>>> clusters of the same engine type on a shared catalog. For >>>>>>>>>>>> example, AI and >>>>>>>>>>>> BI clusters on Spark sharing the same views in a REST catalog. >>>>>>>>>>>> >>>>>>>>>>>> Coincidentally, I think the ultimate solution is along the >>>>>>>>>>>> lines of something Russell proposed last year: >>>>>>>>>>>> >>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7 >>>>>>>>>>>> >>>>>>>>>>>> We've been looking at this interoperable identifier problem >>>>>>>>>>>> through the lens of catalog resolution but maybe the right >>>>>>>>>>>> approach is >>>>>>>>>>>> really about templating. >>>>>>>>>>>> >>>>>>>>>>>> I would extend Russell's idea to allow identifiers in a view to >>>>>>>>>>>> span catalogs to support non-Iceberg tables. Also, the >>>>>>>>>>>> default-catalog >>>>>>>>>>>> property could be templated as well. >>>>>>>>>>>> >>>>>>>>>>>> Thoughts? >>>>>>>>>>>> Benny >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa < >>>>>>>>>>>> wa.moust...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark >>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL >>>>>>>>>>>>> extensions and/or >>>>>>>>>>>>> Spark catalog APIs for that? >>>>>>>>>>>>> >>>>>>>>>>>>> How do you recommend reconciling the inconsistencies I shared >>>>>>>>>>>>> regarding many resolution methods not consistently being followed >>>>>>>>>>>>> in >>>>>>>>>>>>> different scenarios (view vs child table resolution, query vs view >>>>>>>>>>>>> resolution)? Note these occur when the default catalog is set to >>>>>>>>>>>>> a non-null >>>>>>>>>>>>> value. If it helps, I can share concrete examples. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Walaa. >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu < >>>>>>>>>>>>> stevenz...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> The core issue is on the fall back behavior when >>>>>>>>>>>>>> `default-catalog` is >>>>>>>>>>>>>> not defined. Current view spec says the fallback should be >>>>>>>>>>>>>> the catalog >>>>>>>>>>>>>> where the view is defined. It doesn't really matter what the >>>>>>>>>>>>>> catalog >>>>>>>>>>>>>> is named (catalogX) by the read engine. >>>>>>>>>>>>>> - If a view refers to the tables in the same catalog, this is >>>>>>>>>>>>>> a >>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior. >>>>>>>>>>>>>> - If a view refers to tables from another catalog, catalog >>>>>>>>>>>>>> names >>>>>>>>>>>>>> should be included in the reference name already. So no >>>>>>>>>>>>>> ambiguity >>>>>>>>>>>>>> there either. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Potential inconsistent naming of catalog is a separate >>>>>>>>>>>>>> problem, which >>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can only >>>>>>>>>>>>>> recommend that >>>>>>>>>>>>>> catalog should be named consistently across usage for better >>>>>>>>>>>>>> interoperability on name references. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This proposal is to change the fallback behavior to engine's >>>>>>>>>>>>>> session >>>>>>>>>>>>>> default catalog. I am not sure it is better than the current >>>>>>>>>>>>>> fallback >>>>>>>>>>>>>> behavior. >>>>>>>>>>>>>> >>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from this idea. >>>>>>>>>>>>>> Spark resolves table identifiers during view creation using the >>>>>>>>>>>>>> session’s >>>>>>>>>>>>>> default catalog, not a supplied `default-catalog`. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I would argue that is a Spark implementation issue for not >>>>>>>>>>>>>> conforming >>>>>>>>>>>>>> to the spec. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa >>>>>>>>>>>>>> <wa.moust...@gmail.com> wrote: >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Hi Jan, >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Thanks again for continuing the discussion. I want to >>>>>>>>>>>>>> highlight a few fundamental issues around the interpretation of >>>>>>>>>>>>>> default-catalog: >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Here is the real catch: >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > * default-catalog cannot logically be defined at view >>>>>>>>>>>>>> creation time. It would be circular: the view needs to exist >>>>>>>>>>>>>> before its >>>>>>>>>>>>>> metadata (and hence default-catalog) can exist. This is visible >>>>>>>>>>>>>> in Spark’s >>>>>>>>>>>>>> implementation, where `default-catalog` is not used. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > * Introducing a creation-time default-catalog setting would >>>>>>>>>>>>>> require extending SQL syntax and engine APIs to promote it to a >>>>>>>>>>>>>> first-class >>>>>>>>>>>>>> view concept. This would be intrusive, non-intuitive, and >>>>>>>>>>>>>> realistically >>>>>>>>>>>>>> very difficult to standardize across engines. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from this idea. >>>>>>>>>>>>>> Spark resolves table identifiers during view creation using the >>>>>>>>>>>>>> session’s >>>>>>>>>>>>>> default catalog, not a supplied `default-catalog`. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > * Hypothetically even if we patched in a creation-time >>>>>>>>>>>>>> default-catalog, it would create an inconsistent binding model >>>>>>>>>>>>>> between >>>>>>>>>>>>>> tables vs views (early vs late), and between tables in views and >>>>>>>>>>>>>> in queries >>>>>>>>>>>>>> (again early vs late). For example, views and tables in queries >>>>>>>>>>>>>> can >>>>>>>>>>>>>> withstand default catalog renames, but tables cannot when they >>>>>>>>>>>>>> are used >>>>>>>>>>>>>> inside views -- it even applies to views inside views, which >>>>>>>>>>>>>> makes this >>>>>>>>>>>>>> very hard to reason about considering nesting. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Thanks, >>>>>>>>>>>>>> > Walaa >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul >>>>>>>>>>>>>> <jank...@mailbox.org.invalid> <jank...@mailbox.org.invalid> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> @Walaa: >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW statement >>>>>>>>>>>>>> the query engine knowns which catalog the view is being created >>>>>>>>>>>>>> in. So even >>>>>>>>>>>>>> though we typically use late binding to resolve the view catalog >>>>>>>>>>>>>> at query >>>>>>>>>>>>>> time, it can also be used at creation time. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> The query engine would need to keep track of the "view >>>>>>>>>>>>>> catalog" where the view is going to be created in. It can use >>>>>>>>>>>>>> that catalog >>>>>>>>>>>>>> to resolve partial table identifiers if "default-catalog" is not >>>>>>>>>>>>>> set. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where partial >>>>>>>>>>>>>> identifiers in the view query resolve to a different catalog >>>>>>>>>>>>>> compared to >>>>>>>>>>>>>> using them outside of a view. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS SELECT * from >>>>>>>>>>>>>> sales.orders; >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> If the session default catalog is not "catalogA", the >>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as just >>>>>>>>>>>>>> referencing >>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because >>>>>>>>>>>>>> without a >>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would >>>>>>>>>>>>>> default to >>>>>>>>>>>>>> "catalogA", which is the view's catalog. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Thanks, >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Jan >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote: >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> For example, if we want to validate that the tables >>>>>>>>>>>>>> referenced in the view exist, how can we do that when >>>>>>>>>>>>>> default-catalog isn't >>>>>>>>>>>>>> defined, since the view hasn't been created or loaded yet? >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> I don't think this is related to view spec. How do we >>>>>>>>>>>>>> validate that a table exists without a default catalog, or do we >>>>>>>>>>>>>> always use >>>>>>>>>>>>>> the current session catalog? >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Thanks, >>>>>>>>>>>>>> >> Manu >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin Moustafa < >>>>>>>>>>>>>> wa.moust...@gmail.com> wrote: >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> Hi Jan, >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> I think we still share the same understanding. Just to >>>>>>>>>>>>>> clarify: when I referred to late binding as “similar” to the >>>>>>>>>>>>>> proposal, I >>>>>>>>>>>>>> was acknowledging the distinction between view-level and >>>>>>>>>>>>>> table-level >>>>>>>>>>>>>> resolution. But as you noted, both follow a late binding model. >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> That said, this still raises an interesting question and >>>>>>>>>>>>>> a potential gap: if default-catalog is only defined at query >>>>>>>>>>>>>> time, how >>>>>>>>>>>>>> should resolution work during view creation? For example, if we >>>>>>>>>>>>>> want to >>>>>>>>>>>>>> validate that the tables referenced in the view exist, how can >>>>>>>>>>>>>> we do that >>>>>>>>>>>>>> when default-catalog isn't defined, since the view hasn't been >>>>>>>>>>>>>> created or >>>>>>>>>>>>>> loaded yet? >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> Thanks, >>>>>>>>>>>>>> >>> Walaa. >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul >>>>>>>>>>>>>> <jank...@mailbox.org.invalid> <jank...@mailbox.org.invalid> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view catalog is >>>>>>>>>>>>>> resolved at query time. >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> As you mentioned before, it's good to distinguish >>>>>>>>>>>>>> between the physical catalog and it's reference used in SQL >>>>>>>>>>>>>> statements. The >>>>>>>>>>>>>> important part is that the physical catalog of the view and the >>>>>>>>>>>>>> tables >>>>>>>>>>>>>> referenced in it's definition stay consistent. You could create >>>>>>>>>>>>>> a view in a >>>>>>>>>>>>>> given physical catalog by referring to it as "catalogA", as in >>>>>>>>>>>>>> your first >>>>>>>>>>>>>> point. If you then, given a different setup, refer to the same >>>>>>>>>>>>>> physical >>>>>>>>>>>>>> catalog as "catalogB" in another session/environment, the >>>>>>>>>>>>>> behavior should >>>>>>>>>>>>>> still work. >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late binding >>>>>>>>>>>>>> applies to the view catalog name and by extension to all partial >>>>>>>>>>>>>> table >>>>>>>>>>>>>> references when no "default-catalog" is present. Resolving the >>>>>>>>>>>>>> view catalog >>>>>>>>>>>>>> name at query time is not opposed to storing the view metadata >>>>>>>>>>>>>> in a catalog. >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you mean. >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Thanks >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Jan >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote: >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Hi Jan, >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> > The view is executed when it's being referenced in a >>>>>>>>>>>>>> SQL statement. That statement contains the information for the >>>>>>>>>>>>>> query engine >>>>>>>>>>>>>> to resolve the catalog of the view. >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means: >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM >>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the view’s >>>>>>>>>>>>>> catalog. >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT * FROM >>>>>>>>>>>>>> catalogB.namespace.view (after renaming catalogA to catalogB, >>>>>>>>>>>>>> and keeping >>>>>>>>>>>>>> everything else the same), then catalogB becomes the view’s >>>>>>>>>>>>>> catalog. >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it sounds to me >>>>>>>>>>>>>> like the catalog is resolved at query time, based on how the >>>>>>>>>>>>>> view is >>>>>>>>>>>>>> referenced, not from any stored metadata. That would imply some >>>>>>>>>>>>>> sort of a >>>>>>>>>>>>>> late binding behavior (similar to the proposal), as opposed to >>>>>>>>>>>>>> using some >>>>>>>>>>>>>> catalog that "stores" the view definition. >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Thanks, >>>>>>>>>>>>>> >>>> Walaa >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul >>>>>>>>>>>>>> <jank...@mailbox.org.invalid> <jank...@mailbox.org.invalid> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> Hi Walaa, >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of non-determinism. >>>>>>>>>>>>>> Let me try to address your questions. >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current spec: The >>>>>>>>>>>>>> view is executed when it's being referenced in a SQL statement. >>>>>>>>>>>>>> That >>>>>>>>>>>>>> statement contains the information for the query engine to >>>>>>>>>>>>>> resolve the >>>>>>>>>>>>>> catalog of the view. The query engine then uses that information >>>>>>>>>>>>>> to fetch >>>>>>>>>>>>>> the view metadata from the catalog. It also needs to temporarily >>>>>>>>>>>>>> keep track >>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can then >>>>>>>>>>>>>> use that >>>>>>>>>>>>>> information to resolve the table references in the views SQL >>>>>>>>>>>>>> definition in >>>>>>>>>>>>>> case no default catalog is specified. >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can be >>>>>>>>>>>>>> referenced at execution time. As long as that's the case I would >>>>>>>>>>>>>> assume the >>>>>>>>>>>>>> view can be created in any catalog. >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> I think your point is really valuable because the >>>>>>>>>>>>>> current specification can lead to some unintuitive behavior. For >>>>>>>>>>>>>> example >>>>>>>>>>>>>> for the following statement: >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS SELECT * >>>>>>>>>>>>>> from sales.orders; >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> If the session default catalog is not "catalogA", the >>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as just >>>>>>>>>>>>>> referencing >>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because >>>>>>>>>>>>>> without a >>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would >>>>>>>>>>>>>> default to >>>>>>>>>>>>>> "catalogA". >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> However, I like the current design of the view spec, >>>>>>>>>>>>>> because it has the "closure" property. Because of the fact that >>>>>>>>>>>>>> the "view >>>>>>>>>>>>>> catalog" has to be known when executing a view, all the >>>>>>>>>>>>>> information >>>>>>>>>>>>>> required to resolve the table identifiers is contained in the >>>>>>>>>>>>>> view metadata >>>>>>>>>>>>>> (and the "view catalog"). I think that if you make the identifier >>>>>>>>>>>>>> resolution dependent on external parameters, it hinders >>>>>>>>>>>>>> portability. >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> Thanks, >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> Jan >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote: >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> Hi Jan, >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback. >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point before >>>>>>>>>>>>>> going deeper: >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session fallback >>>>>>>>>>>>>> behavior—it’s a fundamental limitation of using table >>>>>>>>>>>>>> identifiers alone, >>>>>>>>>>>>>> regardless of whether we use the current rule, the proposed >>>>>>>>>>>>>> fallback to the >>>>>>>>>>>>>> session’s default catalog, or even early vs. late binding. >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g., >>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects >>>>>>>>>>>>>> depending solely >>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So >>>>>>>>>>>>>> determinism isn’t >>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." The >>>>>>>>>>>>>> only >>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the >>>>>>>>>>>>>> proposed use of >>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix for >>>>>>>>>>>>>> correctness. >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> To move the conversation forward, could you help >>>>>>>>>>>>>> clarify two things in the context of the current spec: >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog” stored, >>>>>>>>>>>>>> so that an engine knows to fall back to it if default-catalog is >>>>>>>>>>>>>> null? >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the session's >>>>>>>>>>>>>> default catalog (i.e., without specifying a catalog) in the >>>>>>>>>>>>>> current Iceberg >>>>>>>>>>>>>> spec? >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> These questions are important because if we can’t >>>>>>>>>>>>>> unambiguously recover the "view catalog" from metadata, then >>>>>>>>>>>>>> defaulting to >>>>>>>>>>>>>> it is problematic. And if views can't be created in the default >>>>>>>>>>>>>> catalog, >>>>>>>>>>>>>> then the fallback rule doesn’t generalize. >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> Thanks, >>>>>>>>>>>>>> >>>>> Walaa. >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul >>>>>>>>>>>>>> <jank...@mailbox.org.invalid> <jank...@mailbox.org.invalid> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> Hi Walaa, >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood >>>>>>>>>>>>>> correctly, you proposal is composed of three parts: >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for >>>>>>>>>>>>>> "default-catalog" >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for >>>>>>>>>>>>>> "default-namepace" >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> I have some comments regarding these points. >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for >>>>>>>>>>>>>> "default-catalog" >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the current >>>>>>>>>>>>>> session setup is in my opinion the definition of >>>>>>>>>>>>>> "non-determinism". You >>>>>>>>>>>>>> could be running the same query-engine and catalog-setup on >>>>>>>>>>>>>> different days, >>>>>>>>>>>>>> with different default session catalogs (which is rather >>>>>>>>>>>>>> common), and would >>>>>>>>>>>>>> be getting different results. >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view always >>>>>>>>>>>>>> produces the same results. The current behavior has some rough >>>>>>>>>>>>>> edges in >>>>>>>>>>>>>> very niche use cases but I think is solid for most uses cases. >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for >>>>>>>>>>>>>> "default-namespace" >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> Similar to the above. >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current >>>>>>>>>>>>>> implementation already uses late binding. >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the setup more >>>>>>>>>>>>>> robust. Which is great. However, having UUID validation still >>>>>>>>>>>>>> requires us >>>>>>>>>>>>>> to have a portable table identifier specification. Even if we >>>>>>>>>>>>>> have the >>>>>>>>>>>>>> UUIDs of the referenced tables from the view, there simply isn't >>>>>>>>>>>>>> an >>>>>>>>>>>>>> interface that let's us use those UUIDs. The catalog interface >>>>>>>>>>>>>> is defined >>>>>>>>>>>>>> in terms of table identifiers. >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup and >>>>>>>>>>>>>> suiting table identifiers to obtain the table metadata. We can >>>>>>>>>>>>>> use the >>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this can >>>>>>>>>>>>>> only be done >>>>>>>>>>>>>> after we used some identifier. Which means there is no way of >>>>>>>>>>>>>> using UUIDs >>>>>>>>>>>>>> without a functioning catalog/identifier setup. >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior for >>>>>>>>>>>>>> "default-catalog" because it is more deterministic in my >>>>>>>>>>>>>> opinion. And I >>>>>>>>>>>>>> think the current spec does a good job for multi-engine table >>>>>>>>>>>>>> identifier >>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional >>>>>>>>>>>>>> hardening >>>>>>>>>>>>>> strategy. >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> Thanks >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> Jan >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote: >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> Thanks Renjie! >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on resolving >>>>>>>>>>>>>> catalogs on the fly already (to address the case of view text >>>>>>>>>>>>>> with table >>>>>>>>>>>>>> identifiers missing the catalog part). The guidance is to use >>>>>>>>>>>>>> the catalog >>>>>>>>>>>>>> where the view is stored. But I find this rule hard to interpret >>>>>>>>>>>>>> or use. >>>>>>>>>>>>>> The catalog itself is a logical construct—such as a federated >>>>>>>>>>>>>> catalog that >>>>>>>>>>>>>> delegates to multiple physical backends (e.g., HMS and REST). In >>>>>>>>>>>>>> such >>>>>>>>>>>>>> cases, the catalog (e.g., `my_catalog` in >>>>>>>>>>>>>> `my_catalog.namespace1.table1`) >>>>>>>>>>>>>> doesn’t physically store the tables; it only routes requests to >>>>>>>>>>>>>> underlying >>>>>>>>>>>>>> stores. Therefore, defaulting identifier resolution based on the >>>>>>>>>>>>>> catalog >>>>>>>>>>>>>> where the view is "stored" doesn’t align with how catalogs >>>>>>>>>>>>>> actually behave >>>>>>>>>>>>>> in practice. >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> Thanks, >>>>>>>>>>>>>> >>>>>> Walaa. >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu < >>>>>>>>>>>>>> liurenjie2...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> >>>>>>> Hi, Walaa: >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> >>>>>>> Thanks for the proposal. >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I have some >>>>>>>>>>>>>> concerns with resolving catalog names on the fly with query >>>>>>>>>>>>>> engine defined >>>>>>>>>>>>>> catalog names. This introduces some flexibility at first glance, >>>>>>>>>>>>>> but also >>>>>>>>>>>>>> makes misconfiguration difficult to explain. >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should store >>>>>>>>>>>>>> resolved table uuid in view metadata, as table/view renaming may >>>>>>>>>>>>>> introduce >>>>>>>>>>>>>> errors that's difficult to understand for user. >>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin Moustafa < >>>>>>>>>>>>>> wa.moust...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> Hi Everyone, >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum and >>>>>>>>>>>>>> closing out the MV spec as well. I’m hoping we can proceed to a >>>>>>>>>>>>>> vote next >>>>>>>>>>>>>> week. >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The proposal >>>>>>>>>>>>>> outlines a strategy for handling table identifiers in Iceberg >>>>>>>>>>>>>> view >>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, portability, >>>>>>>>>>>>>> and engine >>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at read >>>>>>>>>>>>>> time (late >>>>>>>>>>>>>> binding) rather than creation time, and introduces UUID-based >>>>>>>>>>>>>> validation to >>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. It >>>>>>>>>>>>>> also revises >>>>>>>>>>>>>> how default-catalog and default-namespace are handled >>>>>>>>>>>>>> (defaulting both to >>>>>>>>>>>>>> the session context if not explicitly set) to better align with >>>>>>>>>>>>>> engine >>>>>>>>>>>>>> behavior and improve cross-engine interoperability. >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts. >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>> Walaa. >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin Moustafa >>>>>>>>>>>>>> <wa.moust...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed the >>>>>>>>>>>>>> comments. >>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that catalog names >>>>>>>>>>>>>> in the spec refer to logical catalogs—i.e., the first part of a >>>>>>>>>>>>>> three-part >>>>>>>>>>>>>> identifier. These correspond to Spark's DataSourceV2 catalogs, >>>>>>>>>>>>>> Trino >>>>>>>>>>>>>> connectors, and similar constructs. This is a level of >>>>>>>>>>>>>> abstraction above >>>>>>>>>>>>>> physical catalogs, which are not referenced or used in the view >>>>>>>>>>>>>> spec. The >>>>>>>>>>>>>> reason is that table identifiers in the view definition/text >>>>>>>>>>>>>> itself refer >>>>>>>>>>>>>> to logical catalogs, not physical ones (since they interface >>>>>>>>>>>>>> directly with >>>>>>>>>>>>>> the engine and not a specific metastore). >>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>> Walaa. >>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun < >>>>>>>>>>>>>> sungwy...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I think view >>>>>>>>>>>>>> portability is a very important topic for us to continue >>>>>>>>>>>>>> discussing as it >>>>>>>>>>>>>> relies on many assumptions within the data ecosystem for it to >>>>>>>>>>>>>> function >>>>>>>>>>>>>> like you've highlighted well in the document. >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how this may >>>>>>>>>>>>>> impact the permission questions the engines will be asking, and >>>>>>>>>>>>>> whether >>>>>>>>>>>>>> that is the desired behavior. >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> Sung >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard Tudenhöfner >>>>>>>>>>>>>> <etudenhoef...@apache.org> wrote: >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem. I've >>>>>>>>>>>>>> added a few comments to get a better understanding of how this >>>>>>>>>>>>>> will look >>>>>>>>>>>>>> like in the actual implementation. >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> Eduard >>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa Eldin >>>>>>>>>>>>>> Moustafa <wa.moust...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone, >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our discussion on >>>>>>>>>>>>>> how to reference table identifiers from Iceberg metadata, a key >>>>>>>>>>>>>> aspect of >>>>>>>>>>>>>> the view specification, particularly in relation to the MV >>>>>>>>>>>>>> (materialized >>>>>>>>>>>>>> view) extensions. >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with a few >>>>>>>>>>>>>> community members to better understand how the current spec is >>>>>>>>>>>>>> being >>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new >>>>>>>>>>>>>> proposal on how >>>>>>>>>>>>>> table identifier references could be represented in metadata. >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I look >>>>>>>>>>>>>> forward to your feedback and working together to move this >>>>>>>>>>>>>> forward so we >>>>>>>>>>>>>> can finalize the MV spec as well. >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0 >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>> Walaa. >>>>>>>>>>>>>> >>>>>>>>>>>>>
