Thanks Steven! So would you agree that resolution using default-catalog and default-namespace does not provide full determinism, and requires a supporting safety mechanism?
Thanks, Walaa. On Wed, May 7, 2025 at 10:30 PM Steven Wu <stevenz...@gmail.com> wrote: > > If the current model is considered deterministic, do you think > `default-catalog` and `default-namespace` fields provide enough determinism > to eliminate the need for UUIDs when storing table identifiers? > > I am fine with storing UUIDs for table identifiers in the view. Basically, > view creation resolves all referenced tables/views with UUIDs. View > consumers can validate resolved tables/views with the stored UUIDs and fail > the query if mismatch. > > The UUID change doesn't really change the table identifier resolution rule > though. It is more of a safety protection. > > On Wed, May 7, 2025 at 10:02 PM Walaa Eldin Moustafa < > wa.moust...@gmail.com> wrote: > >> Hi Steven, >> >> Thanks for the reply. >> >> > I agree with Dan that we shouldn't solve catalog naming in the Iceberg >> view spec. >> >> To clarify, I don't believe the proposal is trying to solve catalog >> naming. What it’s doing is simply this: >> >> * Proposing that table names inside views resolve the same way as they do >> elsewhere (e.g., queries). >> * Adopting a model that is already widely used and supported in the >> existing ecosystem, which allows for: >> -- Renaming catalog aliases >> -- Swapping catalog implementations behind consistent names >> -- Having different default catalog names across engines that still >> point to the same underlying tables >> >> These are common patterns in production data lakes. Saying Iceberg views >> cannot operate in those environments feels unrealistic. In practice, it >> means the spec breaks down in situations that users encounter regularly. >> >> > The recommendation of using engines’ current catalog and database can >> cause context-dependent resolution results. >> >> * As noted in the doc and earlier replies, fixing a catalog name doesn’t >> actually guarantee determinism either. All the failure scenarios above >> still apply even when a default-catalog is stored. >> * The current spec also allows default-catalog to be null, in which case >> it falls back to the view’s catalog, yet that catalog is determined based >> on how the view is referenced in the query, which would be considered >> non-deterministic based on the same criteria you shared. >> * The only true form of determinism here is UUID-based validation, which >> protects against silent drift in any resolution model. >> >> If the current model is considered deterministic, do you think >> `default-catalog` and `default-namespace` fields provide enough determinism >> to eliminate the need for UUIDs when storing table identifiers? >> Or put another way: Would you be comfortable relying solely on >> default-catalog + default-namespace + table name to re-identify the correct >> table, without UUID validation? >> >> +1 on involving other communities. I’m happy to help facilitate a >> cross-community discussion if we aren’t able to reach a resolution here. >> >> Thanks, >> Walaa. >> >> >> >> On Wed, May 7, 2025 at 9:20 PM Steven Wu <stevenz...@gmail.com> wrote: >> >>> I agree with Dan that we shouldn't solve catalog naming in the Iceberg >>> view spec. I am not convinced that the proposed change will make the table >>> identifier resolution more clear and portable. The recommendation of using >>> engines' current catalog and database can cause context dependent >>> resolution results, which seems non-deterministic to me. >>> >>> Walaa, you raised a point in the doc that the current catalog resolution >>> logic (default-catalog field, then view catalog) is challenging and >>> unrealistic for engines (like Spark and Trino). It will be great to get >>> more inputs from the broader community on this part. >>> >>> >>> On Tue, May 6, 2025 at 9:21 AM Benny Chow <btc...@gmail.com> wrote: >>> >>>> In Spark, I believe that the USE commands sets the current catalog and >>>> namespace. This affects both where the view is created and how unqualified >>>> table identifiers are resolved. I also don't see an issue with saving the >>>> current catalog and namespace into the view metadata's default-catalog and >>>> default-namespace fields. >>>> >>>> On Wed, Apr 30, 2025 at 5:12 PM Walaa Eldin Moustafa < >>>> wa.moust...@gmail.com> wrote: >>>> >>>>> > I think that's the lesser evil compared to Iceberg specifying how >>>>> engines should resolve identifiers >>>>> >>>>> I think this is also similar to the previous point. It is the other >>>>> way around. Right now the spec dictates how to resolve (through employing >>>>> a >>>>> view-specific `default-catalog` field). The proposal is suggesting to get >>>>> out of this space and let engines handle it similar to how they handle all >>>>> identifiers. >>>>> >>>>> On Wed, Apr 30, 2025 at 5:07 PM Walaa Eldin Moustafa < >>>>> wa.moust...@gmail.com> wrote: >>>>> >>>>>> > I thought "default-catalog" could be set via the USE command. >>>>>> >>>>>> Benny, I think this is a misconception or miscommunication. The USE >>>>>> command has no impact on the `default-catalog` field. In fact, the >>>>>> proposal's direction is exactly to establish that USE command should >>>>>> influence how tables are resolved, same like everywhere else. Right now >>>>>> it >>>>>> is not the case under the current spec. >>>>>> >>>>>> >>>>>> On Wed, Apr 30, 2025 at 3:17 PM Benny Chow <btc...@gmail.com> wrote: >>>>>> >>>>>>> > there is no SQL construct today to explicitly set default-catalog >>>>>>> >>>>>>> I thought "default-catalog" could be set via the USE command. >>>>>>> >>>>>>> I generally agree with Dan about requiring consistent catalog >>>>>>> names. I think that's the lesser evil compared to Iceberg specifying >>>>>>> how >>>>>>> engines should resolve identifiers. Another thing to consider is that >>>>>>> identifier resolution can be very expensive at query validation time if >>>>>>> identifiers need to be looked up from a bunch of places. Hopefully, it >>>>>>> should be possible to define a view in such a way that identifiers can >>>>>>> be >>>>>>> resolved on the first try. >>>>>>> >>>>>>> Benny >>>>>>> >>>>>>> On Tue, Apr 29, 2025 at 10:29 PM Walaa Eldin Moustafa < >>>>>>> wa.moust...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Rishabh, >>>>>>>> >>>>>>>> You're right that the proposal touches on two aspects, and >>>>>>>> resolution rules are one of them. The other aspect is the proposal's >>>>>>>> position that table identifiers should be stored in metadata exactly as >>>>>>>> they appear in the view text (e.g., even if they're two-part or >>>>>>>> partially >>>>>>>> qualified), along with their corresponding UUIDs for validation. This >>>>>>>> applies both to referenced input tables and the storage table >>>>>>>> identifier in >>>>>>>> materialized views. >>>>>>>> >>>>>>>> We may be able to converge on this storage format even if we >>>>>>>> haven't yet converged on the resolution fallback rules. I believe both >>>>>>>> resolution strategies currently being discussed would still lead to >>>>>>>> storing >>>>>>>> identifiers in this way. >>>>>>>> >>>>>>>> I'm supportive of moving forward with consensus on the identifier >>>>>>>> storage format. That said, we may continue to run into questions >>>>>>>> related to >>>>>>>> resolution during implementation. For example: Should the storage table >>>>>>>> identifier follow the same default-catalog and default-namespace >>>>>>>> resolution >>>>>>>> behavior as other table references? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Walaa. >>>>>>>> >>>>>>>> On Tue, Apr 29, 2025 at 10:07 PM Rishabh Bhatia < >>>>>>>> bhatiarishab...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hello Walaa, >>>>>>>>> >>>>>>>>> Thanks for starting this discussion. >>>>>>>>> >>>>>>>>> I think we should decouple at least the MV Spec from the proposal >>>>>>>>> to change the current behavior of view resolution. >>>>>>>>> >>>>>>>>> We can continue having the discussion if the current view spec >>>>>>>>> needs to be changed or not. Based on the decision at a later point if >>>>>>>>> required we can update the view resolution rule. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Rishabh >>>>>>>>> >>>>>>>>> On Mon, Apr 28, 2025 at 3:22 PM Walaa Eldin Moustafa < >>>>>>>>> wa.moust...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Correction of typo: both engines seem to set default-catalog to >>>>>>>>>> the view catalog if it is defined, or to null if the view catalog is >>>>>>>>>> not >>>>>>>>>> defined. >>>>>>>>>> >>>>>>>>>> On Mon, Apr 28, 2025 at 3:06 PM Walaa Eldin Moustafa < >>>>>>>>>> wa.moust...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Dan, >>>>>>>>>>> >>>>>>>>>>> Thanks again for your response. >>>>>>>>>>> >>>>>>>>>>> I agree that catalog renaming is an environmental event, but >>>>>>>>>>> it's a real one that happens frequently in practice. >>>>>>>>>>> Saying that the Iceberg spec cannot accommodate something as >>>>>>>>>>> common as catalog renaming feels very restrictive, and could make >>>>>>>>>>> the spec >>>>>>>>>>> less practical, even unusable, for real-world deployments. >>>>>>>>>>> I’m sharing this from the perspective of a large data lake >>>>>>>>>>> environment where views are heavily deployed and operationalized. >>>>>>>>>>> >>>>>>>>>>> Further, it's worth noting that the table spec is resilient to >>>>>>>>>>> catalog renaming, but the view spec is not. If we have an >>>>>>>>>>> opportunity to >>>>>>>>>>> make the view spec similarly resilient, I wonder why not? >>>>>>>>>>> Both specifications are deterministic in their definition, but >>>>>>>>>>> one is more fragile to environmental changes than the other. >>>>>>>>>>> Improving >>>>>>>>>>> resilience does not sacrifice determinism. It simply makes views >>>>>>>>>>> safer and >>>>>>>>>>> more portable over time. >>>>>>>>>>> >>>>>>>>>>> Separately, given that there is no SQL construct today to >>>>>>>>>>> explicitly set default-catalog at creation time, what is the >>>>>>>>>>> intuition >>>>>>>>>>> behind how engines like Spark and Trino currently assign >>>>>>>>>>> default-catalog? >>>>>>>>>>> Today, both engines seem to set default-catalog to null if the >>>>>>>>>>> view catalog is defined, or to the view catalog if not. >>>>>>>>>>> What was the intended thought process behind this behavior? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Walaa >>>>>>>>>>> >>>>>>>>>>> On Mon, Apr 28, 2025 at 1:33 PM Daniel Weeks <dwe...@apache.org> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Walaa, >>>>>>>>>>>> >>>>>>>>>>>> > tables inside views remain reachable after a catalog rename >>>>>>>>>>>> >>>>>>>>>>>> This problem stems from the exact environmental/configuration >>>>>>>>>>>> issue that we should not be trying to address. I don't think we >>>>>>>>>>>> would >>>>>>>>>>>> expect references to survive a catalog rename. That's not >>>>>>>>>>>> something >>>>>>>>>>>> covered by the spec and needs to be handled separately as a >>>>>>>>>>>> platform-level >>>>>>>>>>>> migration specific to the affected environment. >>>>>>>>>>>> >>>>>>>>>>>> The identifier resolution logic is clear and deterministic. It >>>>>>>>>>>> should not matter whether an engine resolves and encodes the >>>>>>>>>>>> default-catalog or leaves it to the resolution rules. >>>>>>>>>>>> >>>>>>>>>>>> The issue isn't with how the spec is defined, but rather view >>>>>>>>>>>> behavior when you start altering the environment around it, which >>>>>>>>>>>> isn't >>>>>>>>>>>> something we should be trying to define here. >>>>>>>>>>>> >>>>>>>>>>>> -Dan >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:17 PM Walaa Eldin Moustafa < >>>>>>>>>>>> wa.moust...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Dan, >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for chiming in. >>>>>>>>>>>>> >>>>>>>>>>>>> I believe the issues we’re seeing now go beyond just catalog >>>>>>>>>>>>> naming consistency. The behavior around default-catalog itself >>>>>>>>>>>>> introduces >>>>>>>>>>>>> resolution inconsistencies even when catalog names are consistent. >>>>>>>>>>>>> For example: >>>>>>>>>>>>> >>>>>>>>>>>>> * When default-catalog is set to null, tables inside views >>>>>>>>>>>>> remain reachable after a catalog rename. But if it is set to a >>>>>>>>>>>>> non-null >>>>>>>>>>>>> value, table references will break. >>>>>>>>>>>>> >>>>>>>>>>>>> * default-catalog causes table references inside views to be >>>>>>>>>>>>> early bound (i.e., bound at view creation time, especially when >>>>>>>>>>>>> using a >>>>>>>>>>>>> non-null value), while table references inside standalone queries >>>>>>>>>>>>> are late >>>>>>>>>>>>> bound (bound at query time). This creates inconsistencies when >>>>>>>>>>>>> resolving >>>>>>>>>>>>> the same table name inside and outside views, even within the >>>>>>>>>>>>> same job. >>>>>>>>>>>>> >>>>>>>>>>>>> * It causes Spark's and Trino behavior to drift from the spec. >>>>>>>>>>>>> There is no way to fully align Spark's behavior without making >>>>>>>>>>>>> invasive >>>>>>>>>>>>> changes to the Spark SQL grammar and the View DataSource API >>>>>>>>>>>>> (specifically >>>>>>>>>>>>> on the CREATE side). This challenge would extend to other engines >>>>>>>>>>>>> too. Both >>>>>>>>>>>>> Spark and Trino set this field based on a heuristic in today's >>>>>>>>>>>>> implementation. >>>>>>>>>>>>> >>>>>>>>>>>>> * With view nesting (views depending on views), these >>>>>>>>>>>>> inconsistencies amplify further, forcing users and engines to >>>>>>>>>>>>> reason about >>>>>>>>>>>>> catalog resolution at every level in the view tree. >>>>>>>>>>>>> >>>>>>>>>>>>> * It will be difficult to migrate Hive views to Iceberg with >>>>>>>>>>>>> that model. Migrated Hive views will have to unfollow that spec. >>>>>>>>>>>>> >>>>>>>>>>>>> How would you suggest approaching the engine-level changes >>>>>>>>>>>>> required to support the current default-catalog field? >>>>>>>>>>>>> Also, do you believe the Spark and Trino communities would >>>>>>>>>>>>> align around having table resolution behave inconsistently >>>>>>>>>>>>> between queries >>>>>>>>>>>>> and views, or inconsistency between Iceberg and other types of >>>>>>>>>>>>> views? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Walaa >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Apr 28, 2025 at 11:34 AM Daniel Weeks < >>>>>>>>>>>>> dwe...@apache.org> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I would agree with Jan's summary of why 'default-catalog' was >>>>>>>>>>>>>> introduced, but I think we need to step back and align on what >>>>>>>>>>>>>> we are >>>>>>>>>>>>>> really attempting to support in the spec. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The issues we're discussing largely stem from using multiple >>>>>>>>>>>>>> engines with cross catalog references and configurations where >>>>>>>>>>>>>> catalog >>>>>>>>>>>>>> names are not aligned. If we have multiple engines that all >>>>>>>>>>>>>> have the same >>>>>>>>>>>>>> catalog names/configurations, the current spec implementation is >>>>>>>>>>>>>> well >>>>>>>>>>>>>> defined for table resolution even across catalogs. The >>>>>>>>>>>>>> 'default-catalog' >>>>>>>>>>>>>> (and namespace equivalent) was intended to address the >>>>>>>>>>>>>> resolution within >>>>>>>>>>>>>> the context of the sql text, not to address catalog/naming >>>>>>>>>>>>>> inconsistencies. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I feel like we're trying to adapt the original intent to >>>>>>>>>>>>>> address the catalog naming/configuration and would argue that we >>>>>>>>>>>>>> shouldn't >>>>>>>>>>>>>> attempt to do that as part of the spec. Inconsistently named >>>>>>>>>>>>>> catalogs are >>>>>>>>>>>>>> a reality, but we should consider that a >>>>>>>>>>>>>> configuration/environmental issue, >>>>>>>>>>>>>> not something to solve for in the spec. >>>>>>>>>>>>>> >>>>>>>>>>>>>> We should support and advocate for consistency in catalog >>>>>>>>>>>>>> naming and define the spec along those lines. The fact is that >>>>>>>>>>>>>> with all of >>>>>>>>>>>>>> the recent work that's gone into making catalogs pluggable, it >>>>>>>>>>>>>> makes more >>>>>>>>>>>>>> sense to just register catalog configuration with consistent >>>>>>>>>>>>>> names (even if >>>>>>>>>>>>>> you have to duplicate the configuration for supporting existing >>>>>>>>>>>>>> readers/writers). I think it's better to provide a path toward >>>>>>>>>>>>>> consistency >>>>>>>>>>>>>> than to normalize complicated schemes to workaround the issues >>>>>>>>>>>>>> caused by >>>>>>>>>>>>>> environmental/configuration inconsistencies. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If the goal is to create clever ways to hack the late binding >>>>>>>>>>>>>> resolution to swap in different catalogs or make references >>>>>>>>>>>>>> contextual, I >>>>>>>>>>>>>> feel like that is something we should strongly discourage as it >>>>>>>>>>>>>> leads to >>>>>>>>>>>>>> confusion about what is resolved as part of the query. >>>>>>>>>>>>>> >>>>>>>>>>>>>> At this point, I don't see a good argument to add >>>>>>>>>>>>>> additional configuration or change the resolution behaviors. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Dan >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Apr 28, 2025 at 12:40 AM Jan Kaul >>>>>>>>>>>>>> <jank...@mailbox.org.invalid> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think the intention with the "default-catalog" was that >>>>>>>>>>>>>>> every query engine uses it to store its session default catalog >>>>>>>>>>>>>>> at the time >>>>>>>>>>>>>>> of creating the view. This way the view could be reused in >>>>>>>>>>>>>>> another session. >>>>>>>>>>>>>>> The idea was not to introduce an additional SQL syntax to set >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> default-catalog. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Generally we have different environments we want to support >>>>>>>>>>>>>>> with the view spec: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. Consistent catalog naming >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> When the environment supports it, using consistent catalog >>>>>>>>>>>>>>> names can have a great benefit for multi-catalog, multi-engine >>>>>>>>>>>>>>> setups. With >>>>>>>>>>>>>>> consistent catalog names, using the "default-catalog" field >>>>>>>>>>>>>>> works without >>>>>>>>>>>>>>> any issues. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2. Inconsistent catalog naming >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This can be the case when different query engines refer to >>>>>>>>>>>>>>> the same physical catalog by different names. This often >>>>>>>>>>>>>>> happens because >>>>>>>>>>>>>>> different query engines use different strategies to setup the >>>>>>>>>>>>>>> catalogs. If >>>>>>>>>>>>>>> catalogs have inconsistent naming, using the "default-catalog" >>>>>>>>>>>>>>> field does >>>>>>>>>>>>>>> not work because it is not guaranteed that the catalog name can >>>>>>>>>>>>>>> be resolved >>>>>>>>>>>>>>> with another engine. Using the "view catalog" as a fallback is >>>>>>>>>>>>>>> a better >>>>>>>>>>>>>>> solution for this use case, as it avoids catalog names >>>>>>>>>>>>>>> altogether. It is >>>>>>>>>>>>>>> however limited to table references in the same catalog. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> What do you think of introducing a view property that >>>>>>>>>>>>>>> specifies if the "default-catalog" or the "view catalog" should >>>>>>>>>>>>>>> be used? >>>>>>>>>>>>>>> This way, you could use the "default-catalog" in environments >>>>>>>>>>>>>>> where you can >>>>>>>>>>>>>>> guarantee consistent naming, but you would be able to directly >>>>>>>>>>>>>>> fallback to >>>>>>>>>>>>>>> the "view-catalog" when you don't have consistent naming. The >>>>>>>>>>>>>>> query engines >>>>>>>>>>>>>>> could set the default for this view property at creation time. >>>>>>>>>>>>>>> Spark for >>>>>>>>>>>>>>> example could set it to automatically use the "view catalog". >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Jan >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 4/26/25 05:33, Walaa Eldin Moustafa wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> To help folks catch up on the latest discussions and >>>>>>>>>>>>>>> interpretation of the spec, I have summarized everything we >>>>>>>>>>>>>>> discussed so >>>>>>>>>>>>>>> far at the top of the proposal document (here >>>>>>>>>>>>>>> <https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>). >>>>>>>>>>>>>>> I have slightly updated the proposal to be in sync with the new >>>>>>>>>>>>>>> interpretation to avoid confusion. In summary: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> * Remove default-catalog and default-namespace fields from >>>>>>>>>>>>>>> the view spec completely. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> * Hence, we do not attempt to define separate view-level >>>>>>>>>>>>>>> default catalogs or namespaces. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Instead: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> * If a table identifier inside a view lacks a catalog >>>>>>>>>>>>>>> qualifier, engines should resolve it using the current engine >>>>>>>>>>>>>>> catalog at >>>>>>>>>>>>>>> query time. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> * Reference table identifiers in the metadata exactly as >>>>>>>>>>>>>>> they appear in the view SQL text. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> * If an identifier lacks the catalog part at creation, it >>>>>>>>>>>>>>> should still lack a catalog in the stored metadata. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> * Store UUIDs alongside table identifiers whenever possible. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Walaa. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa < >>>>>>>>>>>>>>> wa.moust...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for the contribution Benny! +1 to the confusion the >>>>>>>>>>>>>>>> fallback creates. Also just to be clear, at this point and >>>>>>>>>>>>>>>> after clarifying >>>>>>>>>>>>>>>> the current spec intentions, I am convinced that we should >>>>>>>>>>>>>>>> remove the >>>>>>>>>>>>>>>> default catalog and default namespace fields altogether. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Walaa. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 5:13 PM Benny Chow < >>>>>>>>>>>>>>>> btc...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'd like to contribute my opinions on this: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> - I don't particularly like the current behavior of >>>>>>>>>>>>>>>>> "default to the view's catalog when default-catalog is not >>>>>>>>>>>>>>>>> set". >>>>>>>>>>>>>>>>> Fundamentally, I believe the intent of default-catalog and >>>>>>>>>>>>>>>>> default-namespace is there to help users write more concise >>>>>>>>>>>>>>>>> SQL. >>>>>>>>>>>>>>>>> - spark session catalog is engine specific and I don't >>>>>>>>>>>>>>>>> think we should design something that says first use this >>>>>>>>>>>>>>>>> catalog, then >>>>>>>>>>>>>>>>> that catalog.. or that catalog. For example, resolving >>>>>>>>>>>>>>>>> identifiers using >>>>>>>>>>>>>>>>> default-catalog -> view's catalog -> session catalog is not >>>>>>>>>>>>>>>>> good. >>>>>>>>>>>>>>>>> - We gotta support non-Iceberg tables otherwise I see no >>>>>>>>>>>>>>>>> value in putting views in the catalog to share with other >>>>>>>>>>>>>>>>> engines >>>>>>>>>>>>>>>>> - Interoperability between different engine types is very >>>>>>>>>>>>>>>>> hard due to dialect issues... so I think we should focus on >>>>>>>>>>>>>>>>> supporting >>>>>>>>>>>>>>>>> different clusters of the same engine type on a shared >>>>>>>>>>>>>>>>> catalog. For >>>>>>>>>>>>>>>>> example, AI and BI clusters on Spark sharing the same views >>>>>>>>>>>>>>>>> in a REST >>>>>>>>>>>>>>>>> catalog. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Coincidentally, I think the ultimate solution is along the >>>>>>>>>>>>>>>>> lines of something Russell proposed last year: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We've been looking at this interoperable identifier >>>>>>>>>>>>>>>>> problem through the lens of catalog resolution but maybe the >>>>>>>>>>>>>>>>> right approach >>>>>>>>>>>>>>>>> is really about templating. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I would extend Russell's idea to allow identifiers in a >>>>>>>>>>>>>>>>> view to span catalogs to support non-Iceberg tables. Also, >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> default-catalog property could be templated as well. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thoughts? >>>>>>>>>>>>>>>>> Benny >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa < >>>>>>>>>>>>>>>>> wa.moust...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks Steven! How do you recommend making Spark >>>>>>>>>>>>>>>>>> implementation conform to the spec? Do we need Spark SQL >>>>>>>>>>>>>>>>>> extensions and/or >>>>>>>>>>>>>>>>>> Spark catalog APIs for that? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> How do you recommend reconciling the inconsistencies I >>>>>>>>>>>>>>>>>> shared regarding many resolution methods not consistently >>>>>>>>>>>>>>>>>> being followed in >>>>>>>>>>>>>>>>>> different scenarios (view vs child table resolution, query >>>>>>>>>>>>>>>>>> vs view >>>>>>>>>>>>>>>>>> resolution)? Note these occur when the default catalog is >>>>>>>>>>>>>>>>>> set to a non-null >>>>>>>>>>>>>>>>>> value. If it helps, I can share concrete examples. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> Walaa. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 3:52 PM Steven Wu < >>>>>>>>>>>>>>>>>> stevenz...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The core issue is on the fall back behavior when >>>>>>>>>>>>>>>>>>> `default-catalog` is >>>>>>>>>>>>>>>>>>> not defined. Current view spec says the fallback should >>>>>>>>>>>>>>>>>>> be the catalog >>>>>>>>>>>>>>>>>>> where the view is defined. It doesn't really matter what >>>>>>>>>>>>>>>>>>> the catalog >>>>>>>>>>>>>>>>>>> is named (catalogX) by the read engine. >>>>>>>>>>>>>>>>>>> - If a view refers to the tables in the same catalog, >>>>>>>>>>>>>>>>>>> this is a >>>>>>>>>>>>>>>>>>> non-ambiguous and reasonable fallback behavior. >>>>>>>>>>>>>>>>>>> - If a view refers to tables from another catalog, >>>>>>>>>>>>>>>>>>> catalog names >>>>>>>>>>>>>>>>>>> should be included in the reference name already. So no >>>>>>>>>>>>>>>>>>> ambiguity >>>>>>>>>>>>>>>>>>> there either. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Potential inconsistent naming of catalog is a separate >>>>>>>>>>>>>>>>>>> problem, which >>>>>>>>>>>>>>>>>>> Iceberg view spec probably cannot solve. We can only >>>>>>>>>>>>>>>>>>> recommend that >>>>>>>>>>>>>>>>>>> catalog should be named consistently across usage for >>>>>>>>>>>>>>>>>>> better >>>>>>>>>>>>>>>>>>> interoperability on name references. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> This proposal is to change the fallback behavior to >>>>>>>>>>>>>>>>>>> engine's session >>>>>>>>>>>>>>>>>>> default catalog. I am not sure it is better than the >>>>>>>>>>>>>>>>>>> current fallback >>>>>>>>>>>>>>>>>>> behavior. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> > Today’s Spark behavior explicitly differs from this >>>>>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation >>>>>>>>>>>>>>>>>>> using the >>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I would argue that is a Spark implementation issue for >>>>>>>>>>>>>>>>>>> not conforming >>>>>>>>>>>>>>>>>>> to the spec. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa >>>>>>>>>>>>>>>>>>> <wa.moust...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Hi Jan, >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Thanks again for continuing the discussion. I want to >>>>>>>>>>>>>>>>>>> highlight a few fundamental issues around the >>>>>>>>>>>>>>>>>>> interpretation of >>>>>>>>>>>>>>>>>>> default-catalog: >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Here is the real catch: >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > * default-catalog cannot logically be defined at view >>>>>>>>>>>>>>>>>>> creation time. It would be circular: the view needs to >>>>>>>>>>>>>>>>>>> exist before its >>>>>>>>>>>>>>>>>>> metadata (and hence default-catalog) can exist. This is >>>>>>>>>>>>>>>>>>> visible in Spark’s >>>>>>>>>>>>>>>>>>> implementation, where `default-catalog` is not used. >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > * Introducing a creation-time default-catalog setting >>>>>>>>>>>>>>>>>>> would require extending SQL syntax and engine APIs to >>>>>>>>>>>>>>>>>>> promote it to a >>>>>>>>>>>>>>>>>>> first-class view concept. This would be intrusive, >>>>>>>>>>>>>>>>>>> non-intuitive, and >>>>>>>>>>>>>>>>>>> realistically very difficult to standardize across engines. >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > * Today’s Spark behavior explicitly differs from this >>>>>>>>>>>>>>>>>>> idea. Spark resolves table identifiers during view creation >>>>>>>>>>>>>>>>>>> using the >>>>>>>>>>>>>>>>>>> session’s default catalog, not a supplied `default-catalog`. >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > * Hypothetically even if we patched in a creation-time >>>>>>>>>>>>>>>>>>> default-catalog, it would create an inconsistent binding >>>>>>>>>>>>>>>>>>> model between >>>>>>>>>>>>>>>>>>> tables vs views (early vs late), and between tables in >>>>>>>>>>>>>>>>>>> views and in queries >>>>>>>>>>>>>>>>>>> (again early vs late). For example, views and tables in >>>>>>>>>>>>>>>>>>> queries can >>>>>>>>>>>>>>>>>>> withstand default catalog renames, but tables cannot when >>>>>>>>>>>>>>>>>>> they are used >>>>>>>>>>>>>>>>>>> inside views -- it even applies to views inside views, >>>>>>>>>>>>>>>>>>> which makes this >>>>>>>>>>>>>>>>>>> very hard to reason about considering nesting. >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > Thanks, >>>>>>>>>>>>>>>>>>> > Walaa >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul >>>>>>>>>>>>>>>>>>> <jank...@mailbox.org.invalid> >>>>>>>>>>>>>>>>>>> <jank...@mailbox.org.invalid> wrote: >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> @Walaa: >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> I would argue that when you run a CREATE VIEW >>>>>>>>>>>>>>>>>>> statement the query engine knowns which catalog the view is >>>>>>>>>>>>>>>>>>> being created >>>>>>>>>>>>>>>>>>> in. So even though we typically use late binding to resolve >>>>>>>>>>>>>>>>>>> the view >>>>>>>>>>>>>>>>>>> catalog at query time, it can also be used at creation time. >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> The query engine would need to keep track of the >>>>>>>>>>>>>>>>>>> "view catalog" where the view is going to be created in. It >>>>>>>>>>>>>>>>>>> can use that >>>>>>>>>>>>>>>>>>> catalog to resolve partial table identifiers if >>>>>>>>>>>>>>>>>>> "default-catalog" is not >>>>>>>>>>>>>>>>>>> set. >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> It can lead to some unintuitive behavior, where >>>>>>>>>>>>>>>>>>> partial identifiers in the view query resolve to a >>>>>>>>>>>>>>>>>>> different catalog >>>>>>>>>>>>>>>>>>> compared to using them outside of a view. >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> CREATE VIEW catalogA.sales.monthly_orders AS SELECT * >>>>>>>>>>>>>>>>>>> from sales.orders; >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> If the session default catalog is not "catalogA", the >>>>>>>>>>>>>>>>>>> "sales.orders" in the view query would not be the same as >>>>>>>>>>>>>>>>>>> just referencing >>>>>>>>>>>>>>>>>>> "sales.orders" in a normal SQL statement. This is because >>>>>>>>>>>>>>>>>>> without a >>>>>>>>>>>>>>>>>>> "default-catalog", the catalog name of "sales.orders" would >>>>>>>>>>>>>>>>>>> default to >>>>>>>>>>>>>>>>>>> "catalogA", which is the view's catalog. >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> Thanks, >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> Jan >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> On 4/25/25 04:05, Manu Zhang wrote: >>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>> >>> For example, if we want to validate that the tables >>>>>>>>>>>>>>>>>>> referenced in the view exist, how can we do that when >>>>>>>>>>>>>>>>>>> default-catalog isn't >>>>>>>>>>>>>>>>>>> defined, since the view hasn't been created or loaded yet? >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> I don't think this is related to view spec. How do we >>>>>>>>>>>>>>>>>>> validate that a table exists without a default catalog, or >>>>>>>>>>>>>>>>>>> do we always use >>>>>>>>>>>>>>>>>>> the current session catalog? >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> Thanks, >>>>>>>>>>>>>>>>>>> >> Manu >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin Moustafa < >>>>>>>>>>>>>>>>>>> wa.moust...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>> >>> Hi Jan, >>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>> >>> I think we still share the same understanding. Just >>>>>>>>>>>>>>>>>>> to clarify: when I referred to late binding as “similar” to >>>>>>>>>>>>>>>>>>> the proposal, I >>>>>>>>>>>>>>>>>>> was acknowledging the distinction between view-level and >>>>>>>>>>>>>>>>>>> table-level >>>>>>>>>>>>>>>>>>> resolution. But as you noted, both follow a late binding >>>>>>>>>>>>>>>>>>> model. >>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>> >>> That said, this still raises an interesting question >>>>>>>>>>>>>>>>>>> and a potential gap: if default-catalog is only defined at >>>>>>>>>>>>>>>>>>> query time, how >>>>>>>>>>>>>>>>>>> should resolution work during view creation? For example, >>>>>>>>>>>>>>>>>>> if we want to >>>>>>>>>>>>>>>>>>> validate that the tables referenced in the view exist, how >>>>>>>>>>>>>>>>>>> can we do that >>>>>>>>>>>>>>>>>>> when default-catalog isn't defined, since the view hasn't >>>>>>>>>>>>>>>>>>> been created or >>>>>>>>>>>>>>>>>>> loaded yet? >>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>> >>> Thanks, >>>>>>>>>>>>>>>>>>> >>> Walaa. >>>>>>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>>>>>> >>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul >>>>>>>>>>>>>>>>>>> <jank...@mailbox.org.invalid> >>>>>>>>>>>>>>>>>>> <jank...@mailbox.org.invalid> wrote: >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> Yes, I have the same understanding. The view >>>>>>>>>>>>>>>>>>> catalog is resolved at query time. >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> As you mentioned before, it's good to distinguish >>>>>>>>>>>>>>>>>>> between the physical catalog and it's reference used in SQL >>>>>>>>>>>>>>>>>>> statements. The >>>>>>>>>>>>>>>>>>> important part is that the physical catalog of the view and >>>>>>>>>>>>>>>>>>> the tables >>>>>>>>>>>>>>>>>>> referenced in it's definition stay consistent. You could >>>>>>>>>>>>>>>>>>> create a view in a >>>>>>>>>>>>>>>>>>> given physical catalog by referring to it as "catalogA", as >>>>>>>>>>>>>>>>>>> in your first >>>>>>>>>>>>>>>>>>> point. If you then, given a different setup, refer to the >>>>>>>>>>>>>>>>>>> same physical >>>>>>>>>>>>>>>>>>> catalog as "catalogB" in another session/environment, the >>>>>>>>>>>>>>>>>>> behavior should >>>>>>>>>>>>>>>>>>> still work. >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> I would however rephrase your last point. Late >>>>>>>>>>>>>>>>>>> binding applies to the view catalog name and by extension >>>>>>>>>>>>>>>>>>> to all partial >>>>>>>>>>>>>>>>>>> table references when no "default-catalog" is present. >>>>>>>>>>>>>>>>>>> Resolving the view >>>>>>>>>>>>>>>>>>> catalog name at query time is not opposed to storing the >>>>>>>>>>>>>>>>>>> view metadata in a >>>>>>>>>>>>>>>>>>> catalog. >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> Or maybe I don't entirely understand what you mean. >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> Thanks >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> Jan >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote: >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> Hi Jan, >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> > The view is executed when it's being referenced >>>>>>>>>>>>>>>>>>> in a SQL statement. That statement contains the information >>>>>>>>>>>>>>>>>>> for the query >>>>>>>>>>>>>>>>>>> engine to resolve the catalog of the view. >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> If I’m understanding correctly, that means: >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> * If the view is queried as SELECT * FROM >>>>>>>>>>>>>>>>>>> catalogA.namespace.view, then catalogA is considered the >>>>>>>>>>>>>>>>>>> view’s catalog. >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> * If the same view is later queried as SELECT * >>>>>>>>>>>>>>>>>>> FROM catalogB.namespace.view (after renaming catalogA to >>>>>>>>>>>>>>>>>>> catalogB, and >>>>>>>>>>>>>>>>>>> keeping everything else the same), then catalogB becomes >>>>>>>>>>>>>>>>>>> the view’s catalog. >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> Is that interpretation correct? If so, it sounds to >>>>>>>>>>>>>>>>>>> me like the catalog is resolved at query time, based on how >>>>>>>>>>>>>>>>>>> the view is >>>>>>>>>>>>>>>>>>> referenced, not from any stored metadata. That would imply >>>>>>>>>>>>>>>>>>> some sort of a >>>>>>>>>>>>>>>>>>> late binding behavior (similar to the proposal), as opposed >>>>>>>>>>>>>>>>>>> to using some >>>>>>>>>>>>>>>>>>> catalog that "stores" the view definition. >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> Thanks, >>>>>>>>>>>>>>>>>>> >>>> Walaa >>>>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>>>> >>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul >>>>>>>>>>>>>>>>>>> <jank...@mailbox.org.invalid> >>>>>>>>>>>>>>>>>>> <jank...@mailbox.org.invalid> wrote: >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> Hi Walaa, >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> Thanks for clarifying the aspects of >>>>>>>>>>>>>>>>>>> non-determinism. Let me try to address your questions. >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> 1. This is my interpretation of the current spec: >>>>>>>>>>>>>>>>>>> The view is executed when it's being referenced in a SQL >>>>>>>>>>>>>>>>>>> statement. That >>>>>>>>>>>>>>>>>>> statement contains the information for the query engine to >>>>>>>>>>>>>>>>>>> resolve the >>>>>>>>>>>>>>>>>>> catalog of the view. The query engine then uses that >>>>>>>>>>>>>>>>>>> information to fetch >>>>>>>>>>>>>>>>>>> the view metadata from the catalog. It also needs to >>>>>>>>>>>>>>>>>>> temporarily keep track >>>>>>>>>>>>>>>>>>> of which catalog it used to fetch the view metadata. It can >>>>>>>>>>>>>>>>>>> then use that >>>>>>>>>>>>>>>>>>> information to resolve the table references in the views >>>>>>>>>>>>>>>>>>> SQL definition in >>>>>>>>>>>>>>>>>>> case no default catalog is specified. >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> 2. The important part is that the catalog can be >>>>>>>>>>>>>>>>>>> referenced at execution time. As long as that's the case I >>>>>>>>>>>>>>>>>>> would assume the >>>>>>>>>>>>>>>>>>> view can be created in any catalog. >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> I think your point is really valuable because the >>>>>>>>>>>>>>>>>>> current specification can lead to some unintuitive >>>>>>>>>>>>>>>>>>> behavior. For example >>>>>>>>>>>>>>>>>>> for the following statement: >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> CREATE VIEW catalogA.sales.monthly_orders AS >>>>>>>>>>>>>>>>>>> SELECT * from sales.orders; >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> If the session default catalog is not "catalogA", >>>>>>>>>>>>>>>>>>> the "sales.orders" in the view query would not be the same >>>>>>>>>>>>>>>>>>> as just >>>>>>>>>>>>>>>>>>> referencing "sales.orders" in a normal SQL statement. This >>>>>>>>>>>>>>>>>>> is because >>>>>>>>>>>>>>>>>>> without a "default-catalog", the catalog name of >>>>>>>>>>>>>>>>>>> "sales.orders" would >>>>>>>>>>>>>>>>>>> default to "catalogA". >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> However, I like the current design of the view >>>>>>>>>>>>>>>>>>> spec, because it has the "closure" property. Because of the >>>>>>>>>>>>>>>>>>> fact that the >>>>>>>>>>>>>>>>>>> "view catalog" has to be known when executing a view, all >>>>>>>>>>>>>>>>>>> the information >>>>>>>>>>>>>>>>>>> required to resolve the table identifiers is contained in >>>>>>>>>>>>>>>>>>> the view metadata >>>>>>>>>>>>>>>>>>> (and the "view catalog"). I think that if you make the >>>>>>>>>>>>>>>>>>> identifier >>>>>>>>>>>>>>>>>>> resolution dependent on external parameters, it hinders >>>>>>>>>>>>>>>>>>> portability. >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> Thanks, >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> Jan >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote: >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> Hi Jan, >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> Thanks for the thoughtful feedback. >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> I think it’s important we clarify a key point >>>>>>>>>>>>>>>>>>> before going deeper: >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> Non-determinism is not caused by session fallback >>>>>>>>>>>>>>>>>>> behavior—it’s a fundamental limitation of using table >>>>>>>>>>>>>>>>>>> identifiers alone, >>>>>>>>>>>>>>>>>>> regardless of whether we use the current rule, the proposed >>>>>>>>>>>>>>>>>>> fallback to the >>>>>>>>>>>>>>>>>>> session’s default catalog, or even early vs. late binding. >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> The same fully qualified identifier (e.g., >>>>>>>>>>>>>>>>>>> catalogA.namespace.table) can resolve to different objects >>>>>>>>>>>>>>>>>>> depending solely >>>>>>>>>>>>>>>>>>> on engine-specific routing logic or catalog aliases. So >>>>>>>>>>>>>>>>>>> determinism isn’t >>>>>>>>>>>>>>>>>>> guaranteed just because an identifier is "fully qualified." >>>>>>>>>>>>>>>>>>> The only >>>>>>>>>>>>>>>>>>> reliable anchor for identity is the UUID. That’s why the >>>>>>>>>>>>>>>>>>> proposed use of >>>>>>>>>>>>>>>>>>> UUIDs is not just a hardening strategy. It’s the actual fix >>>>>>>>>>>>>>>>>>> for correctness. >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> To move the conversation forward, could you help >>>>>>>>>>>>>>>>>>> clarify two things in the context of the current spec: >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> * Where in the metadata is the “view catalog” >>>>>>>>>>>>>>>>>>> stored, so that an engine knows to fall back to it if >>>>>>>>>>>>>>>>>>> default-catalog is >>>>>>>>>>>>>>>>>>> null? >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> * Are we even allowed to create views in the >>>>>>>>>>>>>>>>>>> session's default catalog (i.e., without specifying a >>>>>>>>>>>>>>>>>>> catalog) in the >>>>>>>>>>>>>>>>>>> current Iceberg spec? >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> These questions are important because if we can’t >>>>>>>>>>>>>>>>>>> unambiguously recover the "view catalog" from metadata, >>>>>>>>>>>>>>>>>>> then defaulting to >>>>>>>>>>>>>>>>>>> it is problematic. And if views can't be created in the >>>>>>>>>>>>>>>>>>> default catalog, >>>>>>>>>>>>>>>>>>> then the fallback rule doesn’t generalize. >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> Thanks, >>>>>>>>>>>>>>>>>>> >>>>> Walaa. >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>>>>>> >>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul >>>>>>>>>>>>>>>>>>> <jank...@mailbox.org.invalid> >>>>>>>>>>>>>>>>>>> <jank...@mailbox.org.invalid> wrote: >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> Hi Walaa, >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> thank you for your proposal. If I understood >>>>>>>>>>>>>>>>>>> correctly, you proposal is composed of three parts: >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> - session default catalog as fallback for >>>>>>>>>>>>>>>>>>> "default-catalog" >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> - session default namespace as fallback for >>>>>>>>>>>>>>>>>>> "default-namepace" >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> - Late binding + UUID validation >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> I have some comments regarding these points. >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> 1. Session default catalog as fallback for >>>>>>>>>>>>>>>>>>> "default-catalog" >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> Introducing a behavior that depends on the >>>>>>>>>>>>>>>>>>> current session setup is in my opinion the definition of >>>>>>>>>>>>>>>>>>> "non-determinism". >>>>>>>>>>>>>>>>>>> You could be running the same query-engine and >>>>>>>>>>>>>>>>>>> catalog-setup on different >>>>>>>>>>>>>>>>>>> days, with different default session catalogs (which is >>>>>>>>>>>>>>>>>>> rather common), and >>>>>>>>>>>>>>>>>>> would be getting different results. >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> Whereas with the current behavior, the view >>>>>>>>>>>>>>>>>>> always produces the same results. The current behavior has >>>>>>>>>>>>>>>>>>> some rough edges >>>>>>>>>>>>>>>>>>> in very niche use cases but I think is solid for most uses >>>>>>>>>>>>>>>>>>> cases. >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> 2. Session default namespace as fallback for >>>>>>>>>>>>>>>>>>> "default-namespace" >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> Similar to the above. >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> 3. Late binding + UUID validation >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> If I understand it correctly, the current >>>>>>>>>>>>>>>>>>> implementation already uses late binding. >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> Generally, having UUID validation makes the setup >>>>>>>>>>>>>>>>>>> more robust. Which is great. However, having UUID >>>>>>>>>>>>>>>>>>> validation still requires >>>>>>>>>>>>>>>>>>> us to have a portable table identifier specification. Even >>>>>>>>>>>>>>>>>>> if we have the >>>>>>>>>>>>>>>>>>> UUIDs of the referenced tables from the view, there simply >>>>>>>>>>>>>>>>>>> isn't an >>>>>>>>>>>>>>>>>>> interface that let's us use those UUIDs. The catalog >>>>>>>>>>>>>>>>>>> interface is defined >>>>>>>>>>>>>>>>>>> in terms of table identifiers. >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> So we always require a working catalog setup and >>>>>>>>>>>>>>>>>>> suiting table identifiers to obtain the table metadata. We >>>>>>>>>>>>>>>>>>> can use the >>>>>>>>>>>>>>>>>>> UUIDs to verify if we loaded the correct table. But this >>>>>>>>>>>>>>>>>>> can only be done >>>>>>>>>>>>>>>>>>> after we used some identifier. Which means there is no way >>>>>>>>>>>>>>>>>>> of using UUIDs >>>>>>>>>>>>>>>>>>> without a functioning catalog/identifier setup. >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> In conclusion, I prefer the current behavior for >>>>>>>>>>>>>>>>>>> "default-catalog" because it is more deterministic in my >>>>>>>>>>>>>>>>>>> opinion. And I >>>>>>>>>>>>>>>>>>> think the current spec does a good job for multi-engine >>>>>>>>>>>>>>>>>>> table identifier >>>>>>>>>>>>>>>>>>> resolution. I see the UUID validation more of an additional >>>>>>>>>>>>>>>>>>> hardening >>>>>>>>>>>>>>>>>>> strategy. >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> Thanks >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> Jan >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote: >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> Thanks Renjie! >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> The existing spec has some guidance on resolving >>>>>>>>>>>>>>>>>>> catalogs on the fly already (to address the case of view >>>>>>>>>>>>>>>>>>> text with table >>>>>>>>>>>>>>>>>>> identifiers missing the catalog part). The guidance is to >>>>>>>>>>>>>>>>>>> use the catalog >>>>>>>>>>>>>>>>>>> where the view is stored. But I find this rule hard to >>>>>>>>>>>>>>>>>>> interpret or use. >>>>>>>>>>>>>>>>>>> The catalog itself is a logical construct—such as a >>>>>>>>>>>>>>>>>>> federated catalog that >>>>>>>>>>>>>>>>>>> delegates to multiple physical backends (e.g., HMS and >>>>>>>>>>>>>>>>>>> REST). In such >>>>>>>>>>>>>>>>>>> cases, the catalog (e.g., `my_catalog` in >>>>>>>>>>>>>>>>>>> `my_catalog.namespace1.table1`) >>>>>>>>>>>>>>>>>>> doesn’t physically store the tables; it only routes >>>>>>>>>>>>>>>>>>> requests to underlying >>>>>>>>>>>>>>>>>>> stores. Therefore, defaulting identifier resolution based >>>>>>>>>>>>>>>>>>> on the catalog >>>>>>>>>>>>>>>>>>> where the view is "stored" doesn’t align with how catalogs >>>>>>>>>>>>>>>>>>> actually behave >>>>>>>>>>>>>>>>>>> in practice. >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> Thanks, >>>>>>>>>>>>>>>>>>> >>>>>> Walaa. >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu < >>>>>>>>>>>>>>>>>>> liurenjie2...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> Hi, Walaa: >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> Thanks for the proposal. >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> I've reviewed the doc, but in general I have >>>>>>>>>>>>>>>>>>> some concerns with resolving catalog names on the fly with >>>>>>>>>>>>>>>>>>> query engine >>>>>>>>>>>>>>>>>>> defined catalog names. This introduces some flexibility at >>>>>>>>>>>>>>>>>>> first glance, >>>>>>>>>>>>>>>>>>> but also makes misconfiguration difficult to explain. >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> But I agree with one part that we should store >>>>>>>>>>>>>>>>>>> resolved table uuid in view metadata, as table/view >>>>>>>>>>>>>>>>>>> renaming may introduce >>>>>>>>>>>>>>>>>>> errors that's difficult to understand for user. >>>>>>>>>>>>>>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin >>>>>>>>>>>>>>>>>>> Moustafa <wa.moust...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> Hi Everyone, >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> Looking forward to keeping up the momentum and >>>>>>>>>>>>>>>>>>> closing out the MV spec as well. I’m hoping we can proceed >>>>>>>>>>>>>>>>>>> to a vote next >>>>>>>>>>>>>>>>>>> week. >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> Here is a summary in case that helps. The >>>>>>>>>>>>>>>>>>> proposal outlines a strategy for handling table identifiers >>>>>>>>>>>>>>>>>>> in Iceberg view >>>>>>>>>>>>>>>>>>> metadata, with the goal of ensuring correctness, >>>>>>>>>>>>>>>>>>> portability, and engine >>>>>>>>>>>>>>>>>>> compatibility. It recommends resolving table identifiers at >>>>>>>>>>>>>>>>>>> read time (late >>>>>>>>>>>>>>>>>>> binding) rather than creation time, and introduces >>>>>>>>>>>>>>>>>>> UUID-based validation to >>>>>>>>>>>>>>>>>>> maintain identity guarantees across engines, or sessions. >>>>>>>>>>>>>>>>>>> It also revises >>>>>>>>>>>>>>>>>>> how default-catalog and default-namespace are handled >>>>>>>>>>>>>>>>>>> (defaulting both to >>>>>>>>>>>>>>>>>>> the session context if not explicitly set) to better align >>>>>>>>>>>>>>>>>>> with engine >>>>>>>>>>>>>>>>>>> behavior and improve cross-engine interoperability. >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> Please let me know your thoughts. >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> >>>>>>>> Walaa. >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin >>>>>>>>>>>>>>>>>>> Moustafa <wa.moust...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks Eduard and Sung! I have addressed the >>>>>>>>>>>>>>>>>>> comments. >>>>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>> One key point to keep in mind is that catalog >>>>>>>>>>>>>>>>>>> names in the spec refer to logical catalogs—i.e., the first >>>>>>>>>>>>>>>>>>> part of a >>>>>>>>>>>>>>>>>>> three-part identifier. These correspond to Spark's >>>>>>>>>>>>>>>>>>> DataSourceV2 catalogs, >>>>>>>>>>>>>>>>>>> Trino connectors, and similar constructs. This is a level >>>>>>>>>>>>>>>>>>> of abstraction >>>>>>>>>>>>>>>>>>> above physical catalogs, which are not referenced or used >>>>>>>>>>>>>>>>>>> in the view spec. >>>>>>>>>>>>>>>>>>> The reason is that table identifiers in the view >>>>>>>>>>>>>>>>>>> definition/text itself >>>>>>>>>>>>>>>>>>> refer to logical catalogs, not physical ones (since they >>>>>>>>>>>>>>>>>>> interface directly >>>>>>>>>>>>>>>>>>> with the engine and not a specific metastore). >>>>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> >>>>>>>>> Walaa. >>>>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun < >>>>>>>>>>>>>>>>>>> sungwy...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>> Thank you Walaa for the proposal. I think >>>>>>>>>>>>>>>>>>> view portability is a very important topic for us to >>>>>>>>>>>>>>>>>>> continue discussing as >>>>>>>>>>>>>>>>>>> it relies on many assumptions within the data ecosystem for >>>>>>>>>>>>>>>>>>> it to function >>>>>>>>>>>>>>>>>>> like you've highlighted well in the document. >>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>> I've added a few comments around how this may >>>>>>>>>>>>>>>>>>> impact the permission questions the engines will be asking, >>>>>>>>>>>>>>>>>>> and whether >>>>>>>>>>>>>>>>>>> that is the desired behavior. >>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>> Sung >>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard >>>>>>>>>>>>>>>>>>> Tudenhöfner <etudenhoef...@apache.org> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Thanks Walaa for tackling this problem. I've >>>>>>>>>>>>>>>>>>> added a few comments to get a better understanding of how >>>>>>>>>>>>>>>>>>> this will look >>>>>>>>>>>>>>>>>>> like in the actual implementation. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>> Eduard >>>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa Eldin >>>>>>>>>>>>>>>>>>> Moustafa <wa.moust...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Hi Everyone, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Starting this thread to resume our >>>>>>>>>>>>>>>>>>> discussion on how to reference table identifiers from >>>>>>>>>>>>>>>>>>> Iceberg metadata, a >>>>>>>>>>>>>>>>>>> key aspect of the view specification, particularly in >>>>>>>>>>>>>>>>>>> relation to the MV >>>>>>>>>>>>>>>>>>> (materialized view) extensions. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> I had the chance to speak offline with a >>>>>>>>>>>>>>>>>>> few community members to better understand how the current >>>>>>>>>>>>>>>>>>> spec is being >>>>>>>>>>>>>>>>>>> interpreted. Those conversations served as inputs to a new >>>>>>>>>>>>>>>>>>> proposal on how >>>>>>>>>>>>>>>>>>> table identifier references could be represented in >>>>>>>>>>>>>>>>>>> metadata. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> You can find the proposal here [1]. I look >>>>>>>>>>>>>>>>>>> forward to your feedback and working together to move this >>>>>>>>>>>>>>>>>>> forward so we >>>>>>>>>>>>>>>>>>> can finalize the MV spec as well. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> Walaa. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>
