Re: Overwrite old properties on table replace with REST catalog

Hi Vladimir, What you said resonated with me: subtle differences will be present across catalog types albeit our strong will of making it consistent. And I don't have an issue with "flexible behavior". We don't have to enforce perfect consistency of catalog behavior, and I would judge case by case

Re: Overwrite old properties on table replace with REST catalog

Hi, Haizhou. Thanks for bringing this up. Yes, this problem appears similar to what I explained but with a slightly different manifestation. What is funny is that you observed it when doing REST integration tests for Spark, and I was doing the same for Trino (the fully-fledged integration tests ar

Re: Overwrite old properties on table replace with REST catalog

Hello Vladimir, I want to raise that we've been observing similar behavior differences regarding CREATE OR REPLACE between Hive/Hadoop catalog and REST catalog at here: https://github.com/apache/iceberg/issues/11109 The context: Iceberg spark integration test has traditionally only included tests

Re: Overwrite old properties on table replace with REST catalog

Hi, Sure, will do. *Vladimir Ozerov* Founder querifylabs.com Ср, 23 окт. 2024 г. в 08:50, Jean-Baptiste Onofré : > I second Ryan here, it would be great to clarify in the > "implementation notes" section. > > Thanks ! > Regards > JB > > On Wed, Oct 23, 2024 at 1:10 AM rdb...@gmail.com wrote:

Re: Overwrite old properties on table replace with REST catalog

I second Ryan here, it would be great to clarify in the "implementation notes" section. Thanks ! Regards JB On Wed, Oct 23, 2024 at 1:10 AM rdb...@gmail.com wrote: > > Thanks Vladimir! Would you like to open a PR to make that change? It sounds > like another good item to put into the "Implement

Re: Overwrite old properties on table replace with REST catalog

Thanks Vladimir! Would you like to open a PR to make that change? It sounds like another good item to put into the "Implementation notes" section. On Sun, Oct 20, 2024 at 11:41 PM Vladimir Ozerov wrote: > Hi Jean-Baptiste, > > Agreed. REST spec looks good. I am talking about the general spec, wh

Re: Overwrite old properties on table replace with REST catalog

Hi Jean-Baptiste, Agreed. REST spec looks good. I am talking about the general spec, where it might be useful to add a hint to engine developers, that CREATE OR REPLACE semantics in Iceberg is expected to follow slightly different semantics. This is already broken in Trino: depending on catalog ty

Re: Overwrite old properties on table replace with REST catalog

Hi Vladimir, As Ryan said, it's not a bug: CREATE OR REPLACE can be seen as "CREATE AND UPDATE" from table format perspective. Specifically for the properties, it makes sense to not delete the current properties as it can be used in several use cases (security, tables grouping, ...). I'm not sure

Re: Overwrite old properties on table replace with REST catalog

Hi Ryan, Thanks for the clarification. Yes, I think my confusion was caused by the fact that many engines treat CREATE OR REPLACE as a semantic equivalent of DROP + CREATE, which is performed atomically (e.g., Flink [1]). Table formats add history on top of that, which is expected to be retained,

Re: Overwrite old properties on table replace with REST catalog

Hi Vladimir, This isn't a bug. The behavior of CREATE OR REPLACE is to replace the data of a table, but to maintain things like other refs, snapshot history, permissions (if supported by the catalog), and table properties. Table properties are replaced if they are set in the operation like `b` in