Re: [DISCUSS] Action to Rewrite Equality Deletes as Position Deletes

Hi Prashant, Interesting stuff! I have a few questions: 1. I understand that it is easier to apply positional deletes than equality deletes. What about the conversion cost? Based on your experience, what is the cost difference between a conversion and a full rewrite? When does it worth to do a de

Re: [DISCUSS] Row Lineage Proposal

Thanks Russel. Not a question on the proposal itself, I find it a bit hard to follow and maintain all the three specs in one place. We are also publishing a unfinalized spec to the website. Would it be better to maintain the spec in a "copy-on-write" style, i.e. each spec having its own format file

Re: [DISCUSS] Action to Rewrite Equality Deletes as Position Deletes

+1, Id be happy to see this feature. Thanks Szehon On Fri, Sep 13, 2024 at 10:33 AM Prashant Singh wrote: > Hi All, > > Starting this thread to revive the discussion on converting Equality > Deletes as Position deletes and see if this is something community wants > now (Happy to contribute in t

Re: [DISCUSS] Iceberg Materialzied Views

The main reason for putting the lineage into the view is so that "another" engine can enumerate out the tables in the view without needing to parse any SQL. But, if we put the lineage under the SQL representation with engine specific catalog names, the "other" engine is not going to be able to use

Re: [DISCUSS] Row Lineage Proposal

Pull Request Available, please focus any remaining comments there and we can wrap this one up https://github.com/apache/iceberg/pull/11130 On Thu, Aug 29, 2024 at 11:20 AM rdb...@gmail.com wrote: > +1 for making row lineage and equality deletes mutually exclusive. > > The idea behind equality d

[DISCUSS] Action to Rewrite Equality Deletes as Position Deletes

Hi All, Starting this thread to revive the discussion on converting Equality Deletes as Position deletes and see if this is something community wants now (Happy to contribute in this) considering : 1/ Now it's not just flink but other writers such Kafka-Connect [1] debezium server for iceberg [2]

[DISCUSS] REST: Standardize vended credentials in Spec

Hey everyone, I'd like to propose standardizing the vended credentials used in loadTable / loadView responses. I opened #8 to track the proposal in GH. Please find the proposal doc here

[DISCUSS] Modify ThreadPools.newWorkerPool to avoid unnecessary Shutdown Hook registration

Hi all, During the investigation of a metaspace memory leak issue in Flink IcebergSource ( https://github.com/apache/iceberg/pull/11073 ), a discussion with @pvary revealed that *ThreadPools.newWorkerPool* currently registers a Shutdown Hook via ExitingExecutorService for all created thread pools.

Re: [DISCUSS] Iceberg Materialzied Views

Hi, regarding our recent discussion on table identifiers with respect to different catalog_names with different query engines. We have the same problem when we want to reference the storage table from the common view. *If we include the catalog_name as part of the identifier, different query