Re: Some questions related to compaction support.

> I just raised a PR to fix it [https://github.com/apache/iceberg/pull/3685/ ] It seems it is not straight forward. Will have discussions with Russell and others in the PR and conclude. Thanks, Ajantha On Wed, Dec 8, 2021 at 11:43 AM Ajantha Bhat wrote: > >>1. Spark 2.4 should also have su

Re: Some questions related to compaction support.

> > >1. Spark 2.4 should also have support via the direct action API for >compaction (and the action API should be sufficient for me); but the class >pointed out > >

Vendor integration strategy

Hi everyone, As we are adding Aliyun as a new vendor integration in the upcoming release, we are discussing the strategy we should take to integrate the iceberg-aliyun package with all the engine runtimes. For some background, we had some discussions about this topic when releasing Nessie and AWS

[GitHub] [iceberg-docs] jackye1995 commented on pull request #1: First version of hugo doc site

jackye1995 commented on pull request #1: URL: https://github.com/apache/iceberg-docs/pull/1#issuecomment-988424006 How are we going to move forward with this PR? If we move to the `next` branch, is this PR for the `main` branch still needed? -- This is an automated message from the Apach

Re: Some questions related to compaction support.

Regarding 4, when you run a RewriteDataFiles, the MergingSnapshotProducer automatically drops a delete file if there is no data file of lower sequence number: https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L523-L525. This serves as a

Re: Some questions related to compaction support.

Ajantha, Jack and Russell, Thanks for the prompt replies. Just consolidating the information, my understanding is: 1. Spark 2.4 should also have support via the direct action API for compaction (and the action API should be sufficient for me); but the class pointed out