Re: [DISCUSS] Kafka Connect delta writer support

The Flink sink contributors are working on a solution for streaming deletes without using equality deletes. I'm hoping we can implement a similar solution in the Kafka Connect sink. -Bryan > On Jun 10, 2025, at 9:37 AM, Ryan Blue wrote: > > I'm strongly against writing equality deletes from t

Re: Wide tables in V4

ts > > Can you elaborate on this one? Are those tables populated by streaming or > batch pipelines? > > On Thu, May 29, 2025 at 5:03 PM Bryan Keller <mailto:brya...@gmail.com>> wrote: >> Hi everyone, >> >> We have been investigating a wide table for

Re: Wide tables in V4

Hi everyone, We have been investigating a wide table format internally for a similar use case, i.e. we have wide ML tables with features generated by different pipelines and teams but want a unified view of the data. We are comparing that against separate tables joined together using a shuffle-

Re: [ANNOUNCE] Apache Iceberg release 1.8.0

I had a couple of small fixes that would be great to get into 1.8.1: https://github.com/apache/iceberg/pull/12305 https://github.com/apache/iceberg/pull/12224 I added those to the GitHub 1.8.1 milestone in case this is possible. Thanks, Bryan > On Feb 18, 2025, at 8:53 AM, Robert Stupp wrote: >

Re: New committer: Scott Donnelly

Congrats! -Bryan > On Dec 11, 2024, at 3:47 AM, Fokko Driesprong wrote: > > Congratulations Scott! > > Kind regards, > Fokko > > Op wo 11 dec 2024 om 11:56 schreef Manu Zhang >: >> Congratulations Scott! >> >> Thanks, >> Manu >> >> On Wed, Dec 11, 2024 at 3:2

Re: New committer: Matt Topol

Congrats! -Bryan > On Dec 10, 2024, at 7:37 AM, Matt Topol wrote: > > Thanks everyone! > > On Tue, Dec 10, 2024 at 9:26 AM Gang Wu > wrote: >> Congrats Matt! >> >> On Tue, Dec 10, 2024 at 8:57 PM Sung Yun > > wrote: >>> Congratulations Matt!

[ANNOUNCE] Apache Iceberg release 1.7.1

I'm pleased to announce the release of Apache Iceberg 1.7.1! Apache Iceberg is an open table format for huge analytic datasets. Iceberg delivers high query performance for tables with tens of petabytes of data, along with atomic commits, concurrent writes, and SQL-compatible table evolution. This

Re: [VOTE] Release Apache Iceberg 1.7.1 RC1

88 >>>> Caused by: >>>> org.testcontainers.containers.ContainerLaunchException at >>>> GenericContainer.java:555 >>>> Caused by: java.lang.IllegalStateException at >>>> GenericContainer.java:487 >>>>

Re: [VOTE] Release Apache Iceberg 1.7.1 RC1

gt; I see that the documentation site has already been updated. >> https://iceberg.apache.org/how-to-release/#setup >> >> Best, >> Kevin Liu >> >> On Fri, Nov 22, 2024 at 6:36 AM Bryan Keller > <mailto:brya...@gmail.com>> wrote: >>> Apologies!

Re: [VOTE] Release Apache Iceberg 1.7.1 RC1

B > > On Fri, Nov 22, 2024 at 6:36 AM Yufei Gu wrote: >> >> Hi Bryan, >> >> This link seems broken, https://dist.apache.org/repos/dist/dev/iceberg/KEYS. >> Should we use another one, like the one in here >> https://downloads.apache.org/iceberg/KEYS? &

[VOTE] Release Apache Iceberg 1.7.1 RC1

Hi Everyone, I propose that we release the following RC as the official Apache Iceberg 1.7.1 release. The commit ID is 4a432839233f2343a9eae8255532f911f06358ef * This corresponds to the tag: apache-iceberg-1.7.1-rc1 * https://github.com/apache/iceberg/commits/apache-iceberg-1.7.1-rc1 * https://

Re: [DISCUSS] Iceberg 1.7.1 release

e. Originally I thought wasb scheme change was a feature >>> change. I will wait for 1.8.0. then. >>> >>> On Fri, Nov 15, 2024 at 7:18 AM Bryan Keller >> <mailto:brya...@gmail.com>> wrote: >>>> I agree, we should leave this one for 1.8. &

Re: [DISCUSS] Deprecate embedded manifests

+1 to deprecate > On Nov 19, 2024, at 3:32 AM, Fokko Driesprong wrote: > > Hi everyone, > > I would like to propose to deprecate embedded manifests > . This has been used before the > manifest-list was introduced, but I don't think they are used s

Re: [DISCUSS] - Deprecate Equality Deletes

s, >>>>>>>>>>> Peter >>>>>>>>>>> >>>>>>>>>>> On Fri, Nov 8, 2024, 22:01 Imran Rashid >>>>>>>>>>> wrote: >>>>>>>>>>>> I'm not down i

Re: [DISCUSS] Iceberg 1.7.1 release

ata type to API: > https://github.com/apache/iceberg/pull/11324 > > Thanks, > Aihua > > On Thu, Nov 14, 2024 at 11:21 AM Bryan Keller <mailto:brya...@gmail.com>> wrote: >> Hi all, >> >> As discussed in the sync meeting, we're planning on a fast-foll

[DISCUSS] Iceberg 1.7.1 release

Hi all, As discussed in the sync meeting, we're planning on a fast-follow 1.7.1 release with a few fixes and enhancements: Support for wasb scheme in ADLSFileIO: https://github.com/apache/iceberg/pull/11504 Fix dependencies included in Kafka Connect distribution: https://github.com/apache/icebe

Re: [PROPOSAL] Create Iceberg DockerHub repository

+1 this would be great! Thanks JB. -Bryan > On Nov 14, 2024, at 8:30 AM, Ajantha Bhat wrote: > > +1 for setting up the DockerHub repo, > > We discussed about this already in > https://www.mail-archive.com/dev@iceberg.apache.org/msg07888.html > > Now that the Docker image PR is ready for the

Re: [ANNOUNCE] Apache Iceberg release 1.7.0

A user discovered an issue with the Kafka Connect distribution as a result of an Azure dependency update. I have a PR up to fix that. It would be nice if we could get this in a follow-up 1.7.1 patch release, as the connector won't run on MSK without this. Issue: https://github.com/apache/iceber

Re: [DISCUSS] - Deprecate Equality Deletes

I also feel we should keep equality deletes until we have an alternative solution for streaming updates/deletes. -Bryan > On Nov 4, 2024, at 8:33 AM, Péter Váry wrote: > > Well, it seems like I'm a little late, so most of the arguments are voiced. > > I agree that we should not deprecate the

Re: [VOTE] Deletion Vectors in V3

+1! > On Oct 30, 2024, at 11:03 AM, Daniel Weeks wrote: > > +1 (binding) > > -Dan > > On Wed, Oct 30, 2024 at 10:51 AM Prashant Singh > wrote: >> +1 (non-binding) >> >> Thanks, >> Prashant >> >> On Wed, Oct 30, 2024 at 10:16 AM Russell Spitzer >

Re: [DISCUSS] Remove iceberg-pig module ?

+1 > On Oct 17, 2024, at 1:51 PM, Anton Okolnychyi wrote: > > +1 > > чт, 17 жовт. 2024 р. о 13:42 Steven Wu > пише: >> +1 >> >> On Thu, Oct 17, 2024 at 10:44 AM John Zhuge > > wrote: >>> +1 (non-binding) >>> >>> On Thu, Oct 17, 2024 at 1

Re: [VOTE] Standardize vended credentials in OpenAPI spec

+1 > On Oct 15, 2024, at 10:14 AM, Eduard Tudenhöfner > wrote: > > Hey everyone, > > I'd like to vote on #10722 , > which has been open for quite a while now. > I believe we're in agreement on how we want to standardize credentials in the > Open

Re: [DISCUSS] Iceberg Summit 2025 ?

I'd love to see another Iceberg Summit! -Bryan > On Oct 2, 2024, at 12:33 PM, Rodrigo Meneses wrote: > > love it! Thanks! > > On Wed, Oct 2, 2024 at 10:31 AM Steven Wu > wrote: >> Regarding content, we can have multiple tracks. >> - technology deep dive: how thing

Re: [DISCUSS] Improving Position Deletes in V3

Thanks for the doc Anton, I reviewed it and it looks good to me. Let me know if I can help with anything, it is an area of interest for me. -Bryan > On Aug 21, 2024, at 2:28 PM, Anton Okolnychyi wrote: > > Hey folks, > > As discussed during the sync, I've been working on a proposal to improv

Re: [ANNOUNCE] Welcoming new committers and PMC members

Congrats all! > On Jul 24, 2024, at 3:14 AM, Eduard Tudenhöfner > wrote: > > Congrats everyone, it's amazing to see such great people contributing and > improving the Iceberg community. > > > > On Wed, Jul 24, 2024 at 8:04 AM Honah J. > wrote: >> Thank you all! Co

Re: Status of Kafka Connect contribution?

Hi Ajantha, Yes, there is still the coordinator piece to add before the sink is functional at all. There have been discussions with some in the community around the best path forward for that part to ensure we have a good foundation to build on, which is why we have held off on opening the PR.

Re: New committer: Renjie Liu

Belated congrats! > On Mar 11, 2024, at 1:04 PM, Steve Zhang > wrote: > > Congrats Renjie! > > Thanks, > Steve Zhang > > > >> On Mar 11, 2024, at 12:18 PM, Szehon Ho wrote: >> >> Congratulations! >> >> On Mon, Mar 11, 2024 at 12:43 PM Jack Ye > > wrote: >>> Co

Re: New committer: Bryan Keller

2024 at 1:50 PM Fokko Driesprong wrote: > >> Hi everyone, >> >> The Project Management Committee (PMC) for Apache Iceberg has invited >> Bryan Keller to become a committer and we are pleased to announce that he >> has accepted. >> >> Bryan was contrib

Re: Kafka Connect sink

> > Thanks ! > Regards > JB > > On Mon, Nov 13, 2023 at 3:31 PM Bryan Keller wrote: >> >> Hey JB, >> >> Smaller PRs to the current repo are welcome, though I’m trying to keep >> disruptive changes to a minimum during the submission process. I wi

Re: Kafka Connect sink

t; > Thoughts ? > > Thanks > Regards > JB > >> On Thu, Oct 19, 2023 at 11:13 AM Bryan Keller wrote: >> >> Hi JB, >> >> The plan is to move forward, unless there are concerns from anyone. I got a >> little bit sidetracked but will be working

Re: Kafka Connect sink

t 7:13 AM, Jean-Baptiste Onofré wrote: > > Hi Bryan, > > Any update on this thread ? Can I help somehow ? > > Thanks, > Regards > JB > >> On Mon, Oct 2, 2023 at 7:39 PM Bryan Keller wrote: >> >> Hi all, >> >> We at Tabular would like to

Re: Kafka Connect sink

t;> able to add new features and improvements thanks to the community feedback. >>> I would be more than happy to help for donation (I know that the connector >>> is already under Apache license but we have to double check the ICLA for >>> the initial contributors etc , just

Re: [DISCUSSION] Rename master branch as main for the main repository

Hearty +1 from meOn Sep 29, 2023, at 5:37 AM, Brian Olsen wrote:+1000Let me know how I can help!On Fri, Sep 29, 2023 at 7:35 AM Jean-Baptiste Onofré wrote:Hi guys, The Apache CoC (https://www.apache.org/foundation/policies/conduct) especially contains section 5 about the

Kafka Connect sink

Hi all, We at Tabular would like to contribute our Kafka Connect Iceberg sink to the Iceberg project. It would be great to give Iceberg users another option for landing data from Kafka into Iceberg tables that is supported by the Iceberg community. Kafka Connect is a part of systems from AWS, C

Re: [VOTE] Release Apache Iceberg 1.4.0 RC2

+1 (non-binding) I reran the TPC-DS benchmark with RC2, with the same setup as with RC1, and there were no warnings about decimal pushdown, so that appears resolved. The results were also a bit better at 4915 sec. -Bryan On Fri, Sep 29, 2023 at 10:37 PM Anton Okolnychyi wrote: > +1 (binding) >

Re: [VOTE] Release Apache Iceberg 1.4.0 RC1

sure if that is a regression or just a new warning. > On Sep 28, 2023, at 1:46 PM, Bryan Keller wrote: > > +1 (non-binding) > > I ran the TPC-DS 3 TB benchmark with AWS EMR 6.13/Spark 3.4.1, Iceberg 1.4.0 > RC1, and a REST catalog. The results were slightly slower than 1.3

Re: [VOTE] Release Apache Iceberg 1.4.0 RC1

+1 (non-binding) I ran the TPC-DS 3 TB benchmark with AWS EMR 6.13/Spark 3.4.1, Iceberg 1.4.0 RC1, and a REST catalog. The results were slightly slower than 1.3.1 but within the margin of error. 1.3.1 load: 1671 sec 1.4.0 load: 1820 sec 1.3.1 query: 4961 sec 1.4.0 query: 5210 sec On Thu, Sep 28,