Re: Kafka Connect sink

2024-01-04 Thread Jean-Baptiste Onofré
during the submission process. I will have > >> the next PR out as soon as the first is merged. > >> > >> Bryan > >> > >>> On Nov 9, 2023, at 9:30 AM, Jean-Baptiste Onofré > >>> wrote: > >>> > >>> Hi Bryan > >

Re: Kafka Connect sink

2024-01-04 Thread Bryan Keller
ll have >> the next PR out as soon as the first is merged. >> >> Bryan >> >>> On Nov 9, 2023, at 9:30 AM, Jean-Baptiste Onofré wrote: >>> >>> Hi Bryan >>> >>> I would like to follow up about kafka connect sink donation. >&

Re: Kafka Connect sink

2024-01-04 Thread Jean-Baptiste Onofré
as soon as the first is merged. > > Bryan > > > On Nov 9, 2023, at 9:30 AM, Jean-Baptiste Onofré wrote: > > > > Hi Bryan > > > > I would like to follow up about kafka connect sink donation. > > > > If you don't mind, I would like to propose a PR

Re: Kafka Connect sink

2023-11-13 Thread Jean-Baptiste Onofré
ve changes to a minimum during the submission process. I will have > the next PR out as soon as the first is merged. > > Bryan > > > On Nov 9, 2023, at 9:30 AM, Jean-Baptiste Onofré wrote: > > > > Hi Bryan > > > > I would like to follow up about kafka connect

Re: Kafka Connect sink

2023-11-13 Thread Bryan Keller
an > > I would like to follow up about kafka connect sink donation. > > If you don't mind, I would like to propose a PR (on the tabular repo) > with the changes we would need for the donation into the Apache > Iceberg repo. > I should have time to work on this next week. &g

Re: Kafka Connect sink

2023-11-09 Thread Jean-Baptiste Onofré
Hi Bryan I would like to follow up about kafka connect sink donation. If you don't mind, I would like to propose a PR (on the tabular repo) with the changes we would need for the donation into the Apache Iceberg repo. I should have time to work on this next week. Thoughts ? Thanks Regar

Re: Kafka Connect sink

2023-10-19 Thread Renjie Liu
Great to hear that. I can help to review. On Thu, Oct 19, 2023 at 17:26 Jean-Baptiste Onofré wrote: > Awesome ! Thanks for the update Bryan. > > I'm looking forward to the PR ! > > Regards > JB > > On Thu, Oct 19, 2023 at 11:13 AM Bryan Keller wrote: > > > > Hi JB, > > > > The plan is to move f

Re: Kafka Connect sink

2023-10-19 Thread Jean-Baptiste Onofré
Awesome ! Thanks for the update Bryan. I'm looking forward to the PR ! Regards JB On Thu, Oct 19, 2023 at 11:13 AM Bryan Keller wrote: > > Hi JB, > > The plan is to move forward, unless there are concerns from anyone. I got a > little bit sidetracked but will be working on addressing comments

Re: Kafka Connect sink

2023-10-19 Thread Bryan Keller
Hi JB, The plan is to move forward, unless there are concerns from anyone. I got a little bit sidetracked but will be working on addressing comments in the first PR then following up with more PRs after that one is merged, so stay tuned for those. Thanks, Bryan > On Oct 18, 2023, at 7:13 AM

Re: Kafka Connect sink

2023-10-18 Thread Jean-Baptiste Onofré
Hi Bryan, Any update on this thread ? Can I help somehow ? Thanks, Regards JB On Mon, Oct 2, 2023 at 7:39 PM Bryan Keller wrote: > > Hi all, > > We at Tabular would like to contribute our Kafka Connect Iceberg sink to the > Iceberg project. It would be great to give Iceberg users another optio

Re: Kafka Connect sink

2023-10-03 Thread Jean-Baptiste Onofré
Hi Bryan Yes, very good point. Especially since Kafka Connect is part of the "Kafka ecosystem", basically anyone using Kafka has Kafka Connect available. The only requirement is to "enable" Kafka Connect (standalone or distributed) and then the Kafka Connect API is available. It's a good idea to

Re: Kafka Connect sink

2023-10-03 Thread Bryan Keller
Thanks for the feedback. Many people use Kafka Connect today, for both loading data into Kafka and writing data from Kafka, so having an Iceberg sink allows someone to make use of their existing infrastructure to write to Iceberg. There are sink connectors for Delta/Databricks, Snowflake, Hudi,

Re: Kafka Connect sink

2023-10-02 Thread Jean-Baptiste Onofré
>From my standpoint, Kafka Connect is interesting to also address processing logic without Spark or Flink runtime. Definitely interesting to have Kafka integration/processing (even for me Kafka and Kafka Connect are two different things ;)). For pure data ingestion part, I think it would make sens

Re: Kafka Connect sink

2023-10-02 Thread Ajantha Bhat
Hi Bryan, I am very happy to see this contribution. I have recently tested this project with Nessie catalog and very much liked it. However, I still don't know the benefits of using kafka-connect instead of directly consuming from the kafka like Delta-lake's implementation. https://github.com/del

Re: Kafka Connect sink

2023-10-02 Thread Jean-Baptiste Onofré
Hi Bryan That’s a great news ! Thanks a lot for the proposal. I will take a look on the PR and existing connector. I’m sure the Iceberg community will be very happy to see this and we will able to add new features and improvements thanks to the community feedback. I would be more than happy to he

Kafka Connect sink

2023-10-02 Thread Bryan Keller
Hi all, We at Tabular would like to contribute our Kafka Connect Iceberg sink to the Iceberg project. It would be great to give Iceberg users another option for landing data from Kafka into Iceberg tables that is supported by the Iceberg community. Kafka Connect is a part of systems from AWS, C

Iceberg Kafka Connect Sink

2023-06-21 Thread Brian Olsen
Hey Iceberg Devs, Just wanted to make you all aware of a project Tabular has been building around a new Kafka Connect sink for Iceberg. There have been a few cases we wanted to address for customers and Iceberg users alike that just weren't enabling exactly-once semantics in the sink, c