Hi all, This is an initial proposal for supporting Iceberg natively as a Kafka Connect sink.
As discussed in the community sync up last week, we have created an initial design document for a Kafka Connect Iceberg sink. We are seeking feedback from anybody with knowledge of Iceberg, specifically around the commit process, as the sink needs to be able to support high throughput writing. As there has been discussion for some time in the community about adding a Kafka Connect sink, myself and Kyle Bendickson [@kbendick] are hoping to find people who might be interested in working on the connector or would be willing to review code as time permits (particularly if you have existing knowledge of either Kafka Connect or the Iceberg internals, but all are welcomed). We're also happy to have anybody interested in contributing to the discussion and further design stages. Link to the design doc: https://docs.google.com/document/d/1LQQXbi2gJ2CE5KltBiVj8WSDOVtO_wiVnGJeACIb5cU Please review and comment on the design doc, and if you'd be interested in helping with this effort, feel free to respond to this email or join the #kafka-connect channel in the Iceberg Slack. -- Best, Karthik Kondamudi M: 415 516 4307
