Hi Ajantha, The minimum part needed to have it an "out of the box" Kafka sink is the coordinator. Tombstone handler, etc would be good to have asap but less blocker. It would be great to have a plan for the commit coordinator. Happy to help on this (I started a Camel based sink to experiment as well).
I agree with the 1.6.0 target, it would be a good milestone item. Thanks Regards JB On Mon, Apr 15, 2024 at 7:30 PM Ajantha Bhat <ajanthab...@gmail.com> wrote: > > Hey everyone, > > We've been testing the Kafka Connect connector from the 1.5.0 release, and > unfortunately, it seems it's still not ready for use. We're not quite up to > date on its current status. > > We've noticed that three pull requests have been merged so far: #8701, #9466, > and #9641. There's also a discussion in the comments of PR #9641 about > holding off on supporting upsert > and CDC until we've thoroughly analyzed all the gaps: > https://github.com/apache/iceberg/pull/9641#issuecomment-1970183410 > > At least to make the connector usable for append operations only, it seems we > still need to work on the commit coordinator and integration tests. > What's the plan for addressing this? We're eager to pitch in and contribute. > Additionally, could we aim for a 1.6.0 release to at least have a usable > connector for append operations? > > We'd also like to discuss the gaps related to upsert (equality deletes) and > brainstorm potential solutions, as CDC is a crucial feature for the connector. > Could we also set up an issue tracker to keep track of what tasks are > completed and what's still pending? > > - Ajantha
