Re: Kafka Connect exactly-once semantic and very large transactions

2023-06-09 Thread Vojtech Juranek
Hi Chris, thanks for your response! Yes, we are looking also on other means how to enable exactly-once semantics for existing data (e.g. using incremental snapshot which snapshots the data incrementally and in smaller chunks), but first we would like to fully understand all the implications and

Re: Kafka Connect exactly-once semantic and very large transactions

2023-06-08 Thread Chris Egerton
Hi Vojta, >From my limited understanding of the Debezium snapshot process, I believe that you're correct that producing the entire snapshot in a transaction is the way to provide exactly-once semantics during that phase. If there's a way to recover in-progress snapshots and skip over already-produ