Re: Kafka transactions drastically limit usability of Flink savepoints

2022-11-17 Thread Piotr Nowojski
Hi Yordan, Indeed it looks like a missing feature. Probably someone implementing the new KafkaSink didn't realize how important this is. I've created a ticket to work on this issue [1], but I don't know when or who could fix it. I think a workaround might be to create a new `KafkaSink` instance t

Re: Kafka transactions drastically limit usability of Flink savepoints

2022-11-16 Thread Yordan Pavlov
Hi Piotr, the option you mention is applicable only for the deprecated KafkaProducer, is there an equivalent to the modern KafkaSink? I found this article comparing the behavior of the two: https://ververica.zendesk.com/hc/en-us/articles/360013269680-Best-Practices-for-Using-Kafka-Sources-Sinks-in

Re: Kafka transactions drastically limit usability of Flink savepoints

2022-11-15 Thread Piotr Nowojski
Hi Yordan, I don't understand where the problem is, why do you think savepoints are unusable? If you recover with `ignoreFailuresAfterTransactionTimeout` enabled, the current Flink behaviour shouldn't cause any problems (except for maybe some logged errors). Best, Piotrek wt., 15 lis 2022 o 15:3