I don't particularly know the our Kafka connector, but it sounds like a
matter of whether a given consumer does dirty reads.
Flink does not, whereas the other tools you're using do.
On 12/28/2020 7:57 AM, Daniel Peled wrote:
Hello,
We have 2 flink jobs that communicate with each other through a KAFKA
topic.
Both jobs use checkpoints with EXACTLY ONCE semantic.
We have seen the following behaviour and we want to make sure and ask
if this is the expected behaviour or maybe it is a bug.
When the first job produces a message to KAFKA, the message is
consumed by the second job with a latency that depends on the *first*
job _checkpoint interval_.
We are able to read the message using the kafka tool or using another
kafka consumer, but NOT with a flink kafka consumer that again depends
on the checkpoint interval of the first job.
How come the consumer of the second job depends on the producer
transaction commit time of the first job ?
BR,
Danny
