Hi Nick,
Are you using EXACTLY_ONCE semantics ? If so the sink would use
transactions, and only commit the transaction on checkpoint complete to ensure
end-to-end exactly-once. A detailed description could be find in [1]
Best,
Yun
[1]
https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html
------------------------------------------------------------------
Sender:nick toker<[email protected]>
Date:2020/12/21 23:52:34
Recipient:user<[email protected]>
Theme:checkpoint delay consume message
Hello,
We noticed the following behavior:
If we enable the flink checkpoints, we saw that there is a delay between the
time we write a message to the KAFKA topic and the time the flink kafka
connector consumes this message.
The delay is closely related to checkpointInterval and/or
minPauseBetweenCheckpoints meening that the MAX delay when consuming a message
from KAFKA will be one of these parameters.
Could you please advise how we can remove/control this delay?
we use flink 1.11.2
BR
nick
