Checkpoint can be done synchronously and asynchronously, the latter is the default . If you chooese the synchronous way , it may cause this problem.
nick toker <[email protected]> 于2020年12月23日周三 下午3:53写道： > Hi Yun, > > Sorry but we didn't understand your questions. > The delay we are experiencing is on the *read* side. > The message is written to kafka topic and consumed by flink with a delay > that depends on the checkpoints interval > When we disabled the checkpoints the messages are immediately consumed > We use the EXACTLY-ONCE semantic. > > Please advise. > BR, > Nick > > בתאריך יום ג׳, 22 בדצמ׳ 2020 ב-9:32 מאת Yun Gao <[email protected] > >: > >> Hi nick, >> >> Sorry I initially think that the data is also write into Kafka with >> flink . So it could be ensured that there is no delay in the write side, >> right ? Does the delay in the read side keeps existing ? >> >> Best, >> Yun >> >> >> >> ------------------Original Mail ------------------ >> *Sender:*nick toker <[email protected]> >> *Send Date:*Tue Dec 22 01:43:50 2020 >> *Recipients:*Yun Gao <[email protected]> >> *CC:*user <[email protected]> >> *Subject:*Re: checkpoint delay consume message >> >>> hi >>> >>> i am confused >>> >>> the delay in in the source when reading message not on the sink >>> >>> nick >>> >>> בתאריך יום ב׳, 21 בדצמ׳ 2020 ב-18:12 מאת Yun Gao < >>> [email protected]>: >>> >>>> Hi Nick, >>>> >>>> Are you using EXACTLY_ONCE semantics ? If so the sink would use >>>> transactions, and only commit the transaction on checkpoint complete to >>>> ensure end-to-end exactly-once. A detailed description could be find in [1] >>>> >>>> >>>> Best, >>>> Yun >>>> >>>> >>>> [1] >>>> https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html >>>> >>>> ------------------------------------------------------------------ >>>> Sender:nick toker<[email protected]> >>>> Date:2020/12/21 23:52:34 >>>> Recipient:user<[email protected]> >>>> Theme:checkpoint delay consume message >>>> >>>> Hello, >>>> >>>> We noticed the following behavior: >>>> If we enable the flink checkpoints, we saw that there is a delay >>>> between the time we write a message to the KAFKA topic and the time the >>>> flink kafka connector consumes this message. >>>> The delay is closely related to checkpointInterval and/or >>>> minPauseBetweenCheckpoints meening that the MAX delay when consuming a >>>> message from KAFKA will be one of these parameters. >>>> >>>> Could you please advise how we can remove/control this delay? >>>> >>>> we use flink 1.11.2 >>>> >>>> BR >>>> nick >>>> >>>>
