Hi Yun, Sorry but we didn't understand your questions. The delay we are experiencing is on the *read* side. The message is written to kafka topic and consumed by flink with a delay that depends on the checkpoints interval When we disabled the checkpoints the messages are immediately consumed We use the EXACTLY-ONCE semantic.
Please advise. BR, Nick בתאריך יום ג׳, 22 בדצמ׳ 2020 ב-9:32 מאת Yun Gao <yungao...@aliyun.com >: > Hi nick, > > Sorry I initially think that the data is also write into Kafka with > flink . So it could be ensured that there is no delay in the write side, > right ? Does the delay in the read side keeps existing ? > > Best, > Yun > > > > ------------------Original Mail ------------------ > *Sender:*nick toker <nick.toker....@gmail.com> > *Send Date:*Tue Dec 22 01:43:50 2020 > *Recipients:*Yun Gao <yungao...@aliyun.com> > *CC:*user <user@flink.apache.org> > *Subject:*Re: checkpoint delay consume message > >> hi >> >> i am confused >> >> the delay in in the source when reading message not on the sink >> >> nick >> >> בתאריך יום ב׳, 21 בדצמ׳ 2020 ב-18:12 מאת Yun Gao < >> yungao...@aliyun.com>: >> >>> Hi Nick, >>> >>> Are you using EXACTLY_ONCE semantics ? If so the sink would use >>> transactions, and only commit the transaction on checkpoint complete to >>> ensure end-to-end exactly-once. A detailed description could be find in [1] >>> >>> >>> Best, >>> Yun >>> >>> >>> [1] >>> https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html >>> >>> ------------------------------------------------------------------ >>> Sender:nick toker<nick.toker....@gmail.com> >>> Date:2020/12/21 23:52:34 >>> Recipient:user<user@flink.apache.org> >>> Theme:checkpoint delay consume message >>> >>> Hello, >>> >>> We noticed the following behavior: >>> If we enable the flink checkpoints, we saw that there is a delay between >>> the time we write a message to the KAFKA topic and the time the flink kafka >>> connector consumes this message. >>> The delay is closely related to checkpointInterval and/or >>> minPauseBetweenCheckpoints meening that the MAX delay when consuming a >>> message from KAFKA will be one of these parameters. >>> >>> Could you please advise how we can remove/control this delay? >>> >>> we use flink 1.11.2 >>> >>> BR >>> nick >>> >>>
