Hi Raghavendar, In Flink, checkpoints are global, meaning that a checkpoint is successful only if all operators acknowledge it. So the offset will be stored in state and then committed to Kafka [1] only after all the tasks acknowledge that checkpoint. At that moment, the element must be either emitted to the external system, stored in the operator state (e.g. window); or in channel state (with Unaligned checkpoints).
So you don't need to re-process it manually (given that the sink provides exactly once guarantee). [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-offset-committing-behaviour-configuration Regards, Roman On Thu, Apr 29, 2021 at 12:53 PM Raghavendar T S <raghav280...@gmail.com> wrote: > Hi Team > > Assume that we have a job (Checkpoint enabled) with Kafka source and a > stateless operator which consumes events from Kafka source. > We have a stream of records 1, 2, 3, 4, 5,...Let's say the event 1 reaches > the Flat Map operator and is being processed. Then the Kafka source has > made a successful checkpoint. > In this case, does the offset of event 1 will be part of the checkpoint? > Will Flink track the event from source to all downstream operators? If > this is a true case and If the processing of the event is failed (any third > party API/DB failure) in the Flat Map after a successful checkpoint, do we > need to manually re-process (retry using queue or any other business logic) > the event? > > Job: > Kafka Source -> Flat Map > > Thank you > > -- > Raghavendar T S > www.teknosrc.com > > > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > Virus-free. > www.avast.com > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> > <#m_-166071601284684373_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >
