Any help here please. Regards, Kartik
On Fri, Feb 9, 2024, 8:33 AM Kartik Kushwaha <kushwaha.karti...@gmail.com> wrote: > I am using flink checkpointing to restore states of my job. I am using > unaligned checkpointing with 100 ms as the checkpointing interval. I see > few events getting dropped that were sucessfully processed by the operators > or were in-flight that were yet to be captured by checkpoint. That is these > were new events which came into the pipeline between the previously > captured checkpoint state and the failure. > <https://stackoverflow.com/posts/77955164/timeline> > > My project acknowledges(commits) back to the topic after the event read > and mongo ingestion. But the pipeline has transformation, enrichment and > sink operators after that. These missing events were read, ack'd back to > the topic and transformed successfully before failure and were not yet > checkpointed (withing the 100 ms interval between checkpoints) were dropped. > > Pipeline: Source (solace topic, queue reader) --> [MongoWrite + > sourcecommit] --> transform --> enrich --> sink (solace topic) > > Checkpoint: Unaligned, Exactly-once, 100ms interval, 10ms Min pause > between checkpoint > > 1. > > I see that whenever runtime exception is thrown it triggers the close > method in each of the functions one by one. Do we have to store the states > which were not yet captured by the checkpoint before failure? What happens > Network failures or task manager crash or any other abrupt failure? > 2. > > Or do we have to shift the source topic acknowledgment to the last ( > but we will have to chain all this operators to run in a single thread and > carry the bytearray message object from solace queue to do ack at the end). > > Is there anything else Iam missing here? > > > Note: Sources and Sinks are fully Solace based in all the Flink pipelines > ( queues and topics) > > >
