Re: File Source Exactly Once Delivery Semantics

2023-08-02 Thread Shammon FY
Hi Kirti, Simply speaking, sink needs to support `two-stage commit`, the sink can `write` data as normal and only `commit` data after the checkpoint is successful. This ensures that even if a failover occurs and data needs to be replayed, the previously written data is not visible to the user. How

File Source Exactly Once Delivery Semantics

2023-08-02 Thread Kirti Dhar Upadhyay K via user
Hi Team, I am using Flink File Source in one of my use case. I observed that, while reading file by source reader it stores its position in checkpointed data. In case application crashes, it restores its position from checkpointed data, once application comes up, which may result in re-emitting