Hi Martin! For an example of a source that acknowledges received messages, you could check the MessageAcknowledgingSourceBase and the MultipleIdsMessageAcknowledgingSourceBase that ship with Flink. I hope this will give you some ideas.
Now for the Flink version on top of which to implement your source, I would suggest the Flink 1.3. The reason is that it will come out soon (~1 month) and it will include a lot of new features and bug-fixes. Until then, it may change a bit, but the APIs that you will be using, will not change. So why not going straight for the more future-proof way? Thanks, Kostas > On Apr 24, 2017, at 11:20 PM, Martin Eden <martineden...@gmail.com> wrote: > > Hi everyone, > > Are there any examples of how to implement a reliable (zero data loss) Flink > source reading from a system that is not replay-able but supports > acknowledging messages? > > Or any pointers of how one can achieve this and how Flink can help? > > I imagine it should involve a write ahead log but not yet clear of how to > implement it and how to integrate with the Flink fault tolerance mechanism. > Can Flink maintain the write ahead log for me? > > Also, does it make sense to start implementing this in the current stable > Flink release 1.2 or is there any advantage in implementing it directly in > Flink 1.3 since it is coming up soon anyway? > > Thanks, > M
