Re: Checkpoint for exact-once proccessing

2016-01-13 Thread Don Frascuchon
tions in operators need not to be implemented >> idempotent. >> >> Hope this helps answer your question! >> >> Cheers, >> Gordon >> >> >> >> -- >> View this message in context: >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Checkpoint-for-exact-once-proccessing-tp4261p4264.html >> Sent from the Apache Flink User Mailing List archive. mailing list >> archive at Nabble.com. >> > >

Re: Checkpoint for exact-once proccessing

2016-01-13 Thread Stephan Ewen
functions in operators need not to be implemented > idempotent. > > Hope this helps answer your question! > > Cheers, > Gordon > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Checkpoint-for-exact-once-proccessing-tp4261p4264.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >

Re: Checkpoint for exact-once proccessing

2016-01-13 Thread Tzu-Li (Gordon) Tai
ng-list-archive.2336050.n4.nabble.com/Checkpoint-for-exact-once-proccessing-tp4261p4264.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Checkpoint for exact-once proccessing

2016-01-13 Thread Don Frascuchon
Hi Stephan, Thanks for your quickly response. So, consider an operator task with two processed records and no barrier incoming. If the task fail and must be records, the last consistent snapshot will be used, which no includes information about the processed but no checkpointed records. What abo

Re: Checkpoint for exact-once proccessing

2016-01-13 Thread Stephan Ewen
Hi! I think there is a misunderstanding. There are no identifiers maintained and no individual records deleted. On recovery, all operators reset their state to a consistent snapshot: https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/stream_checkpointing.html Greetings, Step

Checkpoint for exact-once proccessing

2016-01-13 Thread Don Frascuchon
Hello, I'm trying to understand the process of checkpoint processing for exact-once in Flink, and I have some doubts. The documentation says that when there is a failure and the state of an operator is restored, the already processed records are deleted based on their identifiers. My doubts is,