RE: Re: Lantency caised Flink Checkpoint on EXACTLY_ONCE mode

2019-04-09 Thread min.tan
lto:guowei@gmail.com] Sent: Sonntag, 7. April 2019 08:42 To: Tan, Min Cc: user@flink.apache.org Subject: [External] Re: Lantency caised Flink Checkpoint on EXACTLY_ONCE mode If your implementation only commits your changing after the complete of a checkpoint I think the latency of e2e is at

Re: Lantency caised Flink Checkpoint on EXACTLY_ONCE mode

2019-04-08 Thread Fabian Hueske
Hi Min, Guowei is right, the comment in the documentation about exactly-once in embarrassingly parallel data flows refers to exactly-once *state consistency*, not *end-to-end* exactly-once. However, in strictly forwarding pipelines, enabling exactly-once checkpoints should not have drawbacks compa

Re: Lantency caised Flink Checkpoint on EXACTLY_ONCE mode

2019-04-06 Thread Guowei Ma
If your implementation only commits your changing after the complete of a checkpoint I think the latency of e2e is at least the interval of checkpoint. I think the document wants to say that a topology, which only has flatmap/filter/map(no task has more than one input) could achieve the exactly o

Lantency caised Flink Checkpoint on EXACTLY_ONCE mode

2019-04-06 Thread min.tan
Hi, I have a simple data pipeline of a Kafka source, a flink map operator and a Kafka sink. I have a quick question about latency caused by the checkpoint on the exactly once mode. Due to the changes are committed and visible on a checkpoint completion, so the latency could be as long