Re: Lantency caised Flink Checkpoint on EXACTLY_ONCE mode

2019-04-06 Thread Guowei Ma
If your implementation only commits your changing after the complete of a checkpoint I think the latency of e2e is at least the interval of checkpoint. I think the document wants to say that a topology, which only has flatmap/filter/map(no task has more than one input) could achieve the exactly o

Lantency caised Flink Checkpoint on EXACTLY_ONCE mode

2019-04-06 Thread min.tan
Hi, I have a simple data pipeline of a Kafka source, a flink map operator and a Kafka sink. I have a quick question about latency caused by the checkpoint on the exactly once mode. Due to the changes are committed and visible on a checkpoint completion, so the latency could be as long

Re: Partitioning key range

2019-04-06 Thread Congxian Qiu
Hi Davood Maybe a custom KeySelector can be helpful, you can define the key used to partition the stream. You can ref the code[1] for detail. [1]  https://github.com/apache/flink/blob/8d05e91945c6c8d83f9924c00890ccf350f1f36f/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/pa