Re: Flink Kafka consumer auto-commit timeout

2020-03-13 Thread Rong Rong
Yes, this is a Kafka side issue. Since the affected version of Kafka is all below 1.1.0, ideally speaking we should upgrade Kafka minor version on flink-connector-kafka-0.10/0.11 once the fix was back-ported on the Kafka side. However based on the fact that the PR has been merged for 2 years, I am

Re: Flink Kafka consumer auto-commit timeout

2020-03-13 Thread Aljoscha Krettek
Thanks for the update! On 13.03.20 13:47, Rong Rong wrote: 1. I think we have finally pinpointed what the root cause to this issue is: When partitions are assigned manually (e.g. with assign() API instead subscribe() API) the client will not try to rediscover the coordinator if it dies [1]. This

Re: Flink Kafka consumer auto-commit timeout

2020-03-13 Thread Rong Rong
Hi Aljoscha, Thank you for the help and reply. 1. I think we have finally pinpointed what the root cause to this issue is: When partitions are assigned manually (e.g. with assign() API instead subscribe() API) the client will not try to rediscover the coordinator if it dies [1]. This seems to no

Re: Flink Kafka consumer auto-commit timeout

2020-03-11 Thread Aljoscha Krettek
On 09.03.20 06:10, Rong Rong wrote: - Is this feature (disabling checkpoint and restarting job from Kafka committed GROUP_OFFSET) not supported? I believe the Flink community never put much (any?) effort into this because the Flink Kafka Consumer does its own offset handling. Starting from th