We have a Flink stream job that uses FlinkKafkaConsumer010. We keep the kafkaconsumer operator's parallelism same as the number of partition in kafka topic so that each partition is attached to one subtask. Quite frequently the job stops reading from certain partitions. On investigating under job metrics tab(committed offsets & current offset metrics), few partitions are not associated with any subtask while few partitions are associated with more than 1 subtasks(which seems wrong behavior). I have debugged it locally and seen that initially the consumer starts fine with uniform association with partitions but over the time some partitions disassociate from the subtask.
Here is our kafka consumer config -checkpoint disabled. -enable.auto.commit true -auto.commit.interval.ms 5 minutes -request.timeout.ms 3 minutes Kafka Version: 0.10.1.0 Flink version: 1.3.1 Has someone else faced this issue? Any help or pointers would be much appreciated. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
