Hi! The only occasions which the consumer group is used is: 1. When committing offsets back to Kafka. Since Flink 1.3, this can be disabled completely (both when checkpointing is enabled or disabled). See [1] on details about that. 2. When starting fresh (not starting from some savepoint), if you choose to use GROUP_OFFSETS as the start position, then the consumer group would also be used. If starting from a savepoint, then this is irrelevant. See [2].
Note that it actually isn’t used in any critical paths for Flink’s exactly-once processing guarantees, or partition to source parallel instance assignments. So, the desired behavior in which you described is exactly the expected behavior for the Flink Kafka Consumer. Cheers, Gordon [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/kafka.html#kafka-consumers-offset-committing-behaviour-configuration [2] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/kafka.html#kafka-consumers-start-position-configuration On 19 July 2017 at 3:23:01 PM, Moiz Jinia (moiz.ji...@gmail.com) wrote: Below is a plan for downtime-free upgrade of a Flink job. The downstream consumer of the Flink job is duplicate proof. Scenario 1 - 1. Start Flink job A with consumer group G1 (12 slot job) 2. While job A is running, take a savepoint AS. 3. Start newer version of Flink job A' from savepoint AS with consumer group *G1* (12 slot job again) 4. Stop job A. Scenario 2 - 1. Start Flink job A with consumer group G1 (12 slot job) 2. While job A is running, take a savepoint AS. 3. Start newer version of Flink job A' from savepoint AS with consumer group *G2* (12 slot job again) 4. Stop job A Does it matter what consumer group job A' uses? The desired behavior is that during the window when both A and A' are running, all messages should go to both jobs. (And of course I want that job A' should start consuming from the offsets in the savepoint and not the earliest). -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Does-FlinkKafkaConsumer010-care-about-consumer-group-tp14323.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
