Hi, 1. What Flink Kafka connector version are you using? 2. How is your non-Flink consumer fetching data from the topic? Is it using the old SimpleConsumer, old High-Level Consumer, or the new consumer API? 3. If you are using the new consumer API, are you using "consumer.assign(…)" or "consumer.subscribe(…)"?
Internally, the Flink Kafka connectors don’t use the consumer group management functionality because they are using lower-level APIs (SimpleConsumer in 0.8, and KafkaConsumer#assign(…) in 0.9) on each parallel instance for more control on individual partition consumption. So, essentially, the “group.id” setting in the Flink Kafka connector is only used for committing offsets back to ZK / Kafka brokers. >From your description I’m not entirely sure of what’s happening, so it’d be helpful if you can provide info about the above questions so we can clarify things further :) Regards, Gordon On July 29, 2016 at 12:11:19 AM, 王萌 (wa...@unicall.cc) wrote: Hi, I am using kafka in couple of programs including flink and I am quite confused that how group.id parameter work in flink with kafka comsumer. I have 2 consumers (one in side flink, one outside) running on the same topic and same group.id. From my inpection, they work isolately: If I send one message into the kafka topic, both consumer would receive it. But acording to the kafka document, only one (same topic same group) should be able to get the message. And i tried to start 2 consumers both inside the flink main function, it seems that only one consumer would be receiving all messages from kafka. So, it's quite confused how does flink deal with this kind of situation? Thanks a lot
