RE: Re: KafkaIO does not make use of Kafka Consumer Groups [kafka] [java] [io]

Can't agree with Shahar Frank more On 2023/04/19 18:17:15 Shahar Frank wrote: > Hi Daniel, > > I think I've already answered these in a previous email but let me answer > them again. > > I was specifically responding to quoted points from your last email. I >> really don't understand why you,

Re: KafkaIO does not make use of Kafka Consumer Groups [kafka] [java] [io]

Hi, my two cents on this. While it would perfectly possible to use consumer group in KafkaIO, it has its own issues. The most visible would be, that using subscriptions might introduce unnecessary duplicates in downstream processing. The reason for this is that consumer in a consumer group mi

Re: KafkaIO does not make use of Kafka Consumer Groups [kafka] [java] [io]

Can't agree with Shahar Frank more On 2023/04/19 18:17:15 Shahar Frank wrote: > Hi Daniel, > > I think I've already answered these in a previous email but let me answer > them again. > > I was specifically responding to quoted points from your last email. I > > really don't understand why you,

Re: KafkaIO does not make use of Kafka Consumer Groups [kafka] [java] [io]

Why do you want to use a consumer group? If you have consumers in other jobs, your beam job will fail to receive all messages it should for the topic. > It seems the code attempts to circumvent the partition assignment mechanism provided by Kafka to use it's own. All beam I/Os for partitioned sou

KafkaIO does not make use of Kafka Consumer Groups [kafka] [java] [io]

Hi All, Posting here as suggested here . I'm using KafkaIO to consume events from a Kafka topic. I've added "group.id" to the consumer properties. When running the pipeline I can see this value sent to Kafka in the consumer prop