Hi – short answer is consumers can read from a specific partition, but in general for a consumer group you want to balance the partitions across the available consumers for high throughput – if a consumer fails or is kicked off the group because it times out etc then the remainder of the consumers are rebalanced across the partitions etc. Paul
From: Chain Head <mrchainh...@gmail.com> Date: Wednesday, 22 January 2025 at 5:19 pm To: users@kafka.apache.org <users@kafka.apache.org> Subject: Re: JoinGroup API response timing. EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Thanks for the explanation. Slight digression and perhaps a silly question w.r.t. consumers. Since multiple groups are possible, at a high level, the broker effectively sends data of a given topic-partition to multiple consumers while keeping track of offsets. So, why not let consumers specify the partition ID they want to consume? Is the concept of consumer groups only because the consumers wouldn't know in advance how many partitions exist in a topic? Best regards. On Wed, Jan 22, 2025 at 7:41 AM Greg Harris <greg.har...@aiven.io.invalid> wrote: > Hi, > > Thanks for the follow up. > > By "classic" I meant the protocol implemented by the SyncGroup/JoinGroup > API [1]. It's a general group protocol that is still fully supported in > Kraft, and at this time has no intention of being deprecated. > It's called "classic" to distinguish it from the newer KIP-848 [2] > ConsumerGroupHeartbeat API and the share group protocol used in KIP-932 > [3]. It is my understanding that these other protocols are _not_ a > synchronizing barrier in the same way the "classic" protocol is. > > Hope this helps, > Greg > > [1] > > https://urldefense.com/v3/__https://github.com/apache/kafka/blob/adb033211497e539725366960e1013a4638de59f/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/classic/ClassicGroup.java__;!!Nhn8V6BzJA!RUMrj1UUerrRHrSLYNqS64hcjRVWOfQEBJYb0SYerdVNtC92K2MjTB8p00y7-OkYac1rhR0rYeBbneBwtCWqMDDfrw$<https://urldefense.com/v3/__https:/github.com/apache/kafka/blob/adb033211497e539725366960e1013a4638de59f/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/classic/ClassicGroup.java__;!!Nhn8V6BzJA!RUMrj1UUerrRHrSLYNqS64hcjRVWOfQEBJYb0SYerdVNtC92K2MjTB8p00y7-OkYac1rhR0rYeBbneBwtCWqMDDfrw$> > [2] > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-848*3A*The*Next*Generation*of*the*Consumer*Rebalance*Protocol__;JSsrKysrKysr!!Nhn8V6BzJA!RUMrj1UUerrRHrSLYNqS64hcjRVWOfQEBJYb0SYerdVNtC92K2MjTB8p00y7-OkYac1rhR0rYeBbneBwtCVcPWDEEg$<https://urldefense.com/v3/__https:/cwiki.apache.org/confluence/display/KAFKA/KIP-848*3A*The*Next*Generation*of*the*Consumer*Rebalance*Protocol__;JSsrKysrKysr!!Nhn8V6BzJA!RUMrj1UUerrRHrSLYNqS64hcjRVWOfQEBJYb0SYerdVNtC92K2MjTB8p00y7-OkYac1rhR0rYeBbneBwtCVcPWDEEg$> > [3] > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-932*3A*Queues*for*Kafka__;JSsrKw!!Nhn8V6BzJA!RUMrj1UUerrRHrSLYNqS64hcjRVWOfQEBJYb0SYerdVNtC92K2MjTB8p00y7-OkYac1rhR0rYeBbneBwtCWxDptKnA$<https://urldefense.com/v3/__https:/cwiki.apache.org/confluence/display/KAFKA/KIP-932*3A*Queues*for*Kafka__;JSsrKw!!Nhn8V6BzJA!RUMrj1UUerrRHrSLYNqS64hcjRVWOfQEBJYb0SYerdVNtC92K2MjTB8p00y7-OkYac1rhR0rYeBbneBwtCWxDptKnA$> > > On Tue, Jan 21, 2025 at 4:41 PM Chain Head <mrchainh...@gmail.com> wrote: > > > Thanks. > > > > By "classic" you mean pre-KRaft consensus? What is the "current" Kafka > > Group protocol? > > > > Best regards. > > > > On Tue, Jan 21, 2025 at 9:55 PM Greg Harris <greg.har...@aiven.io.invalid > > > > wrote: > > > > > Hi, > > > > > > Yes you are correct. The "classic" Kafka Group Protocol is a > > synchronizing > > > barrier for all members. > > > All JoinGroup member responses are returned after all JoinGroup member > > > requests are received. > > > > > > Thanks, > > > Greg > > > > > > On Tue, Jan 21, 2025 at 7:10 AM Chain Head <mrchainh...@gmail.com> > > wrote: > > > > > > > Assume that three consumers of a certain group want to connect to a > > > broker > > > > for a topic with 3 partitions. After the FindCoordinator API is done, > > the > > > > consumers send JoinGroup. Since the broker cannot know in advance how > > > many > > > > consumers are expected to join, it waits > > > group.initial.rebalance.delay.ms > > > > before starting a rebalance. > > > > > > > > Therefore, does this mean the JoinGroup API response of each request > is > > > > "held" until the waiting period is over? > > > > > > > > Best regards. > > > > > > > > > >
