Hi Andrew, Thanks for taking a look. I think I corrected too much for the classic protocol when I did my investigation because that’s the one I’ve been working with for the majority and haven’t dabbled in the kip-848 mechanisms.
You’re right of course about the gap that the consumer protocol brings. I’ve had to follow my way back from the rebalance call sites into the heartbeatresponse/ group coordinator flow. I’ve updated the KIP to reflect 3 major changes: Expanded the RebalanceCause enums to contain the various coordinator states that can cause a trigger (member joins/leaves, subscription changes, topic metadata changed, regex matched new topics, new assignor) Publish broker metrics ( GC metrics ) for each of the points where an epoch is bumped and we currently update a consumer-group-rebalance* meter sensor. I noticed we’re not publishing a metric when a member leaves/is fenced. So I’ve addressed that gap as well. Add a short field on the heartbeat response to contain a bitmask of rebalance causes between the member and group epochs. This addresses the “everything is coordinator initiated” point you highlighted. The client reads this bit and computes the cause field (there may be multiple causes during high churn - in which case we fallback on coordinator_initiated). I feel the HB response makes the cause and client metrics more possible than the previous client-centric design. One thing I also discovered is that share and streams also have their own HB responses but I’ve deferred both until a later KIP. LMK if it’s better to rope those in now. Curious to know your thoughts on the updated KIP. Best, Aditya > On Jul 16, 2026, at 10:02, Andrew Schofield <[email protected]> wrote: > > Hi Aditya, > Thanks for the KIP. > > I can see the idea behind the KIP but I'm not convinced that it's possible to > work out a coherent set of RebalanceCause cases, in particular for the > consumer group protocol. In this protocol, rebalances are always > "coordinator-initiated". Just because the application calls > Consumer.subscribe() resulting in a ConsumerGroupHeartbeat request being sent > to the GC, it's not certain that the next ConsumerGroupHeartbeat response > will contain the newly assigned partitions. Assignment is asynchronous and > the GC makes various optimisations to avoid excessive rebalances for groups > with a lot of churn. I'm not sure I could tell the difference between > "coordinator-initiated" and "subscription-changed". If you can explain in > more detail how each of the cases corresponds to ConsumerGroupHeartbeat > response, it would help. > > We do need to put more work into the broker-side metrics for rebalance for > the consumer group protocol. We'd like more visibility than we have today for > sure. > > Thanks, > Andrew > >> On 2026/07/08 21:19:20 Aditya Kousik wrote: >> Hi all, >> >> I’d like to start a discussion on KIP-1364 as a follow-up to KIP-1306. >> >> KIP link: https://cwiki.apache.org/confluence/x/jYE_Gg >> >> A Kafka consumer client today cannot tell why a rebalance happened without >> grepping logs. The rebalance trigger can help with triaging: >> >> * max.poll.interval.ms exceeded implies misconfig or delayed poll loop; >> * Member fencing points at session timeout >> * Coordinator-initiated rebalances are regular occurrences due to join and >> leave requests. >> * Application calls to Consumer.enforceRebalance()are expected and the >> application owns the context. >> >> Existing rebalance metrics describe frequency and latency but carry no cause >> dimension: >> >> rebalance-rate-per-hour, rebalance-latency-{avg,max}, rebalance-total >> last-rebalance-seconds-ago, last-poll-seconds-ago >> >> So, the cause would be useful in diagnosing issues by surfacing the event to >> the user. >> >> Looking forward to hearing your thoughts on this. >> >> Best, >> Aditya Kousik
