We have upgraded our Kafka version as an attempt to solve this issue. However, the issue is still present in Kafka 1.0.0.
Can I log a bug for this in JIRA? Wouter On 5 February 2018 at 09:22, Wouter Bancken <wouter.banc...@aca-it.be> wrote: > The consumers in consumer group 'X' do not have a regex subscription > matching the newly created topic 'C'. They simply subscribe with > the subscribe(java.util.Collection<java.lang.String> topics) method on > topics 'A' and 'B'. > > Shouldn't the consumer group have a different state from "Stable" during a > rebalancing regardless of the cause? How else can we determine the consumer > lag of the group during the rebalancing? > > Best regards, > Wouter > > Have a look at our brand NEW job website: jobs.aca-it.be ! > > > *ACA IT-Solutions NV* > *HQ:* Herkenrodesingel 8B 2.01 | 3500 Hasselt > T +32(0)11 26 50 10 | F +32(0)11 26 50 11 > www.aca-it.be | Twitter <https://twitter.com/dorien_jorissen> | Facebook > <http://www.facebook.com/pages/ACA-IT-Solutions/278520212202909> | > Linkedin <http://www.linkedin.com/company/6524> > > On 5 February 2018 at 00:13, Hans Jespersen <h...@confluent.io> wrote: > >> Do the consumers in consumer group ‘X’ have a regex subscription that >> matches the newly created topic ‘C’? >> >> If they do then they will only discover this new topic once their ‘ >> metadata.max.age.ms’ metadata refresh interval has passed, which >> defaults to 5 minutes. >> >> metadata.max.age.ms The period of time in milliseconds after which >> we force a refresh of metadata even if we haven't seen any partition >> leadership changes to proactively discover any new brokers or partitions >> -hans >> >> >> > On Feb 4, 2018, at 2:16 PM, Wouter Bancken <wouter.banc...@aca-it.be> >> wrote: >> > >> > Hi Hans, >> > >> > Thanks for the response! >> > >> > However, I get this result for all topics, not just for the newly >> created >> > topic. >> > >> > Situation sketch: >> > 1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with >> > partition assignments and lag information. Consumer group 'X' is >> "Stable". >> > 2a. Topic 'C' is (being) created. >> > 2b. During this creation, I do not have a partition assignment for >> consumer >> > group 'X' for topics 'A' and 'B' but the consumer group is still >> "Stable". >> > 3. A second later: I have a partition assignment for consumer group 'X' >> for >> > topics 'A' and 'B' again and the consumer group is still "Stable". >> > >> > I expected the state of consumer group 'X' during step 2b to be >> > "PreparingRebalance" or "AwaitingSync". >> > >> > Best regards, >> > Wouter >> > >> >> On 4 February 2018 at 21:25, Hans Jespersen <h...@confluent.io> wrote: >> >> >> >> I believe this is expected behavior. >> >> >> >> If there are no subscriptions to a new topic, and therefor no partition >> >> assignments, and definitely no committed offsets, then lag is an >> undefined >> >> concept. When the consumers subscribe to this new topic they may chose >> to >> >> start at the beginning or end of the commit log so the lag cannot be >> >> predicted in advance. >> >> >> >> -hans >> >> >> >>> On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wouter.banc...@aca-it.be >> > >> >> wrote: >> >>> >> >>> Can anyone clarify if this is a bug in Kafka or the expected behavior? >> >>> >> >>> Best regards, >> >>> Wouter >> >>> >> >>> >> >>> On 30 January 2018 at 21:04, Wouter Bancken <wouter.banc...@aca-it.be >> > >> >>> wrote: >> >>> >> >>>> Hi, >> >>>> >> >>>> I'm trying to write an external tool to monitor consumer lag on >> Apache >> >>>> Kafka. >> >>>> >> >>>> For this purpose, I'm using the kafka-consumer-groups tool to fetch >> the >> >>>> consumer offsets. >> >>>> >> >>>> When using this tool, partition assignments seem to be unavailable >> >>>> temporarily during the creation of a new topic even if the consumer >> >> group >> >>>> has no subscription on this new topic. This seems to match the >> >>>> documentation >> >>>> <https://cwiki.apache.org/confluence/display/KAFKA/ >> >> Kafka+Client-side+Assignment+Proposal> >> >>>> saying *"Topic metadata changes which have no impact on subscriptions >> >>>> cause resync"*. >> >>>> >> >>>> However, when this occurs I'd expect the state of the consumer to be >> >>>> "PreparingRebalance" or "AwaitingSync" but it is simply "Stable". >> >>>> >> >>>> Is this a bug in the tooling or is there a different way to obtain >> the >> >>>> correct offsets for a consumer group during a rebalance? >> >>>> >> >>>> I'm using Kafka 10.2.1 but I haven't found any related issues in >> recent >> >>>> changelogs. >> >>>> Best regards, >> >>>> Wouter >> >>>> >> >> >> > >
