I would encourage you todo so.
I also think its not reasonable behavior
On 13.02.2018 11:28, Wouter Bancken wrote:
We have upgraded our Kafka version as an attempt to solve this issue.
However, the issue is still present in Kafka 1.0.0.
Can I log a bug for this in JIRA?
Wouter
On 5 February 2018 at 09:22, Wouter Bancken <wouter.banc...@aca-it.be>
wrote:
The consumers in consumer group 'X' do not have a regex subscription
matching the newly created topic 'C'. They simply subscribe with
the subscribe(java.util.Collection<java.lang.String> topics) method on
topics 'A' and 'B'.
Shouldn't the consumer group have a different state from "Stable" during a
rebalancing regardless of the cause? How else can we determine the consumer
lag of the group during the rebalancing?
Best regards,
Wouter
Have a look at our brand NEW job website: jobs.aca-it.be !
*ACA IT-Solutions NV*
*HQ:* Herkenrodesingel 8B 2.01 | 3500 Hasselt
T +32(0)11 26 50 10 | F +32(0)11 26 50 11
www.aca-it.be | Twitter <https://twitter.com/dorien_jorissen> | Facebook
<http://www.facebook.com/pages/ACA-IT-Solutions/278520212202909> |
Linkedin <http://www.linkedin.com/company/6524>
On 5 February 2018 at 00:13, Hans Jespersen <h...@confluent.io> wrote:
Do the consumers in consumer group ‘X’ have a regex subscription that
matches the newly created topic ‘C’?
If they do then they will only discover this new topic once their ‘
metadata.max.age.ms’ metadata refresh interval has passed, which
defaults to 5 minutes.
metadata.max.age.ms The period of time in milliseconds after which
we force a refresh of metadata even if we haven't seen any partition
leadership changes to proactively discover any new brokers or partitions
-hans
On Feb 4, 2018, at 2:16 PM, Wouter Bancken <wouter.banc...@aca-it.be>
wrote:
Hi Hans,
Thanks for the response!
However, I get this result for all topics, not just for the newly
created
topic.
Situation sketch:
1. I have a consumer group 'X' subscribed to topics 'A' and 'B' with
partition assignments and lag information. Consumer group 'X' is
"Stable".
2a. Topic 'C' is (being) created.
2b. During this creation, I do not have a partition assignment for
consumer
group 'X' for topics 'A' and 'B' but the consumer group is still
"Stable".
3. A second later: I have a partition assignment for consumer group 'X'
for
topics 'A' and 'B' again and the consumer group is still "Stable".
I expected the state of consumer group 'X' during step 2b to be
"PreparingRebalance" or "AwaitingSync".
Best regards,
Wouter
On 4 February 2018 at 21:25, Hans Jespersen <h...@confluent.io> wrote:
I believe this is expected behavior.
If there are no subscriptions to a new topic, and therefor no partition
assignments, and definitely no committed offsets, then lag is an
undefined
concept. When the consumers subscribe to this new topic they may chose
to
start at the beginning or end of the commit log so the lag cannot be
predicted in advance.
-hans
On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wouter.banc...@aca-it.be
wrote:
Can anyone clarify if this is a bug in Kafka or the expected behavior?
Best regards,
Wouter
On 30 January 2018 at 21:04, Wouter Bancken <wouter.banc...@aca-it.be
wrote:
Hi,
I'm trying to write an external tool to monitor consumer lag on
Apache
Kafka.
For this purpose, I'm using the kafka-consumer-groups tool to fetch
the
consumer offsets.
When using this tool, partition assignments seem to be unavailable
temporarily during the creation of a new topic even if the consumer
group
has no subscription on this new topic. This seems to match the
documentation
<https://cwiki.apache.org/confluence/display/KAFKA/
Kafka+Client-side+Assignment+Proposal>
saying *"Topic metadata changes which have no impact on subscriptions
cause resync"*.
However, when this occurs I'd expect the state of the consumer to be
"PreparingRebalance" or "AwaitingSync" but it is simply "Stable".
Is this a bug in the tooling or is there a different way to obtain
the
correct offsets for a consumer group during a rebalance?
I'm using Kafka 10.2.1 but I haven't found any related issues in
recent
changelogs.
Best regards,
Wouter
