I'll explain. Say there are two topics, foo and bar. Foo has two partitions (foo-0 and foo-1). Bar has one partition (bar-0). The application uses one consumer group for all KafkaConsumers called "app". The application has two consumers always consuming from the two foo partitions, but creates/destroys a consumer for bar as needed. Connecting to bar to consume shouldn't cause a rebalance of foo, but because all three consumers use "app" as their group, it does.
One of my assumptions is that topics and rebalancing should operate independently from each other regardless of the group. Thanks, Cliff On Thu, Jan 7, 2016 at 3:44 PM, Jason Gustafson <ja...@confluent.io> wrote: > > > > If a new consumer joins an existing group, it triggers a rebalance even > > if it's > > consuming from a new topic (which matches my symptoms). > > > > Not sure I understand the issue. Rebalances are triggered when either 1) > group membership changes, 2) a consumer's subscription changes, or 3) when > the number of partitions for a topic changes. > > -Jason > > On Thu, Jan 7, 2016 at 1:38 PM, Cliff Rhyne <crh...@signal.co> wrote: > > > I'll give the 0.9.0 trunk a try. > > > > By the way, it looks to me that this might be a separate issue. I just > > setup unique group IDs for the stop/start topics from the always-on topic > > and my test passed. I think the issue is that the > > GroupCoordinator.doJoinGroup() only tracks the group and not the topic > when > > deciding whether to rebalance (GroupCoordinator.prepareRebalance()). If > a > > new consumer joins an existing group, it triggers a rebalance even if > it's > > consuming from a new topic (which matches my symptoms). > > > > What do you think? > > > > On Thu, Jan 7, 2016 at 3:27 PM, Jason Gustafson <ja...@confluent.io> > > wrote: > > > > > There have been bugs affecting both the client and the server. The one > I > > > mentioned above only affected the client, so you could try updating it > > > alone if that's easier, but it would be better to do both. > > > > > > I'll leave it to others to comment on the release timeline. I haven't > > seen > > > any major consumer-related bugs in the past couple weeks, so my feeling > > is > > > that it's starting to stabilize. It would be nice to get KIP-41 into > the > > > next release though. > > > > > > -Jason > > > > > > On Thu, Jan 7, 2016 at 1:18 PM, Cliff Rhyne <crh...@signal.co> wrote: > > > > > > > Hi Jason, > > > > > > > > I'm just on the 0.9.0.0 release. Are the fixes in the client, the > > kafka > > > > service, or both? I'll give it a try. > > > > > > > > Is there a timeline for when 0.9.0.1 would be released? > > > > > > > > Thanks, > > > > Cliff > > > > > > > > On Thu, Jan 7, 2016 at 3:14 PM, Jason Gustafson <ja...@confluent.io> > > > > wrote: > > > > > > > > > Hey Cliff, > > > > > > > > > > Are you using the 0.9.0.0 release? We've fixed a few problems in > the > > > > 0.9.0 > > > > > branch, some of which might explain the behavior you're seeing. > There > > > was > > > > > one bug in particular which resulted in the consumer not fetching > > data > > > > for > > > > > a set of partitions after a rebalance. > > > > > > > > > > -Jason > > > > > > > > > > On Thu, Jan 7, 2016 at 1:01 PM, Cliff Rhyne <crh...@signal.co> > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I'm testing out some changes with the 0.9.0.0 new KafkaConsumer > > API. > > > > We > > > > > > re-use the same consumer group ID across different components of > > the > > > > > > application (which consume from different topics). One topic is > > > always > > > > > > being consumed from, the rest are turned on and off. > > > > > > > > > > > > If I only run the always-on consumer, I have a very low > occurrence > > > rate > > > > > of > > > > > > the log message below: > > > > > > > > > > > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator - > > > > Attempt > > > > > > to heart beat failed since the group is rebalancing, try to > re-join > > > > > group. > > > > > > > > > > > > > > > > > > If I run both types of consumers, the log message occurs > frequently > > > and > > > > > the > > > > > > alway-on consumer eventually doesn't succeed in rejoining (I see > > the > > > > > > attempt in the logs to rejoin but nothing happens after that). I > > > only > > > > > have > > > > > > logs on the client side to work with; there's nothing showing up > in > > > the > > > > > > kafka logs to show why the group's state isn't stable. > > > > > > > > > > > > Thanks, > > > > > > Cliff > > > > > > > > > > > > -- > > > > > > Cliff Rhyne > > > > > > Software Engineering Lead > > > > > > e: crh...@signal.co > > > > > > signal.co > > > > > > ________________________ > > > > > > > > > > > > Cut Through the Noise > > > > > > > > > > > > This e-mail and any files transmitted with it are for the sole > use > > of > > > > the > > > > > > intended recipient(s) and may contain confidential and privileged > > > > > > information. Any unauthorized use of this email is strictly > > > prohibited. > > > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Cliff Rhyne > > > > Software Engineering Lead > > > > e: crh...@signal.co > > > > signal.co > > > > ________________________ > > > > > > > > Cut Through the Noise > > > > > > > > This e-mail and any files transmitted with it are for the sole use of > > the > > > > intended recipient(s) and may contain confidential and privileged > > > > information. Any unauthorized use of this email is strictly > prohibited. > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > -- > > Cliff Rhyne > > Software Engineering Lead > > e: crh...@signal.co > > signal.co > > ________________________ > > > > Cut Through the Noise > > > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. Any unauthorized use of this email is strictly prohibited. > > ©2015 Signal. All rights reserved. > > > -- Cliff Rhyne Software Engineering Lead e: crh...@signal.co signal.co ________________________ Cut Through the Noise This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use of this email is strictly prohibited. ©2015 Signal. All rights reserved.
