I'll give the 0.9.0 trunk a try. By the way, it looks to me that this might be a separate issue. I just setup unique group IDs for the stop/start topics from the always-on topic and my test passed. I think the issue is that the GroupCoordinator.doJoinGroup() only tracks the group and not the topic when deciding whether to rebalance (GroupCoordinator.prepareRebalance()). If a new consumer joins an existing group, it triggers a rebalance even if it's consuming from a new topic (which matches my symptoms).
What do you think? On Thu, Jan 7, 2016 at 3:27 PM, Jason Gustafson <ja...@confluent.io> wrote: > There have been bugs affecting both the client and the server. The one I > mentioned above only affected the client, so you could try updating it > alone if that's easier, but it would be better to do both. > > I'll leave it to others to comment on the release timeline. I haven't seen > any major consumer-related bugs in the past couple weeks, so my feeling is > that it's starting to stabilize. It would be nice to get KIP-41 into the > next release though. > > -Jason > > On Thu, Jan 7, 2016 at 1:18 PM, Cliff Rhyne <crh...@signal.co> wrote: > > > Hi Jason, > > > > I'm just on the 0.9.0.0 release. Are the fixes in the client, the kafka > > service, or both? I'll give it a try. > > > > Is there a timeline for when 0.9.0.1 would be released? > > > > Thanks, > > Cliff > > > > On Thu, Jan 7, 2016 at 3:14 PM, Jason Gustafson <ja...@confluent.io> > > wrote: > > > > > Hey Cliff, > > > > > > Are you using the 0.9.0.0 release? We've fixed a few problems in the > > 0.9.0 > > > branch, some of which might explain the behavior you're seeing. There > was > > > one bug in particular which resulted in the consumer not fetching data > > for > > > a set of partitions after a rebalance. > > > > > > -Jason > > > > > > On Thu, Jan 7, 2016 at 1:01 PM, Cliff Rhyne <crh...@signal.co> wrote: > > > > > > > Hi, > > > > > > > > I'm testing out some changes with the 0.9.0.0 new KafkaConsumer API. > > We > > > > re-use the same consumer group ID across different components of the > > > > application (which consume from different topics). One topic is > always > > > > being consumed from, the rest are turned on and off. > > > > > > > > If I only run the always-on consumer, I have a very low occurrence > rate > > > of > > > > the log message below: > > > > > > > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator - > > Attempt > > > > to heart beat failed since the group is rebalancing, try to re-join > > > group. > > > > > > > > > > > > If I run both types of consumers, the log message occurs frequently > and > > > the > > > > alway-on consumer eventually doesn't succeed in rejoining (I see the > > > > attempt in the logs to rejoin but nothing happens after that). I > only > > > have > > > > logs on the client side to work with; there's nothing showing up in > the > > > > kafka logs to show why the group's state isn't stable. > > > > > > > > Thanks, > > > > Cliff > > > > > > > > -- > > > > Cliff Rhyne > > > > Software Engineering Lead > > > > e: crh...@signal.co > > > > signal.co > > > > ________________________ > > > > > > > > Cut Through the Noise > > > > > > > > This e-mail and any files transmitted with it are for the sole use of > > the > > > > intended recipient(s) and may contain confidential and privileged > > > > information. Any unauthorized use of this email is strictly > prohibited. > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > -- > > Cliff Rhyne > > Software Engineering Lead > > e: crh...@signal.co > > signal.co > > ________________________ > > > > Cut Through the Noise > > > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. Any unauthorized use of this email is strictly prohibited. > > ©2015 Signal. All rights reserved. > > > -- Cliff Rhyne Software Engineering Lead e: crh...@signal.co signal.co ________________________ Cut Through the Noise This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use of this email is strictly prohibited. ©2015 Signal. All rights reserved.
