Using separate groups would be only a slight inconvenience but is entirely doable. It sounds like that's the preferred model.
Thanks for the explanation. On Thu, Jan 7, 2016 at 4:08 PM, Jason Gustafson <ja...@confluent.io> wrote: > Thanks for the explanation. Unfortunately, the consumer doesn't work that > way. Rebalances affect all members of the group regardless of > subscriptions. Partial rebalances could be an interesting idea to consider > for the future, but we haven't had any cases where a group had differing > subscriptions in a steady state. Usually users just use separate groups. > Would that not make sense in this case? > > -Jason > > On Thu, Jan 7, 2016 at 1:54 PM, Cliff Rhyne <crh...@signal.co> wrote: > > > I'll explain. > > > > Say there are two topics, foo and bar. Foo has two partitions (foo-0 and > > foo-1). Bar has one partition (bar-0). The application uses one > consumer > > group for all KafkaConsumers called "app". The application has two > > consumers always consuming from the two foo partitions, but > > creates/destroys a consumer for bar as needed. Connecting to bar to > > consume shouldn't cause a rebalance of foo, but because all three > consumers > > use "app" as their group, it does. > > > > One of my assumptions is that topics and rebalancing should operate > > independently from each other regardless of the group. > > > > Thanks, > > Cliff > > > > On Thu, Jan 7, 2016 at 3:44 PM, Jason Gustafson <ja...@confluent.io> > > wrote: > > > > > > > > > > If a new consumer joins an existing group, it triggers a rebalance > even > > > > if it's > > > > consuming from a new topic (which matches my symptoms). > > > > > > > > > > Not sure I understand the issue. Rebalances are triggered when either > 1) > > > group membership changes, 2) a consumer's subscription changes, or 3) > > when > > > the number of partitions for a topic changes. > > > > > > -Jason > > > > > > On Thu, Jan 7, 2016 at 1:38 PM, Cliff Rhyne <crh...@signal.co> wrote: > > > > > > > I'll give the 0.9.0 trunk a try. > > > > > > > > By the way, it looks to me that this might be a separate issue. I > just > > > > setup unique group IDs for the stop/start topics from the always-on > > topic > > > > and my test passed. I think the issue is that the > > > > GroupCoordinator.doJoinGroup() only tracks the group and not the > topic > > > when > > > > deciding whether to rebalance (GroupCoordinator.prepareRebalance()). > > If > > > a > > > > new consumer joins an existing group, it triggers a rebalance even if > > > it's > > > > consuming from a new topic (which matches my symptoms). > > > > > > > > What do you think? > > > > > > > > On Thu, Jan 7, 2016 at 3:27 PM, Jason Gustafson <ja...@confluent.io> > > > > wrote: > > > > > > > > > There have been bugs affecting both the client and the server. The > > one > > > I > > > > > mentioned above only affected the client, so you could try updating > > it > > > > > alone if that's easier, but it would be better to do both. > > > > > > > > > > I'll leave it to others to comment on the release timeline. I > haven't > > > > seen > > > > > any major consumer-related bugs in the past couple weeks, so my > > feeling > > > > is > > > > > that it's starting to stabilize. It would be nice to get KIP-41 > into > > > the > > > > > next release though. > > > > > > > > > > -Jason > > > > > > > > > > On Thu, Jan 7, 2016 at 1:18 PM, Cliff Rhyne <crh...@signal.co> > > wrote: > > > > > > > > > > > Hi Jason, > > > > > > > > > > > > I'm just on the 0.9.0.0 release. Are the fixes in the client, > the > > > > kafka > > > > > > service, or both? I'll give it a try. > > > > > > > > > > > > Is there a timeline for when 0.9.0.1 would be released? > > > > > > > > > > > > Thanks, > > > > > > Cliff > > > > > > > > > > > > On Thu, Jan 7, 2016 at 3:14 PM, Jason Gustafson < > > ja...@confluent.io> > > > > > > wrote: > > > > > > > > > > > > > Hey Cliff, > > > > > > > > > > > > > > Are you using the 0.9.0.0 release? We've fixed a few problems > in > > > the > > > > > > 0.9.0 > > > > > > > branch, some of which might explain the behavior you're seeing. > > > There > > > > > was > > > > > > > one bug in particular which resulted in the consumer not > fetching > > > > data > > > > > > for > > > > > > > a set of partitions after a rebalance. > > > > > > > > > > > > > > -Jason > > > > > > > > > > > > > > On Thu, Jan 7, 2016 at 1:01 PM, Cliff Rhyne <crh...@signal.co> > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I'm testing out some changes with the 0.9.0.0 new > KafkaConsumer > > > > API. > > > > > > We > > > > > > > > re-use the same consumer group ID across different components > > of > > > > the > > > > > > > > application (which consume from different topics). One topic > > is > > > > > always > > > > > > > > being consumed from, the rest are turned on and off. > > > > > > > > > > > > > > > > If I only run the always-on consumer, I have a very low > > > occurrence > > > > > rate > > > > > > > of > > > > > > > > the log message below: > > > > > > > > > > > > > > > > > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator - > > > > > > Attempt > > > > > > > > to heart beat failed since the group is rebalancing, try to > > > re-join > > > > > > > group. > > > > > > > > > > > > > > > > > > > > > > > > If I run both types of consumers, the log message occurs > > > frequently > > > > > and > > > > > > > the > > > > > > > > alway-on consumer eventually doesn't succeed in rejoining (I > > see > > > > the > > > > > > > > attempt in the logs to rejoin but nothing happens after > > that). I > > > > > only > > > > > > > have > > > > > > > > logs on the client side to work with; there's nothing showing > > up > > > in > > > > > the > > > > > > > > kafka logs to show why the group's state isn't stable. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Cliff > > > > > > > > > > > > > > > > -- > > > > > > > > Cliff Rhyne > > > > > > > > Software Engineering Lead > > > > > > > > e: crh...@signal.co > > > > > > > > signal.co > > > > > > > > ________________________ > > > > > > > > > > > > > > > > Cut Through the Noise > > > > > > > > > > > > > > > > This e-mail and any files transmitted with it are for the > sole > > > use > > > > of > > > > > > the > > > > > > > > intended recipient(s) and may contain confidential and > > privileged > > > > > > > > information. Any unauthorized use of this email is strictly > > > > > prohibited. > > > > > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Cliff Rhyne > > > > > > Software Engineering Lead > > > > > > e: crh...@signal.co > > > > > > signal.co > > > > > > ________________________ > > > > > > > > > > > > Cut Through the Noise > > > > > > > > > > > > This e-mail and any files transmitted with it are for the sole > use > > of > > > > the > > > > > > intended recipient(s) and may contain confidential and privileged > > > > > > information. Any unauthorized use of this email is strictly > > > prohibited. > > > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Cliff Rhyne > > > > Software Engineering Lead > > > > e: crh...@signal.co > > > > signal.co > > > > ________________________ > > > > > > > > Cut Through the Noise > > > > > > > > This e-mail and any files transmitted with it are for the sole use of > > the > > > > intended recipient(s) and may contain confidential and privileged > > > > information. Any unauthorized use of this email is strictly > prohibited. > > > > ©2015 Signal. All rights reserved. > > > > > > > > > > > > > > > -- > > Cliff Rhyne > > Software Engineering Lead > > e: crh...@signal.co > > signal.co > > ________________________ > > > > Cut Through the Noise > > > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. Any unauthorized use of this email is strictly prohibited. > > ©2015 Signal. All rights reserved. > > > -- Cliff Rhyne Software Engineering Lead e: crh...@signal.co signal.co ________________________ Cut Through the Noise This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use of this email is strictly prohibited. ©2015 Signal. All rights reserved.
