Is your application possibly timing out its zookeeper connection during consumption while doing its processing, thus triggering the rebalance?
-Clark On 8/6/14, 11:18 PM, "Jason Rosenberg" <j...@squareup.com> wrote: >We've noticed that some of our consumers are more likely to repeatedly >trigger rebalancing when the app is consuming messages more slowly (e.g. >persisting data to back-end systems, etc.). > >If on the other hand we 'fast-forward' the consumer (which essentially >means we tell it to consume but do nothing with the messages until all >caught up), it will never decide to do a rebalance during this time. So >it >can go hours without rebalancing while fast forwarding and consuming super >fast, while during normal processing, it might decide to rebalance every >minute or so. > >Is there any simple explanation for this? > >Usually the trigger for rebalance logged is that a "topic info for path X >has changed to Y, triggering rebalance". > >Thanks for any ideas. > >We'd like to reduce the rebalancing, as it essentially slows down >consumption each time it happens. > >Thanks > >Jason
