What Abhinav said. To give some context: the common cause of frequent
rebalances is that your consumer takes too long to process batches. As
long as you don't call into the consumer library, heartbeats aren't
sent so if you take too long working through a batch, the broker
things your consumer is gone and starts re-balancing. The message
batch under processing never gets marked as done, so after
rebalancing, things start over from the same spot.
So the solution is to either make the batches smaller or the heartbeat
interval longer. There are fancier solutions for when this doesn't
work, but it should do the trick for most normal cases.
On Fri, May 13, 2016 at 10:20 AM, Abhinav Solan <abhinav.so...@gmail.com> wrote:
> Hi Sahitya,
>
> Try reducing max.partition.fetch.bytes in your consumer.
> Then also increase heartbeat.interval.ms, this might help in to delay the
> consumer rebalance of your inbound process is taking more time than this
>
> - Abhinav
>
> On Fri, May 13, 2016 at 5:42 AM sahitya agrawal <sahitya2...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am using new Kafka consumer API ( 0.9.0.0) . I created 100 partitions of
>> a topic and started only one consumer to consume. Many of times , In
>> consumer logs I see lot of rebalancing activity and no object is consumed
>> due to that.
>>
>> Is this a known issue? Please let me know if some body can help with regard
>> to this.
>>
>> My Consumer config:
>> props.put("zookeeper.session.timeout.ms", "10000");
>> props.put("rebalance.backoff.ms","10000");
>> props.put("zookeeper.sync.time.ms","200");
>> props.put("rebalance.max.retries","10");
>> props.put("enable.auto.commit", "false");
>> props.put("consumer.timeout.ms","20000");
>> props.put("auto.offset.reset", "smallest");
>>
>> Thanks,
>> Sahitya
>>
--
Cees de Groot
Principal Software Engineer
PagerDuty, Inc.
