Can anyone here help me please? I am at my wit's end. I now have max.poll.records set to just 2. Still I am getting Auto offset commit failed warning. Log file is getting full because of this warning. Session timeout is 5 minutes, max.poll.interval.ms is 10 minutes.
On Wed, May 23, 2018 at 12:42 PM Shantanu Deshmukh <shantanu...@gmail.com> wrote: > > Hello, > > We have a 3 broker Kafka 0.10.0.1 cluster. There we have 3 topics with 10 > partitions each. We have an application which spawns threads as consumers. > We spawn 5 consumers for each topic. I am observing that consider group > randomly keeps rebalancing. Then many times we see logs saying "Revoking > partitions for". This happens almost every 10 minutes. Consumption during > this time completely stops. > > I have applied this configuration > max.poll.records 20 > heartbeat.interval.ms 10000 > Session.timeout.ms 6000 > > Still this did not help. Strange thing is I observed consumer writing logs > saying "auto commit failed because poll() loop spent too much time > processing records" even when there was no data in partition to process. We > have polling interval of 500 ms, specified as argument in poll(). Initially > I had set same consumer group for all three topics' consumers. Then I > specified different CGs for different topics' consumers. Even this is not > helping. > > I am trying to search over the web, checked my code, tried many > combinations of configuration but still no luck. Please help me. > > Thanks & Regards, > > Shantanu Deshmukh >
