Lukasz, Thanks for the reply. I set max.poll.interval.ms to 1 hour and my consumer is polling with no issues every 500 ms and still I'm seeing the problem. I'm not worried about consumer dying and rebalancing instead after the rebalance it forgets it's previous state.
Thanks On Fri, Mar 20, 2020 at 23:54 Łukasz Antoniak <lukasz.anton...@gmail.com> wrote: > Ravi, > > It is not a bug. Broker assumes that your consumer faced live-lock. > You need to tune property max.poll.interval.ms to increase expected > interval between poll() on consumer side. > > -- Lukasz > > sob., 21 mar 2020 o 02:19 Ravi Kanth <ravikanth....@gmail.com> napisał(a): > > > Hi All, > > > > I have a Kafka Consumer that polls the data and gets *paused* for 15-20 > > mins for the post-processing of the polled records. However, during the > > time of pause, the broker assumes that the consumer group is dead(check > > below log) and rebalances the consumer group. > > > > *Application Log:* > > k8s-worker403: 2020-03-20 21:41:14.147 > [kafka-coordinator-heartbeat-thread > > | <consumer_group>] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer > > clientId=consumer-1, groupId=<consumer_group>] Marking the coordinator > > <broker>:9093 (id: 2147483480 rack: null) dead > > > > *Broker Log:* > > Member consumer-1-4eb939a6-c37d-4853-96dc-4255ec8663ff in group <consumer > > group> has failed, removing it from the group > > > > When rebalancing happens, consumer is automatically *resumed* and starts > > polling for more records though I am expecting it to be paused. This is > > causing unexpected behavior. > > > > When reading through Kafka docs, it looks like tuning up > > request.timeout.ms/session.timeout.ms will help avoid running into this > > problem. Please correct if my understanding is incorrect. However, I am > not > > willing to tweak the configurations, instead handle the problem at the > code > > level. > > > > *Question:* > > 1. Is this a bug in Kafka Java API letting the consumer be resumed after > > rebalancing even when it is initially paused? If so, details of the bug > > would be helpful. > > 2. Is there anyother way to avoid running into this problem without > > tweaking configurations? > > > > *Kafka Version Using:* > > *1.0.1* > > > > Any suggestions would be of great help. > > > > Thanks, > > Ravi > > >
