Lisheng , Issue not with fetch max bytes as same message start processing after restarting the consumer
Regards, Sampath On Wed, Aug 21, 2019 at 4:30 PM Lisheng Wang <wanglishen...@gmail.com> wrote: > Hi Sampath > > Can you confirm that "fetch.max.bytes" on consumer is not smaller than > "message.max.bytes" on broker? > > Maybe need you check consumer log to see if can find any clue once you > enable it. if no any error/exception found on consumer side, maybe need > change log level to "debug" to get more detail information. > > Best, > Lisheng > > > sampath kumar <sampath...@gmail.com> 于2019年8月21日周三 下午6:30写道： > > > Hi Lisheng, > > > > Thanks for the response. > > > > Right now we have enabled info in the broker However logs not enabled for > > the consumer client will enable it. > > > > Yes, when we manually stop and start the consumer in affected > microservice > > instance rebalance triggers and consuming resumes. > > > > And in Broker side consumer client status is healthy we verified both in > > Kafka Manager and Consumer Group Command in Broker CLI, so I guess > > heartbeat is not the issue and this issue not affected to not complete > > consumer group only some consumer client in the couple of microservice. > > > > Forex: if one of the Service if we have 38 consumer client/thread > > registered for the consumer group, only 1 client not receiving the > > messages, rest all getting the messages > > > > Anything else you want me to check here? > > > > Regards, > > Sampath > > > > > > > > > > > > > > On Wed, Aug 21, 2019 at 3:21 PM Lisheng Wang <wanglishen...@gmail.com> > > wrote: > > > > > May i know what log level did you configured on consumer and broker? > > > you say it will resume when rebalance happen, so consumer is alive, > can > > > you see any heartbeat information in consumer log? > > > > > > Best, > > > Lisheng > > > > > > > > > sampath kumar <sampath...@gmail.com> 于2019年8月21日周三 下午5:23写道： > > > > > > > Hi, > > > > > > > > Using Broker 5.3.0, new consumers(Consumers managed by brokers). > > Brokers > > > > are deployed in a Kubernetes environment > > > > > > > > Number of brokers : 3, Number of 3 Zookeeper setup > > > > > > > > One of the Topic "inventory.request" we have 3 replication, with > insync > > > > replicas configured as 2 and partition count is 1024 > > > > > > > > We have 20 instances of microservice subscribe to the above topic, > each > > > > instance will have 48 consumers registered as a group > "agent.group.inv" > > > > > > > > Issue : > > > > > > > > Here some times a couple of the consumers suddenly stopped receiving > > the > > > > request, and lag seems to keep increasing. Only option to recover it > > > > restart the consumers and invokes rebalancing > > > > > > > > ``` agent.group.inv inventory.request 543 17423 > 17612 > > > > 189 > > agent19.inv.35-6e6eb252-8d26-489b-8d7f-53b25f182f30 > > > > /10.200.187.103 agent19.inv.35 ``` > > > > > > > > we checked the thread dump of the consumer, the consumer keeps > > performing > > > > polling and assigned with partitions, However not receiving the any > > > > messages > > > > > > > > ``` "inventory.request-agent19.inv.35" #499 prio=1 os_prio=4 > > > > tid=0x00007f88a855b000 nid=0x389 runnable [0x00007f87e8be6000] > > > > java.lang.Thread.State: RUNNABLE > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > > at > > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > > > > at > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > > > > at > > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > > > > - locked <0x00000000aa502730> (a sun.nio.ch.Util$3) > > > > - locked <0x00000000aa5026b0> (a > > > > java.util.Collections$UnmodifiableSet) > > > > - locked <0x00000000aa502668> (a > sun.nio.ch.EPollSelectorImpl) > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > > > > at > > > > org.apache.kafka.common.network.Selector.select(Selector.java:794) > > > > at > > > org.apache.kafka.common.network.Selector.poll(Selector.java:467) > > > > at > > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:539) > > > > at > > > > > > > > > > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262) > > > > at > > > > > > > > > > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233) > > > > at > > > > > > > > > > > > > > org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1281) > > > > at > > > > > > > > > > > > > > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225) > > > > at > > > > > > > > > > > > > > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159) > > > > ``` > > > > > > > > > > > > No errors are observed consumer client, brokers and also resource > issue > > > not > > > > seen. > > > > > > > > Can you please help us in identifying the root cause for this > consumer > > > > client behavior? > > > > > > > > Please let me know if any other details required? > > > > > > > > -- > > > > Regards, > > > > Sampath > > > > > > > > > > > > > -- > > Regards, > > Sampath > > > -- Regards, Sampath
