Hi,
I believe what you sent was the broker log not the client log - client log
would be useful to debug the issue. Also, the configuration you provided
seems to be your application log - there are a few configuration I'm not
sure what they actually mean, e.g. delay, timeout.
P
On Wed, Feb 7, 2024 at 11:49 PM Jain, Ankit <ankit.j...@unisys.com> wrote:
> Update the below response.
>
>
>
> *From:* Marigowda, Santhosh Aditya <santhosh.marigo...@unisys.com>
> *Sent:* Thursday, February 8, 2024 1:14 PM
> *To:* Philip Nee <philip...@gmail.com>; dev@kafka.apache.org
> *Cc:* Jain, Ankit <ankit.j...@unisys.com>; Namboodiri, Vishnu <
> vishnu.nambood...@in.unisys.com>; Mudlapur, Rajesh <
> rajesh.mudla...@au.unisys.com>; Reddy, Thanmai <thanmai.re...@unisys.com>;
> Marigowda, Santhosh Aditya <santhosh.marigo...@unisys.com>
> *Subject:* RE: Kafka consumer group crashing and not able to consume once
> service is up
>
>
>
> Hi Philip,
>
> Thanks for your queries, Please find my response.
>
>
>
> Do you have a single-member consumer group? Yes, We have multiple
> consumer groups (CG1, CG2, CG3, CG4) with single member.
>
>
>
> Did you also mean after restarting the "service", the service was not able
> to resume consuming the messages? Yes, Not able to resume.
>
>
>
> could you also include the client version : 3.5.0
>
>
>
>
>
> Logs would be very helpful to see why these consumers stopped consuming
> messages: If any message is available in the topic will be subscribed and
> printed below in the below logs "Entered
> ReplicationDataListener.onMessage method".
>
> In our case below logs are not at all present in the service logs.
>
> I have attached Zookeper and Kafka logs for reference.
>
>
>
>
>
> --Code to consume messages
>
> public void onMessage(Vertx vertx, KafkaConsumer consumer, String jsonObject)
> {
> *logger*.info("Entered ReplicationDataListener.onMessage method");
> initialize(vertx);
> service = SyncServiceImpl.*getInstance*(vertx);
> try {
>
> }
>
>
>
>
>
>
>
> The setup looks like below.
>
>
>
>
>
> Please refer below image when *service1* shuts down/crashes down. After
> bringing up *Service1 *none of the messages being consumed by *service1*.
>
>
>
>
>
>
>
> Workaround solution, Rename consumer name from *CG1* to *CG5* then
> restart *Service1, *messages are started consuming.
>
>
>
>
>
>
>
> Thanks,
>
> Santhosh Aditya
>
>
>
> *From:* Reddy, Thanmai <thanmai.re...@unisys.com>
> *Sent:* Thursday, February 8, 2024 11:33 AM
> *To:* Marigowda, Santhosh Aditya <santhosh.marigo...@unisys.com>
> *Subject:* FW: Kafka consumer group crashing and not able to consume once
> service is up
>
>
>
>
>
>
>
> *From:* Philip Nee <philip...@gmail.com>
> *Sent:* Thursday, February 8, 2024 11:26 AM
> *To:* dev@kafka.apache.org
> *Cc:* Jain, Ankit <ankit.j...@unisys.com>; Namboodiri, Vishnu <
> vishnu.nambood...@in.unisys.com>; Mudlapur, Rajesh <
> rajesh.mudla...@au.unisys.com>; Reddy, Thanmai <thanmai.re...@unisys.com>
> *Subject:* Re: Kafka consumer group crashing and not able to consume once
> service is up
>
>
>
> Hi Santhosh, Your problem statement confuses me a bit (apologize). You
> mentioned "if one of the kafka consumer(Service)" - Do you have a single
> member consumer group? Could you elaborate on the setup a bit? Did you also
> mean after
>
> Hi Santhosh,
>
>
>
> Your problem statement confuses me a bit (apologize). You mentioned "if
> one of the kafka consumer(Service)" - Do you have a single member consumer
> group? Could you elaborate on the setup a bit? Did you also mean after
> restarting the "service", the service was not able to resume from consuming
> the messages?
>
>
>
> Logs would be very helpful to see why these consumers stopped consuming
> messages. If you include the logs (that would be really helpful), could you
> also include the client version.
>
>
>
> P
>
>
>
> On Wed, Feb 7, 2024 at 9:45 PM Marigowda, Santhosh Aditya <
> santhosh.marigo...@unisys.com.invalid> wrote:
>
> Hi Dev team,
>
> Could you Please let me know if any updates on my query.
>
>
>
> Thanks,
>
> Santhosh Aditya
>
>
>
> *From:* Marigowda, Santhosh Aditya
> *Sent:* Wednesday, January 31, 2024 8:50 AM
> *To:* dev@kafka.apache.org
> *Cc:* Jain, Ankit <ankit.j...@unisys.com>; Namboodiri, Vishnu <
> vishnu.nambood...@in.unisys.com>; Mudlapur, Rajesh <
> rajesh.mudla...@au.unisys.com>; Reddy, Thanmai <thanmai.re...@unisys.com>
> *Subject:* RE: Kafka consumer group crashing and not able to consume once
> service is up
>
>
>
> Thanks Matthias for your response. Sorry, my bad please find the updated
> Image.
>
> · We have 1 topic and 4 partitions, Each consumer is pointing to the
> topic. There is no partition assignment specific to the consumer group from
> our software code.
>
> · Please let me know if you need Kafka server logs or consumer service
> logs, Post restart of our service no logs are printed in onMessage method.
>
> · We are using consumer.subscribe(topic)
>
>
>
>
>
>
>
>
>
> Thanks,
>
> Santhosh Aditya
>
>
>
> *From:* Matthias J. Sax <mj...@apache.org>
> *Sent:* Wednesday, January 31, 2024 12:13 AM
> *To:* dev@kafka.apache.org
> *Subject:* Re: Kafka consumer group crashing and not able to consume once
> service is up
>
>
>
> I am a not sure if I can follow completely. From the figures you show, you
> have a topic with 4 partitions, and 4 consumer groups. Thus, each consumer
> group should read all 4 partitions, but the figure indicate that each group
> would read a single
>
> I am a not sure if I can follow completely.
>
>
>
> From the figures you show, you have a topic with 4 partitions, and 4
>
> consumer groups. Thus, each consumer group should read all 4 partitions,
>
> but the figure indicate that each group would read a single partition only?
>
>
>
> Can you clarify? Are you using `consumer.subscribe` or `consumer.assign`?
>
>
>
> In general, might be good too collect some INFO (or DEBUG) level logs
>
> for the crashing service after restart to see what it's doing.
>
>
>
>
>
> -Matthias
>
>
>
> On 1/30/24 7:17 AM, Marigowda, Santhosh Aditya wrote:
>
> > Hi Kafka Dev Team,
>
> >
>
> > Could you please help us with our problem.
>
> >
>
> > In our POC, if one of the kafka consumer(Service) shuts down or crashes
>
> > then post restart of service none of the messages are getting consumed
>
> > by the crashed Service.
>
> >
>
> > Other services are consuming without any issues.
>
> >
>
> > One of service crash/Shutdown
>
> >
>
> > If we rename the Kafka consumer group name and start the service then
>
> > messages start consuming.
>
> >
>
> > Consumer configuration :
>
> >
>
> > {
>
> >
>
> > delay:1000
>
> >
>
> > timeout:0
>
> >
>
> > topic-name= test
>
> >
>
> > handlers=["Listener"]
>
> >
>
> > source="kafka-consumer"
>
> >
>
> > enable_auto_commit="false"
>
> >
>
> > group="Consumer-Group-1"
>
> >
>
> > }
>
> >
>
> > local-kafka-consumer = {
>
> >
>
> > server=
>
> > "{kafka-hostname}:{kafka-port}"
>
> >
>
> >
>
> > deserializer="org.apache.kafka.common.serialization.StringDeserializer"
>
> >
>
> > auto_offset_reset="latest"
>
> >
>
> > enable_auto_commit="true"
>
> >
>
> > maxRequestSize=20971520
>
> >
>
> > }
>
> >
>
> > Thanks,
>
> >
>
> > Santhosh Aditya
>
> >
>
>
