Re: Kafka Consumer stops consuming from a topic

2016-07-23 Thread OGrandeDiEnne
Mmh... Some time ago we had an issue with Kafka 0.8.x The consumer was extremely slow ( the CPU was sucked up by other processes) and it was not picking up any message. Looking at Zookeeper we saw the offset was committed as the messages were already read by the consumer. We disabled auto back an

Re: Kafka Consumer stops consuming from a topic

2016-07-19 Thread Abhinav Solan
Acks all ... Having one Kafka broker only On Tue, Jul 19, 2016, 9:22 AM David Garcia wrote: > Ah ok. Another dumb question: what about acks? Are you using auto-ack? > > On 7/19/16, 10:00 AM, "Abhinav Solan" wrote: > > If I add 2 more nodes and make it a cluster .. would that help ? Have >

Re: Kafka Consumer stops consuming from a topic

2016-07-19 Thread David Garcia
Ah ok. Another dumb question: what about acks? Are you using auto-ack? On 7/19/16, 10:00 AM, "Abhinav Solan" wrote: If I add 2 more nodes and make it a cluster .. would that help ? Have searched forums and all this kind of thing is not there ... If we have a cluster then might be

Re: Kafka Consumer stops consuming from a topic

2016-07-19 Thread Abhinav Solan
If I add 2 more nodes and make it a cluster .. would that help ? Have searched forums and all this kind of thing is not there ... If we have a cluster then might be Kafka Server has a backup option and it self heals from this behavior ... Just a theory On Tue, Jul 19, 2016, 7:57 AM Abhinav Solan

Re: Kafka Consumer stops consuming from a topic

2016-07-19 Thread Abhinav Solan
No, was monitoring the app at that time .. it was just sitting idle On Tue, Jul 19, 2016, 7:32 AM David Garcia wrote: > Is it possible that your app is thrashing (i.e. FullGC’ing too much and > not processing messages)? > > -David > > On 7/19/16, 9:16 AM, "Abhinav Solan" wrote: > > Hi Every

Re: Kafka Consumer stops consuming from a topic

2016-07-19 Thread David Garcia
Is it possible that your app is thrashing (i.e. FullGC’ing too much and not processing messages)? -David On 7/19/16, 9:16 AM, "Abhinav Solan" wrote: Hi Everyone, can anyone help me on this Thanks, Abhinav On Mon, Jul 18, 2016, 6:19 PM Abhinav Solan wrote: >

Re: Kafka Consumer stops consuming from a topic

2016-07-19 Thread Abhinav Solan
Hi Everyone, can anyone help me on this Thanks, Abhinav On Mon, Jul 18, 2016, 6:19 PM Abhinav Solan wrote: > Hi Everyone, > > Here are my settings > Using Kafka 0.9.0.1, 1 instance (as we are testing things on a staging > environment) > Subscribing to 4 topics from a single Consumer application

Kafka Consumer stops consuming from a topic

2016-07-18 Thread Abhinav Solan
Hi Everyone, Here are my settings Using Kafka 0.9.0.1, 1 instance (as we are testing things on a staging environment) Subscribing to 4 topics from a single Consumer application with 4 threads Now the server keeps on working fine for a while, then after about 3-4 hrs or so, it stops consuming at a