Re: kafka consumer threads are in waiting state and lag is getting increased

2014-04-15 Thread Arjun
There can be different reasons, Can you check the kafka consumer log and see if there is any "conflict in". And can u check weather, the console consumer is working fine or not. Arjun Narasimha Kota On Tuesday 15 April 2014 12:34 PM, ankit tyagi wrote: I have increased the partition for parall

Re: kafka consumer threads are in waiting state and lag is getting increased

2014-04-15 Thread ankit tyagi
I have one more doubt. The Kafka cluster retains all published messages—whether or not they have been consumed—for a configurable period of time. For example if the log retention is set to two days, *then for the two days after a message is published it is available for consumption, after which it

Re: kafka consumer threads are in waiting state and lag is getting increased

2014-04-15 Thread ankit tyagi
I have increased the partition for parallelism but my concern is , if message are present in partition then why consumer thread are blocked at below condition. *KafkaConsumer-24" prio=10 tid=0x7f6da5726800 nid=0x3c8a waiting on condition [0x7f6d4f8f7000] java.lang.Thread.State: WAITING (p

Re: kafka consumer threads are in waiting state and lag is getting increased

2014-04-14 Thread Arjun
Hi, Can you please check weather this is the situation https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped,why? Arjun Kota On Tuesday 15 April 2014 11:49 AM, ankit tyagi wrote: Hi. currently we are using *kafka_2.8.0-0.8.0-beta1 and *and high level consumer

kafka consumer threads are in waiting state and lag is getting increased

2014-04-14 Thread ankit tyagi
Hi. currently we are using *kafka_2.8.0-0.8.0-beta1 and *and high level consumer group to consume messages. Topic has been created with 3 replica and 100 partition so that max 100 threads can consume messages simultaneously, but i am seeing that mostly threads are in waiting state and lag is getti