Consumer stops consuming after 5 gb

2013-07-23 Thread Anurup Raveendran
After consuming about 5 gb of messages , it stopped consuming and it got stuck at a particular offset. After running the following command I got an error regarding the broker. $ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group air-DummyProductionConsumerGroup-After --zkconnect 1

Re: kafka multi threaded consumer with very low throughput

2013-07-09 Thread Anurup Raveendran
got amazing results . thanks On Tue, Jul 9, 2013 at 9:11 AM, Jun Rao wrote: > Try producing at least 100s MB of data and rerun you test. > > Thanks, > > Jun > > > On Mon, Jul 8, 2013 at 8:04 AM, Anurup Raveendran < > anurup.raveend...@fluturasolutions.com> wr

Re: kafka multi threaded consumer with very low throughput

2013-07-08 Thread Anurup Raveendran
ad going to sleep for 10 seconds"); Thread.sleep(900); } catch (InterruptedException ie) { System.out.println("\n\nNote : Execution Interrupted..!!\n\n"); } System.out.println("main thread woke up"); example.shutdown(); } }

Re: kafka multi threaded consumer with very low throughput

2013-07-08 Thread Anurup Raveendran
305 bytes On Mon, Jul 8, 2013 at 8:41 PM, Tom Brown wrote: > What is the size of each message? > > --Tom > > > On Mon, Jul 8, 2013 at 9:04 AM, Anurup Raveendran < > anurup.raveend...@fluturasolutions.com> wrote: > > > I have 2 kafka brokers running on two

kafka multi threaded consumer with very low throughput

2013-07-08 Thread Anurup Raveendran
I have 2 kafka brokers running on two systems with the same configuration CPU - Dual Core RAM - 4 GB I'm trying to benchmark my kafka setup Number of messages - 1 for 2 partitions & 2 threads configuration - consumer consumes in 1.175 seconds for 1 partition & 1 thread configuration - consu