Re: kafka multi threaded consumer with very low throughput

2013-07-09 Thread Anurup Raveendran
got amazing results . thanks On Tue, Jul 9, 2013 at 9:11 AM, Jun Rao wrote: > Try producing at least 100s MB of data and rerun you test. > > Thanks, > > Jun > > > On Mon, Jul 8, 2013 at 8:04 AM, Anurup Raveendran < > anurup.raveend...@fluturasolutions.com> wrote: > > > I have 2 kafka brokers

Re: kafka multi threaded consumer with very low throughput

2013-07-08 Thread Jun Rao
Try producing at least 100s MB of data and rerun you test. Thanks, Jun On Mon, Jul 8, 2013 at 8:04 AM, Anurup Raveendran < anurup.raveend...@fluturasolutions.com> wrote: > I have 2 kafka brokers running on two systems with the same configuration > > CPU - Dual Core > RAM - 4 GB > > I'm trying

Re: kafka multi threaded consumer with very low throughput

2013-07-08 Thread Anurup Raveendran
here is the source code of the program package kafka.examples; import kafka.consumer.ConsumerConfig; import kafka.consumer.KafkaStream; import kafka.javaapi.consumer.ConsumerConnector; import kafka.message.Message; import kafka.consumer.ConsumerIterator; import java.util.HashMap; import java.uti

Re: kafka multi threaded consumer with very low throughput

2013-07-08 Thread Anurup Raveendran
305 bytes On Mon, Jul 8, 2013 at 8:41 PM, Tom Brown wrote: > What is the size of each message? > > --Tom > > > On Mon, Jul 8, 2013 at 9:04 AM, Anurup Raveendran < > anurup.raveend...@fluturasolutions.com> wrote: > > > I have 2 kafka brokers running on two systems with the same configuration > >

Re: kafka multi threaded consumer with very low throughput

2013-07-08 Thread Tom Brown
What is the size of each message? --Tom On Mon, Jul 8, 2013 at 9:04 AM, Anurup Raveendran < anurup.raveend...@fluturasolutions.com> wrote: > I have 2 kafka brokers running on two systems with the same configuration > > CPU - Dual Core > RAM - 4 GB > > I'm trying to benchmark my kafka setup > Nu

kafka multi threaded consumer with very low throughput

2013-07-08 Thread Anurup Raveendran
I have 2 kafka brokers running on two systems with the same configuration CPU - Dual Core RAM - 4 GB I'm trying to benchmark my kafka setup Number of messages - 1 for 2 partitions & 2 threads configuration - consumer consumes in 1.175 seconds for 1 partition & 1 thread configuration - consu