Getting the latest message always from Kafka

2013-05-17 Thread Kishore V. Kopalle
Hi Neha, I am sorry about the design of this program, but what I am trying to do is to expose API to read and write from Kafka. I have some unit test cases as given below. The problem is that the first test case passes but the second test case fails saying that : org.junit.ComparisonFailure: expec

Re: Time difference between message fetch and message send is very high (~900-1200ms)

2013-05-17 Thread Kishore V. Kopalle
interval on the broker but that will drive up iops. > > Thanks, > Neha > On May 16, 2013 11:39 PM, "Kishore V. Kopalle" > wrote: > > > Hi Frank, > > > > Here are the explicitly set properties: > > Consumer: > > Properties props = ne

Re: Kafka performance

2013-05-17 Thread Kishore V. Kopalle
The timezone we are operating is Indian Standard Time (IST). So you will see times like 2013-05-17 11:54:02.973 in the messages below. On Fri, May 17, 2013 at 12:59 PM, Kishore V. Kopalle < kish...@greenmedsoft.com> wrote: > Hi Mathias, > > Yes they are. But we are talking abo

Re: Kafka performance

2013-05-17 Thread Kishore V. Kopalle
using NTP? > > On Fri, May 17, 2013 at 8:28 AM, Kishore V. Kopalle > wrote: > > Hi Francis/Stone, > > > > I have modified log.default.flush.interval.ms to have a value of 1 in > > config/server.properties file. The time did not come down as can be seen > > fro

Our use case and am I right in my definition of Throughput?

2013-05-17 Thread Kishore V. Kopalle
Hello All, Our use case is to display certain aggregates on GUI from a live stream of data coming in at more than 100k messages/sec. Will I be able to use Kafka for handling at least 100k messages/sec and send it to Twitter Storm for aggregate calculations is the question I have. I already know th

Re: Time difference between message fetch and message send is very high (~900-1200ms)

2013-05-16 Thread Kishore V. Kopalle
he producer ... is there any sync > ack ? If so which type ? > > In any case, it would be useful to know the parameters you set on the > consumer and producer. > > Thanks > > Frank > > > Sent from Samsung Mobile > > > > Original message --

Re: Kafka performance

2013-05-16 Thread Kishore V. Kopalle
Hi Francis/Stone, I have modified log.default.flush.interval.ms to have a value of 1 in config/server.properties file. The time did not come down as can be seen from the following output: Message received at 2013-05-17 11:54:02.973: Message produced at 2013-05-17 11:54:01.444: Message_1 Message r

Re: Time difference between message fetch and message send is very high (~900-1200ms)

2013-05-16 Thread Kishore V. Kopalle
Hi Frank, Yes, they are on the same machine. Zoo Keeper, Kafka server and the KafkaConsumerProducerDemo (with one producer and one consumer) are running on the same computer. Regards, Kishore On Fri, May 17, 2013 at 11:38 AM, Francis Dallaire < francis.dalla...@ubisoft.com> wrote: > Quick ques

Time difference between message fetch and message send is very high (~900-1200ms)

2013-05-16 Thread Kishore V. Kopalle
Hello All, In my earlier mail, I am sorry I did not give lot of details on my setup where I am measuring performance between producer and consumer. Here are the details: - I am using Kafka version 0.7 downloaded from Apache web site. - I am using a modified version of KafkaConsumerProducerDemo e