Hi Neha,
I am a newbie to Kafka. My understanding so far is that batch size is
supported only for async producer. Kindly correct me if I am wrong. The
best latency with async producer I am getting is of the order of 300ms. Is
that expected?
I have to stick to 0.7 because KafkaSpout for Storm seems to work only
with 0.7.x.
Regards,
Kishore
On Fri, May 17, 2013 at 7:07 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote:
> Kishore,
>
> The end to end latency will be much lower in 08. For 07, you can try to
> tune the producer batch size so it sends data quicker. You can also reduce
> the log.flush.interval on the broker but that will drive up iops.
>
> Thanks,
> Neha
> On May 16, 2013 11:39 PM, "Kishore V. Kopalle" <kish...@greenmedsoft.com>
> wrote:
>
> > Hi Frank,
> >
> > Here are the explicitly set properties:
> > Consumer:
> > Properties props = new Properties();
> > props.put("zk.connect", KafkaProperties.zkConnect);
> > props.put("groupid", KafkaProperties.groupId);
> > props.put("zk.sessiontimeout.ms", "400");
> > props.put("zk.synctime.ms", "200");
> > props.put("autocommit.interval.ms", "1000");
> >
> > Producer:
> > props.put("serializer.class", "kafka.serializer.StringEncoder");
> > props.put("zk.connect", "localhost:2181");
> >
> > Everything else is out of box except for log.default.flush.interval.ms=1
> > in
> > server.properties which is set according to Stone's suggestion.
> >
> >
> > Regards,
> > Kishore
> >
> > On Fri, May 17, 2013 at 11:48 AM, Francis Dallaire <
> > francis.dalla...@ubisoft.com> wrote:
> >
> > > What are the settings you have set on the producer ... is there any
> sync
> > > ack ? If so which type ?
> > >
> > > In any case, it would be useful to know the parameters you set on the
> > > consumer and producer.
> > >
> > > Thanks
> > >
> > > Frank
> > >
> > >
> > > Sent from Samsung Mobile
> > >
> > >
> > >
> > > -------- Original message --------
> > > From: "Kishore V. Kopalle" <kish...@greenmedsoft.com>
> > > Date: 05-17-2013 2:11 AM (GMT-05:00)
> > > To: users@kafka.apache.org,Francis Dallaire <
> > francis.dalla...@ubisoft.com>
> > > Subject: Re: Time difference between message fetch and message send is
> > > very high (~900-1200ms)
> > >
> > >
> > > Hi Frank,
> > >
> > > Yes, they are on the same machine. Zoo Keeper, Kafka server and the
> > > KafkaConsumerProducerDemo (with one producer and one consumer) are
> > running
> > > on the same computer.
> > >
> > > Regards,
> > > Kishore
> > >
> > > On Fri, May 17, 2013 at 11:38 AM, Francis Dallaire <
> > > francis.dalla...@ubisoft.com<mailto:francis.dalla...@ubisoft.com>>
> > wrote:
> > > Quick question. Are the producer and consumer on the same machine ? If
> > > not, are you accounting for any difference in the time setting between
> > > producer and consumer ?
> > >
> > > Frank
> > >
> > > Sent from Samsung Mobile
> > >
> > >
> > >
> > > -------- Original message --------
> > > From: Kishore Kopalle <kkopa...@gmail.com<mailto:kkopa...@gmail.com>>
> > > Date: 05-17-2013 1:41 AM (GMT-05:00)
> > > To: users@kafka.apache.org<mailto:users@kafka.apache.org>
> > > Subject: Time difference between message fetch and message send is very
> > > high (~900-1200ms)
> > >
> > >
> > > Hello All,
> > >
> > > In my earlier mail, I am sorry I did not give lot of details on my
> setup
> > > where I am measuring performance between producer and consumer. Here
> are
> > > the details:
> > >
> > > - I am using Kafka version 0.7 downloaded from Apache web site.
> > > - I am using a modified version of KafkaConsumerProducerDemo example
> that
> > > comes with the distribution. I have made the following modifications:
> > >
> > > - In Producer java file:
> > >
> > > int messageNo = 1;
> > > SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd
> > > HH:mm:ss.SSS") ;
> > >
> > > while(true)
> > > {
> > > String messageStr = new String("Message_" + messageNo);
> > > producer.send(new ProducerData<Integer, String>(topic,
> > > "Message produced at " + dateFormat.format(new Date())
> > > + ": " + messageStr));
> > > messageNo++;
> > > }
> > >
> > >
> > > - In Consumer.java file:
> > >
> > > SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd
> > > HH:mm:ss.SSS") ;
> > >
> > > while(it.hasNext())
> > > System.out.println("Message received at " + dateFormat.format(new
> > > Date()) + ": " + ExampleUtils.getMessage(it.next().message()));
> > >
> > >
> > > Based on the messages on the console from consumer, I am getting values
> > > ranging from 900ms to 1200ms per message for the time difference
> between
> > > received message time and the timestamp in the message itself.
> > >
> > > Kindly let me know where I am going wrong and are there any
> configuration
> > > steps involved for getting optimal performance for the time taken for a
> > > specific message to be fetched and the time when it was sent by
> producer.
> > >
> > > Regards,
> > > Kishore
> > >
> > >
> >
>
