You probably want to test at least the sequential write latency (with
flush) on your file system.
Thanks,
Jun
On Mon, Jan 12, 2015 at 6:08 AM, Shlomi Hazan <shl...@viber.com> wrote:
> Thank you for the guidance, yet I want to know I am doing the test Jun had
> in mind
>
> On Thu, Jan 8, 2015 at 5:36 PM, Jayesh Thakrar <j_thak...@yahoo.com.invalid
> >
> wrote:
>
> > I do see the Windows based scripts in the tar file - but haven't them
> > though.You should find them under bin/windows.
> > Also you can always use other Windows stress testing tools/suites to
> check
> > your local I/O performance..
> > From: Shlomi Hazan <shl...@viber.com>
> > To: users@kafka.apache.org; Jayesh Thakrar <j_thak...@yahoo.com>
> > Sent: Thursday, January 8, 2015 6:25 AM
> > Subject: Re: latency - how to reduce?
> >
> > I would like to test locally first as it is easier than setting up a test
> > cluster to model the production, yet the script kafka-producer-perf-test
> is
> > not available for windows.
> > Jun, what kind of "basic I/O testing on the local FS" did you have in
> mind?
> >
> > Thanks,
> > Shlomi
> >
> >
> >
> >
> > On Tue, Jan 6, 2015 at 5:40 PM, Jayesh Thakrar
> <j_thak...@yahoo.com.invalid
> > >
> > wrote:
> >
> > > Have you tried using the built-in stress test scripts?
> > > bin/kafka-producer-perf-test.sh
> > > bin/kafka-consumer-perf-test.sh
> > >
> > > Here's how I stress tested them -
> > > nohup ${KAFKA_HOME}/bin/kafka-producer-perf-test.sh --broker-list
> > > ${KAFKA_SERVERS} --topic ${TOPIC_NAME} --new-producer --threads 16
> > > --messages 100000000 1>kafka-producer-perf-test.sh.log 2>&1 &
> > >
> > > nohup ${KAFKA_HOME}/bin/kafka-consumer-perf-test.sh --zookeeper
> > > ${ZOOKEEPER_QUORUM} --topic ${TOPIC_NAME} --threads 16
> > > 1>kafka-consumer-perf-test.sh.log 2>&1 &
> > >
> > > And I used screen scrapping of the jmx ui screens to push metrics into
> > > TSDB to get the following.The rate below is per second - so I could
> push
> > > the Kafka cluster to 140k+ messages/sec on a 4-node cluster with very
> > > little utilization (<30% utilization).
> > >
> > >
> > > From: Shlomi Hazan <shl...@viber.com>
> > > To: users@kafka.apache.org
> > > Sent: Tuesday, January 6, 2015 1:06 AM
> > > Subject: Re: latency - how to reduce?
> > >
> > > Will do. What did you have in mind? just write a big file to disk and
> > > measure the time it took to write? maybe also read back? using specific
> > > API's?
> > > Apart from the local Win machine case, are you aware of any issues with
> > > Amazon EC2 instances that may be causing that same latency in
> production?
> > > Thanks,
> > > Shlomi
> > >
> > >
> > >
> > > On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <j...@confluent.io> wrote:
> > >
> > > > Not setting "log.flush.interval.messages" is good since the default
> > gives
> > > > the best latency. Could you do some basic I/O testing on the local FS
> > in
> > > > your windows machine to make sure the I/O latency is ok?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <shl...@viber.com>
> wrote:
> > > >
> > > > > Happy new year!
> > > > > I did not set "log.flush.interval.messages".
> > > > > I also could not find a default value in the docs.
> > > > > Could you explain about that?
> > > > > Thanks,
> > > > > Shlomi
> > > > >
> > > > > On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <j...@confluent.io> wrote:
> > > > >
> > > > > > What's your setting of log.flush.interval.messages on the broker?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <shl...@viber.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > I am using 0.8.1.1, and I have hundreds of msec latency at best
> > and
> > > > > even
> > > > > > > seconds at worst.
> > > > > > > I have this latency both on production, (with peak load of 30K
> > > > msg/sec,
> > > > > > > replication = 2 across 5 brokers, acks = 1),
> > > > > > > and on the local windows machine using just one process for
> each
> > of
> > > > > > > producer, zookeeper, kafka, consumer.
> > > > > > > Also tried batch.num.messages=1 and producer.type=sync on the
> > local
> > > > > > machine
> > > > > > > but saw no improvement.
> > > > > > > How can I push latency down to several millis, at least when
> > > running
> > > > > > local?
> > > > > > > Thanks,
> > > > > > > Shlomi
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
