with the binaries. When I am trying to enable
> > transaction in the producer script it runs into errors. For example:
> >
> > # bin/kafka-producer-perf-test.sh --topic testtopic1 --num-records 1
> > --throughput -1 --record-size 256 --producer-props
> > bo
rs.
>
> I test max throughput to a single kafka broker but results do not match
> speed limits disk or network.
>
> /kafka-producer-perf-test.sh --topic perf-test --num-records 3000
> --record-size 100 --throughput -1 --producer-props
> bootstrap.servers=localhost:9092
&g
Hello users.
I test max throughput to a single kafka broker but results do not match
speed limits disk or network.
/kafka-producer-perf-test.sh --topic perf-test --num-records 3000
--record-size 100 --throughput -1 --producer-props
bootstrap.servers=localhost:9092
3000 records sent
t; test perf scripts that ship with the binaries. When I am trying to enable
> transaction in the producer script it runs into errors. For example:
>
> # bin/kafka-producer-perf-test.sh --topic testtopic1 --num-records 1
> --throughput -1 --record-size 256 --producer-props
> b
I am trying to test a 3 node Kafka cluster using the producer and consumer test
perf scripts that ship with the binaries. When I am trying to enable
transaction in the producer script it runs into errors. For example:
# bin/kafka-producer-perf-test.sh --topic testtopic1 --num-records 1
I have another very disturbing observation.
The errors go away if I start 2 kafka-producer-perf-test.sh with the same
configs on different hosts.
If I cancel 1 kafka-producer-perf-test.sh then after some time the below
errors start reappearing.
org.apache.kafka.common.errors.TimeoutException
-producer-perf-test.sh --topic
benchmark-6p-3r-2isr-none --num-records 1000 --record-size 100
--throughput -1 --print-metrics --producer-props acks=all
bootstrap.servers=node1:9092,node2:9092,node3:9092 request.timeout.ms=18
max.block.ms=18 buffer.memory=1 >
~/kafka/load_test/results
u...@gmail.com>
> >> wrote:
> >>
> >> > Thanks Ewen. Do you know if kafka-run-class.sh
> >> > org.apache.kafka.clients.tools.ProducerPerformance
> >> > is single threaded? Or is there any way to specify number of threads?
> >> >
> &g
roducerPerformance
>> > is single threaded? Or is there any way to specify number of threads?
>> >
>> > On Fri, Jan 8, 2016 at 1:24 PM, Ewen Cheslack-Postava > >
>> > wrote:
>> >
>> > > Ah, sorry, I missed the version number in your tit
gt; > > Ah, sorry, I missed the version number in your title. I think this tool
> > saw
> > > some rearrangement in 0.9.0 and I was looking at the latest version.
> > > Unfortunately it doesn't look like the old
> > kafka.tools.ProducerPerformance
> > > tha
0 and I was looking at the latest version.
> > Unfortunately it doesn't look like the old
> kafka.tools.ProducerPerformance
> > that is used in kafka-producer-perf-test.sh in 0.8.2.1 supports passing
> in
> > additional properties.
> >
> > -Ewen
> >
> &g
ink this tool saw
> some rearrangement in 0.9.0 and I was looking at the latest version.
> Unfortunately it doesn't look like the old kafka.tools.ProducerPerformance
> that is used in kafka-producer-perf-test.sh in 0.8.2.1 supports passing in
> additional properties.
>
> -Ewen
&g
Ah, sorry, I missed the version number in your title. I think this tool saw
some rearrangement in 0.9.0 and I was looking at the latest version.
Unfortunately it doesn't look like the old kafka.tools.ProducerPerformance
that is used in kafka-producer-perf-test.sh in 0.8.2.1 supports passi
Ewen,
I tried that before like this:
./kafka-producer-perf-test.sh --broker-list test:9092 --topics test8-3
--messages 200 --new-producer --message-size 200
--show-detailed-stats max.request.size=1000
and it does not work. It comletly ignore this option.
And --producer-props is not
Andrew,
kafka-producer-perf-test.sh is just a wrapper around
orga.apache.kafka.clients.tools.ProducerPerformance and all command line
options should be forwarded. Can you just pass a --producer-props to set
max.request.size to a larger value?
-Ewen
On Fri, Jan 8, 2016 at 7:51 AM, Andrej
Hi!
I am testing Kafka's performance with large messages and would like to
specify maximum request size when I run kafka-producer-perf-test.sh:
./kafka-producer-perf-test.sh --broker-list "test1:9092" --topics test8-3
--messages 100 --new-producer --msage-size 110 --show
t;, "
> d...@kafka.apache.org<mailto:d...@kafka.apache.org>" <mailto:d...@kafka.apache.org>>
> Subject: Re: kafka producer-perf-test.sh compression-codec not working
>
> Hi Erik,
>
> Thanks for your inputs.
>
> How can we measure round trip latenc
ers@kafka.apache.org>"
mailto:users@kafka.apache.org>>,
"d...@kafka.apache.org<mailto:d...@kafka.apache.org>"
mailto:d...@kafka.apache.org>>
Subject: Re: kafka producer-perf-test.sh compression-codec not working
Hi Erik,
Thanks for your inputs.
How can we meas
Hi Erik,
Thanks for your inputs.
How can we measure round trip latency using kafka-producer-perf-test.sh ?
or any other tool ?
Regards,
Prabhjot
On Tue, Aug 25, 2015 at 7:41 PM, Helleren, Erik
wrote:
> Prabhjot,
> When no compression is being used, it should have only a tiny imp
,
"d...@kafka.apache.org<mailto:d...@kafka.apache.org>"
mailto:d...@kafka.apache.org>>
Subject: Re: kafka producer-perf-test.sh compression-codec not working
Hi Erik,
I have put my efforts on the produce side till now, Thanks for making me aware
that consumer will decompress a
> goal is for each message to be unique, but still have similar chunks that
> a compression algorithm can detect and compress.
>
> -Erik
>
>
> On 8/25/15, 6:47 AM, "Prabhjot Bharaj" wrote:
>
> >Hi,
> >
> >I have bene trying to use kafka-producer-
your choice. The
goal is for each message to be unique, but still have similar chunks that
a compression algorithm can detect and compress.
-Erik
On 8/25/15, 6:47 AM, "Prabhjot Bharaj" wrote:
>Hi,
>
>I have bene trying to use kafka-producer-perf-test.sh to arrive at cert
Hi,
I have bene trying to use kafka-producer-perf-test.sh to arrive at certain
benchmarks.
When I try to run it with --compression-codec values of 1, 2 and 3, I
notice increased throughput compared to NoCompressionCodec
But, When I checked the Producerperformance.scala, I saw that the the
2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3
>
> Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5
>
>
> This is the output of the kafka-producer-perf-test.sh for request-num-acks
> 1 and request-num-acks -1:-
>
> root@x.x.x.x:~# date;time kafka-producer-perf-test.s
Isr: 4,1,2
>>
>> Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2
>>
>> Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3
>>
>> Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5
>>
>>
>> When I run thi
5,2,3 Isr: 5,3,2
Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3
Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5
This is the output of the kafka-producer-perf-test.sh for request-num-acks
1 and request-num-acks -1:-
root@x.x.x.x:~# date;time kafka-producer-pe
tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2
>
> Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3
>
> Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5
>
>
> When I run this command in parallel from 2 different machines, I get
> SocketTimeoutE
5 Replicas: 5,2,3 Isr: 5,3,2
Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3
Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5
When I run this command in parallel from 2 different machines, I get
SocketTimeoutException:-
time kafka-producer-perf-test.sh --broker-list
x.
If that kafka-producer-per-test.sh doesn't work for you and you choose the
DIY approach, https://github.com/sematext/ActionGenerator may be of help.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Dec 22,
have to conduce load and stress test by publishing at least 10K
messages/sec to my Kafka brokers. Here is the setup:
Number of brokers : 2
Topic : 1
Partition :2. Each broker leader of one.
I would like to know if i can use "kafka-producer-per-test.sh" in first
place. also, if any one knows the
What kind of load testing do you plan to do?
Thanks,
Jun
On Wed, Dec 17, 2014 at 3:42 AM, nitin sharma
wrote:
> Hi Team,
>
> I have to make a decision on whether i should go with Kafka producer test
> utility or build my own java tool for my load testing .
>
> Kindly let me know if anyone know
Hi Team,
I have to make a decision on whether i should go with Kafka producer test
utility or build my own java tool for my load testing .
Kindly let me know if anyone knows any limitation with
"kafka-producer-per-test.sh" when it come to simulation of messages under
load condition?
Regards,
Nit
logger implementation
> >>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> >> further
> >>>> details.
> >>>> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not
> valid
> >>>> (kafka.utils.Ve
ect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not
perty reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420,
] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 14:10:37,560] WARN Property reconnect.inte
perties)
>> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid
>> (kafka.utils.VerifiableProperties)
>> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72,
>> 52.4285, 500, 130893.4789
>>
>>
>> -bash-4.1
nect.interval is not valid
> (kafka.utils.VerifiableProperties)
> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72,
> 52.4285, 500, 130893.4789
>
>
> -bash-4.1$
> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh
> --broker-list
uild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh
--broker-list vp21q12ic-hpaj020921:9092 --messages 500 --topic imessage
--threads 5 --message-size 420 --batch-size 1000 --compression-codec 2
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.
/0.7/lib>
> > ls -l
> > total 2800
> > -rw-r--r--@ 1 cneal staff 1306797 Apr 22 18:37 kafka-0.7.0.jar
> > *-rw-r--r--@ 1 cneal staff55427 Apr 22 18:37 kafka-perf-0.7.0.jar*
> > -rw-r--r--@ 1 cneal staff62913 Apr 22 18:37 zkclient-0.1.jar
> >
> &
l staff55427 Apr 22 18:37 kafka-perf-0.7.0.jar*
> -rw-r--r--@ 1 cneal staff 62913 Apr 22 18:37 zkclient-0.1.jar
>
> But then when I run the .sh scripts as follows, I get yet another exception:
>
> root@ip-10-0-0-42[397]:/opt/kafka/kafka>
>bin/kafka-pr
06797 Apr 22 18:37 kafka-0.7.0.jar
*-rw-r--r--@ 1 cneal staff55427 Apr 22 18:37 kafka-perf-0.7.0.jar*
-rw-r--r--@ 1 cneal staff62913 Apr 22 18:37 zkclient-0.1.jar
But then when I run the .sh scripts as follows, I get yet another exception:
root@ip-10-0-0-42[397]:/opt/kafka/kafka
42 matches
Mail list logo
