Re: kafka producer performance test

2014-10-02 Thread Sa Li
Yes, I did either way, 1: follow the instruction by http://www.gradle.org/installation, 2. apt-get install gradle. Thanks On Thu, Oct 2, 2014 at 2:21 PM, Guozhang Wang wrote: > Did you installed gradle as the README stated? > > "You need to have [gradle](http://www.gradle.org/installation) ins

Re: kafka producer performance test

2014-10-02 Thread Guozhang Wang
Did you installed gradle as the README stated? "You need to have [gradle](http://www.gradle.org/installation) installed." Guozhang On Thu, Oct 2, 2014 at 1:55 PM, Sa Li wrote: > Thanks Guozhang > > I tried this as in KAFKA-1490: > > git clone https://git-wip-us.apache.org/repos/asf/kafka.git >

Re: kafka producer performance test

2014-10-02 Thread Sa Li
I can't really gradle through, even clone the latest trunk, anyone having same issue? On Thu, Oct 2, 2014 at 1:55 PM, Sa Li wrote: > Thanks Guozhang > > I tried this as in KAFKA-1490: > > git clone https://git-wip-us.apache.org/repos/asf/kafka.git > > cd kafka > > gradle > > > but fails to bui

Re: kafka producer performance test

2014-10-02 Thread Sa Li
Thanks Guozhang I tried this as in KAFKA-1490: git clone https://git-wip-us.apache.org/repos/asf/kafka.git cd kafka gradle but fails to build: FAILURE: Build failed with an exception. * Where: Script '/home/stuser/trunk/gradle/license.gradle' line: 2 * What went wrong: A problem occurr

Re: kafka producer performance test

2014-10-02 Thread Guozhang Wang
Hello Sa, KAFKA-1490 introduces a new step of downloading the wrapper, details are included in the latest README file. Guozhang On Thu, Oct 2, 2014 at 11:00 AM, Sa Li wrote: > Thanks, Jay, > > Here is what I did this morning, I git clone the latest version of kafka > from git, (I am currently

Re: kafka producer performance test

2014-10-02 Thread Sa Li
Thanks, Jay, Here is what I did this morning, I git clone the latest version of kafka from git, (I am currently using kafka 8.0) now it is 8.1.1, and it use gradle to build project. I am having trouble to build it. I installed gradle, and run ./gradlew jar in kafka root directory, it comes out: Er

Re: kafka producer performance test

2014-10-01 Thread Jay Kreps
Hi Sa, That script was developed with the new producer that is included on trunk. Checkout trunk and build and it should be there. -Jay On Wed, Oct 1, 2014 at 7:55 PM, Sa Li wrote: > Hi, All > > I built a 3-node kafka cluster, I want to make performance test, I found > someone post following t

Re: kafka producer performance test

2014-10-01 Thread Sa Li
Hi, Ravi Thanks for reply, this is how I build the kafka package 0.8 $ git clone https://git-wip-us.apache.org/repos/asf/kafka.git $ cd /etc/kafka $ git checkout -b 0.8 remotes/origin/0.8 $ ./sbt update $ ./sbt package $ ./sbt assembly-package-dependency So I believe I already build it, but stil

Re: kafka producer performance test

2014-10-01 Thread ravi singh
It is available with Kafka package containing the source code. Download the package, build it and run the above command. Regards, Ravi On Wed, Oct 1, 2014 at 7:55 PM, Sa Li wrote: > Hi, All > > I built a 3-node kafka cluster, I want to make performance test, I found > someone post following th

kafka producer performance test

2014-10-01 Thread Sa Li
Hi, All I built a 3-node kafka cluster, I want to make performance test, I found someone post following thread, that is exactly the problem I have: - While testing kafka producer performance, I found 2 testing scripts. 1) performance testing script in kafka distribution bin/kafka-p

Re: Kafka producer performance test sending 0x0 byte messages

2014-07-02 Thread Bert Corderman
Jun, let me see if I can fix first and then will submit back. Daniel, I was looking at the code some more and was thinking this might work https://github.com/apache/kafka/blob/0.8.1/perf/src/main/scala/kafka/perf/ProducerPerformance.scala on line 246 instead of looping to create messages I co

Re: Kafka producer performance test sending 0x0 byte messages

2014-06-30 Thread Daniel Compton
Hi Bert What you are describing could be done partially with the console producer. It will read from a file and send each line to the Kafka broker. You could make a really big file or alter that code to repeat a certain number of times. The source is pretty readable, I think that might be an ea

Re: Kafka producer performance test sending 0x0 byte messages

2014-06-30 Thread Jun Rao
Yes, this is a problem and will indeed affect the producer performance when compression is turned on. Perhaps we should fill in the values with some randomized bytes. Could you file a jira for this? Thanks, Jun On Sun, Jun 29, 2014 at 11:24 PM, Daniel Compton wrote: > Hi folks > > I was doing

Re: Kafka producer performance test sending 0x0 byte messages

2014-06-30 Thread Bert Corderman
Daniel, We have the same question. We noticed that the compression tests we ran using the built in performance tester was not realistic. I think on disk compression was 200:1. (yes that is two hundred to one) I had planned to try and edit the producer performance tester source and do the foll

Kafka producer performance test sending 0x0 byte messages

2014-06-29 Thread Daniel Compton
Hi folks I was doing some performance testing using the built in Kafka performance tester and it seems like it sends messages of size n bytes but with all bytes having the value 0x0. Is that correct? Reading the source seemed to indicate that too but I'm not a Scala developer so I could be wron