The second script is using the new producer which allows sending multiple in-flight requests whereas the first script use the current (old) producer which sends only one request at a time to a certain broker.
The new producer will be officially released in 0.8.2. Guozhang On Tue, Jul 8, 2014 at 2:22 PM, Chen Song <chen.song...@gmail.com> wrote: > While testing kafka producer performance, I found 2 testing scripts. > > 1) performance testing script in kafka distribution > > bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages > 10000000 --topic test --threads 10 --message-size 100 --batch-size 10000 > --compression-codec 1 > > 2) performance testing script mentioned in > > > https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > > bin/kafka-run-class.sh > org.apache.kafka.clients.tools.ProducerPerformance test6 50000000 100 > -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 > buffer.memory=67108864 batch.size=8196 > > based on org.apache.kafka.clients.producer.Producer. > > > On my local testing environment, > > For 1), the loading is ~15M/s and 150,000 msgs/s for mesage size = 100, no > matter how I adjust batch size and number of loading threads. > For 2), it can load up to 90M/s and 1,000,000 records/s. > > I am wondering what makes these 2 implementations perform so differently? > > -- > Chen Song > -- -- Guozhang
