While testing kafka producer performance, I found 2 testing scripts. 1) performance testing script in kafka distribution
bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages 10000000 --topic test --threads 10 --message-size 100 --batch-size 10000 --compression-codec 1 2) performance testing script mentioned in https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test6 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196 based on org.apache.kafka.clients.producer.Producer. On my local testing environment, For 1), the loading is ~15M/s and 150,000 msgs/s for mesage size = 100, no matter how I adjust batch size and number of loading threads. For 2), it can load up to 90M/s and 1,000,000 records/s. I am wondering what makes these 2 implementations perform so differently? -- Chen Song
