It depends on your network and the producer CPU, usually Kafka producer will likely to saturate the network bandwidth first before it used up the CPU if no compression is used.
On Fri, Jul 11, 2014 at 1:19 PM, Chen Song <chen.song...@gmail.com> wrote: > Thanks Guaozhang. > > I have a follow-up question. > > Say if I can push 1M/s messages (100 size) into 2 node cluster. Is it safe > to say I should be able to produce more or less 2M/s messages into 4 node > cluster? > > > On Tue, Jul 8, 2014 at 6:00 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > The second script is using the new producer which allows sending multiple > > in-flight requests whereas the first script use the current (old) > producer > > which sends only one request at a time to a certain broker. > > > > The new producer will be officially released in 0.8.2. > > > > Guozhang > > > > > > On Tue, Jul 8, 2014 at 2:22 PM, Chen Song <chen.song...@gmail.com> > wrote: > > > > > While testing kafka producer performance, I found 2 testing scripts. > > > > > > 1) performance testing script in kafka distribution > > > > > > bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages > > > 10000000 --topic test --threads 10 --message-size 100 --batch-size > 10000 > > > --compression-codec 1 > > > > > > 2) performance testing script mentioned in > > > > > > > > > > > > https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > > > > > > bin/kafka-run-class.sh > > > org.apache.kafka.clients.tools.ProducerPerformance test6 50000000 100 > > > -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 > > > buffer.memory=67108864 batch.size=8196 > > > > > > based on org.apache.kafka.clients.producer.Producer. > > > > > > > > > On my local testing environment, > > > > > > For 1), the loading is ~15M/s and 150,000 msgs/s for mesage size = 100, > > no > > > matter how I adjust batch size and number of loading threads. > > > For 2), it can load up to 90M/s and 1,000,000 records/s. > > > > > > I am wondering what makes these 2 implementations perform so > differently? > > > > > > -- > > > Chen Song > > > > > > > > > > > -- > > -- Guozhang > > > > > > -- > Chen Song > -- -- Guozhang
