Hi, I’m having trouble understanding the results from running
kafka-consumer-perf-test. For low number of messages, I see very low
throughput in terms of messages / second. Here is a table of results:
fetch.size
data.consumed.in.MB
MB.sec
data.consumed.in.nMsg
nMsg.sec
1048576
0.0003
0
1
0.014
1048576
0.0029
0
10
0.0956
1048576
0.0286
0.0003
100
1.0404
1048576
0.2861
0.0035
1000
12.1222
1048576
2.861
0.033
10000
115.3602
1048576
28.6102
0.4234
100000
1479.7277
1048576
286.1023
4.0228
1000000
14060.5447
1048576
1142.2345
16.6818
3992399
58307.0306
I would have expected a constant time to establish connections and then
nMsg.sec to increase. When consuming just 1000 messages, why is it so slow?
Thanks,
- Baran
(information for reproducing results)
I have a single kafka broker and single zookeeper node. Here are the commands
I ran:
bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test
--replication 1 --partition 1
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
localhost:9092 6000000 300
for i in 1 10 100 1000 10000 100000 1000000 10000000; do
bin/kafka-consumer-perf-test.sh --zookeeper localhost:2181 --messages
$i --topic test --threads 1;
done
