Hello again, I learned that the threads shouldn't be more than partitions. So, I did like below.
bin/kafka-consumer-perf-test.sh --zookeeper localhost:2181 --messages 1000 --topic perftest --threads 1 --fetch-size 5242880 --socket-buffer-size 2147483646 start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec 2015-11-20 10:34:50:999, 2015-11-20 10:34:56:043, 5242880, 8.0000, 181.8182, 2, 45.4545 But I want to have 1000 messages and each message should be 5MB of size. Just like in producer performance test. I mentioned like below. I wanted to know is this the correct way of mentioning? Kindly advise. bin/kafka-consumer-perf-test.sh --zookeeper localhost:2181 --messages 1000 --topic perftest --threads 1 --message-size 5242880 --socket-buffer-size 2147483646 --fetch-size 2147483646 start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec 2015-11-20 10:41:44:427, 2015-11-20 10:41:49:473, 2147483646, 8.0000, 173.9130, 2, 43.4783 Regards Jegan From: BOOPATHY, Jegan Sent: Freitag, 20. November 2015 10:21 To: 'users@kafka.apache.org' Subject: kafka perf test 5MB message Hello, I am using Kafka for my thesis project. I need to do some performance tests. I managed to send 5 MB data in producer performace test like below. I have 3 Zookeeper, 2 brokers, 1 producer and 1 consumer in a single machine for testing. I tried with consumer perf test. But it failed. Kindly help me out with the consumer perf test with 5MB data. bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance perftest 1000 5242880 -1 acks=1 bootstrap.servers=localhost:9093 max.request.size=2147483646 184 records sent, 36.6 records/sec (182.90 MB/sec), 172.8 ms avg latency, 457.0 max latency. 102 records sent, 20.4 records/sec (101.82 MB/sec), 337.2 ms avg latency, 2276.0 max latency. 157 records sent, 31.2 records/sec (155.85 MB/sec), 218.1 ms avg latency, 756.0 max latency. 109 records sent, 20.2 records/sec (101.17 MB/sec), 307.5 ms avg latency, 1796.0 max latency. 167 records sent, 33.3 records/sec (166.73 MB/sec), 218.9 ms avg latency, 1794.0 max latency. 137 records sent, 27.1 records/sec (135.43 MB/sec), 241.8 ms avg latency, 1171.0 max latency. 1000 records sent, 28.738110 records/sec (143.69 MB/sec), 234.60 ms avg latency, 2276.00 ms max latency, 167 ms 50th, 496 ms 95th, 1419 ms 99th, 2276 ms 99.9th. bin/kafka-consumer-perf-test.sh --zookeeper localhost:2181 --messages 1000 --topic perftest --threads 10 --fetch-size 5242880 --socket-buffer-size 2147483646 start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec [2015-11-20 10:19:39,096] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-5 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-3 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-7 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-2 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-6 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-4 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,099] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-8 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,099] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-9 for topic perftest (kafka.consumer.RangeAssignor) 2015-11-20 10:19:40:346, 2015-11-20 10:19:45:382, 5242880, 8.0000, 222.2222, 2, 55.5556 This is how I created a topic with two partitions. Actually, how partitions help in the performance? What does it actually do? bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic perftest --replication-factor 2 --partitions 2 Regards Jegan <html><head></head><body><font color="black" face="arial" size="2"> The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free. </font> </body> </html>
