Hello, I am using Kafka for my thesis project. I need to do some performance tests. I managed to send 5 MB data in producer performace test like below. I have 3 Zookeeper, 2 brokers, 1 producer and 1 consumer in a single machine for testing. I tried with consumer perf test. But it failed. Kindly help me out with the consumer perf test with 5MB data.
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance perftest 1000 5242880 -1 acks=1 bootstrap.servers=localhost:9093 max.request.size=2147483646 184 records sent, 36.6 records/sec (182.90 MB/sec), 172.8 ms avg latency, 457.0 max latency. 102 records sent, 20.4 records/sec (101.82 MB/sec), 337.2 ms avg latency, 2276.0 max latency. 157 records sent, 31.2 records/sec (155.85 MB/sec), 218.1 ms avg latency, 756.0 max latency. 109 records sent, 20.2 records/sec (101.17 MB/sec), 307.5 ms avg latency, 1796.0 max latency. 167 records sent, 33.3 records/sec (166.73 MB/sec), 218.9 ms avg latency, 1794.0 max latency. 137 records sent, 27.1 records/sec (135.43 MB/sec), 241.8 ms avg latency, 1171.0 max latency. 1000 records sent, 28.738110 records/sec (143.69 MB/sec), 234.60 ms avg latency, 2276.00 ms max latency, 167 ms 50th, 496 ms 95th, 1419 ms 99th, 2276 ms 99.9th. bin/kafka-consumer-perf-test.sh --zookeeper localhost:2181 --messages 1000 --topic perftest --threads 10 --fetch-size 5242880 --socket-buffer-size 2147483646 start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec [2015-11-20 10:19:39,096] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-5 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-3 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-7 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-2 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-6 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-4 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,099] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-8 for topic perftest (kafka.consumer.RangeAssignor) [2015-11-20 10:19:39,099] WARN No broker partitions consumed by consumer thread perf-consumer-11312_parana01-1448011178702-e2c92213-9 for topic perftest (kafka.consumer.RangeAssignor) 2015-11-20 10:19:40:346, 2015-11-20 10:19:45:382, 5242880, 8.0000, 222.2222, 2, 55.5556 This is how I created a topic with two partitions. Actually, how partitions help in the performance? What does it actually do? bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic perftest --replication-factor 2 --partitions 2 Regards Jegan <html><head></head><body><font color="black" face="arial" size="2"> The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised. If you are not the intended recipient, please notify Airbus immediately and delete this e-mail. Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately. All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free. </font> </body> </html>
