Data from single partition can not be consumed by multiple threads in a consumer group. So having more consumer threads than the number of partitions will be ended up that no data for some consumer threads.
Consumer parallelism depends on the number of partitions. If a topic has more partitions then subscribers/consumers can consume the data parallely by having larger consumer group(i.e. more consumer threads). In you case, you can at most have consumer group size of 2 (i.e. 2 consumer threads in consumer group) as number of partitions are two. bin/kafka-consumer-perf-test.sh --zookeeper localhost:2181 --messages 1000 --topic perftest *--threads 2* --message-size 5242880 --socket-buffer-size 2147483646 --fetch-size 2147483646 On Fri, Nov 20, 2015 at 5:21 PM, BOOPATHY, Jegan <jegan.boopa...@airbus.com> wrote: > Hello, > > I am using Kafka for my thesis project. I need to do some performance > tests. I managed to send 5 MB data in producer performace test like below. > I have 3 Zookeeper, 2 brokers, 1 producer and 1 consumer in a single > machine for testing. I tried with consumer perf test. But it failed. Kindly > help me out with the consumer perf test with 5MB data. > > bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance > perftest 1000 5242880 -1 acks=1 bootstrap.servers=localhost:9093 > max.request.size=2147483646 > 184 records sent, 36.6 records/sec (182.90 MB/sec), 172.8 ms avg latency, > 457.0 max latency. > 102 records sent, 20.4 records/sec (101.82 MB/sec), 337.2 ms avg latency, > 2276.0 max latency. > 157 records sent, 31.2 records/sec (155.85 MB/sec), 218.1 ms avg latency, > 756.0 max latency. > 109 records sent, 20.2 records/sec (101.17 MB/sec), 307.5 ms avg latency, > 1796.0 max latency. > 167 records sent, 33.3 records/sec (166.73 MB/sec), 218.9 ms avg latency, > 1794.0 max latency. > 137 records sent, 27.1 records/sec (135.43 MB/sec), 241.8 ms avg latency, > 1171.0 max latency. > 1000 records sent, 28.738110 records/sec (143.69 MB/sec), 234.60 ms avg > latency, 2276.00 ms max latency, 167 ms 50th, 496 ms 95th, 1419 ms 99th, > 2276 ms 99.9th. > > bin/kafka-consumer-perf-test.sh --zookeeper localhost:2181 --messages 1000 > --topic perftest --threads 10 --fetch-size 5242880 --socket-buffer-size > 2147483646 > start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, > data.consumed.in.nMsg, nMsg.sec > [2015-11-20 10:19:39,096] WARN No broker partitions consumed by consumer > thread perf-consumer-11312_parana01-1448011178702-e2c92213-5 for topic > perftest (kafka.consumer.RangeAssignor) > [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer > thread perf-consumer-11312_parana01-1448011178702-e2c92213-3 for topic > perftest (kafka.consumer.RangeAssignor) > [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer > thread perf-consumer-11312_parana01-1448011178702-e2c92213-7 for topic > perftest (kafka.consumer.RangeAssignor) > [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer > thread perf-consumer-11312_parana01-1448011178702-e2c92213-2 for topic > perftest (kafka.consumer.RangeAssignor) > [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer > thread perf-consumer-11312_parana01-1448011178702-e2c92213-6 for topic > perftest (kafka.consumer.RangeAssignor) > [2015-11-20 10:19:39,098] WARN No broker partitions consumed by consumer > thread perf-consumer-11312_parana01-1448011178702-e2c92213-4 for topic > perftest (kafka.consumer.RangeAssignor) > [2015-11-20 10:19:39,099] WARN No broker partitions consumed by consumer > thread perf-consumer-11312_parana01-1448011178702-e2c92213-8 for topic > perftest (kafka.consumer.RangeAssignor) > [2015-11-20 10:19:39,099] WARN No broker partitions consumed by consumer > thread perf-consumer-11312_parana01-1448011178702-e2c92213-9 for topic > perftest (kafka.consumer.RangeAssignor) > 2015-11-20 10:19:40:346, 2015-11-20 10:19:45:382, 5242880, 8.0000, > 222.2222, 2, 55.5556 > > This is how I created a topic with two partitions. Actually, how > partitions help in the performance? What does it actually do? > > bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic perftest > --replication-factor 2 --partitions 2 > > > Regards > Jegan > > <html><head></head><body><font color="black" face="arial" size="2"> > The information in this e-mail is confidential. The contents may not be > disclosed or used by anyone other than the addressee. Access to this e-mail > by anyone else is unauthorised. > If you are not the intended recipient, please notify Airbus immediately > and delete this e-mail. > Airbus cannot accept any responsibility for the accuracy or completeness > of this e-mail as it has been sent over public networks. If you have any > concerns over the content of this message or its Accuracy or Integrity, > please contact Airbus immediately. > All outgoing e-mails from Airbus are checked using regularly updated virus > scanning software but you should take whatever measures you deem to be > appropriate to ensure that this message and any attachments are virus free. > </font> > </body> > </html> > > -- Thanks Raju Bairishetti, www.lazada.com
