Hello friends, Request your expertise on this problem I'm facing
Thanks On Sep 4, 2015 8:09 PM, "Prabhjot Bharaj" <prabhbha...@gmail.com> wrote: > Hi, > > I am experiencing super slow throughput when using acks=-1 > Some further progress in continuation to the test in my previous email:- > > *Topic details -* > > Topic:temp PartitionCount:1 ReplicationFactor:3 Configs: > > Topic: temp Partition: 0 Leader: 5 Replicas: 5,1,2 Isr: 5,2,1 > *This is the command I'm running - * > > kafka-producer-perf-test.sh --broker-list 96.7.250.122:9092, > 96.17.183.53:9092,96.17.183.54:9092,96.7.250.117:9092,96.7.250.118:9092 > --messages 100000 --message-size 500 --topics temp --show-detailed-stats > --threads 30 --request-num-acks -1 --batch-size 1000 --request-timeout-ms > 10000 > > *Server.properties:-* > > broker.id=0 > > port=9092 > > *num.network.threads=6* > > num.io.threads=8 > > *socket.send.buffer.bytes=10485760* > > *socket.receive.buffer.bytes=10485760* > > socket.request.max.bytes=104857600 > > log.dirs=/tmp/kafka-logs > > num.partitions=1 > > num.recovery.threads.per.data.dir=1 > > log.retention.hours=168 > > log.segment.bytes=1073741824 > > log.retention.check.interval.ms=300000 > > log.cleaner.enable=false > > zookeeper.connect=localhost:2181 > > zookeeper.connection.timeout.ms=6000 > > *num.replica.fetchers=6* > *Observation:-* > > I have also noticed that if I use acks=1 (without --sync) and immediately > use acks=-1 (without --sync), the test completes very quickly. Also, after > running this, if I describe the topic, it is still not in sync, which > means acks=-1 is treated as 1 only > > Also, when running on a freshly created topic with just acks=-1, it takes > 8 minutes to complete with --sync > > time kafka-producer-perf-test.sh --broker-list 96.7.250.122:9092, > 96.17.183.53:9092,96.17.183.54:9092,96.7.250.117:9092,96.7.250.118:9092 > --messages 100000 --message-size 500 --topics temp --show-detailed-stats > --threads 30 --request-num-acks -1 --batch-size 1000 --request-timeout-ms > 10000 --compression-codec 2 --sync > > start.time, end.time, compression, message.size, batch.size, > total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > > 2015-09-04 12:19:36:223, 2015-09-04 12:27:41:775, 2, 500, 1000, 47.68, > 0.0982, 99990, 205.9306 > > real 8m6.563s > > user 0m19.787s > > sys 0m5.601s > > If I use --sync, it is taking way longer. > > Where am I doing wrong? > > Thanks, > Prabhjot > > On Fri, Sep 4, 2015 at 1:45 AM, Gwen Shapira <g...@confluent.io> wrote: > >> Yes, this should work. Expect lower throughput though. >> >> On Thu, Sep 3, 2015 at 12:52 PM, Prabhjot Bharaj <prabhbha...@gmail.com> >> wrote: >> >> > Hi, >> > >> > Can I use sync for acks = -1? >> > >> > Regards, >> > Prabhjot >> > On Sep 3, 2015 11:49 PM, "Gwen Shapira" <g...@confluent.io> wrote: >> > >> > > The test uses the old producer (we should fix that), and since you >> don't >> > > specify --sync, it runs async. >> > > The old async producer simply sends data and doesn't wait for acks, >> so it >> > > is possible that the messages were never acked... >> > > >> > > On Thu, Sep 3, 2015 at 7:56 AM, Prabhjot Bharaj < >> prabhbha...@gmail.com> >> > > wrote: >> > > >> > > > Hi Folks, >> > > > >> > > > Request your expertise on my doubt here. >> > > > >> > > > *My setup:-* >> > > > >> > > > 5 node kafka cluster (4 cores, 8GB RAM) on RAID-6 (500 GB) >> > > > Using Kafka 0.8.2.1 with modified ProducerPerformance.scala >> > > > I've modified ProducerPerformance.scala to send custom ASCII data, >> > > instead >> > > > of Byte Array of Zeroes >> > > > >> > > > *server.properties:-* >> > > > >> > > > broker.id=0 >> > > > >> > > > log.cleaner.enable=false >> > > > >> > > > log.dirs=/tmp/kafka-logs >> > > > >> > > > log.retention.check.interval.ms=300000 >> > > > >> > > > log.retention.hours=168 >> > > > >> > > > log.segment.bytes=1073741824 >> > > > >> > > > num.io.threads=8 >> > > > >> > > > num.network.threads=3 >> > > > >> > > > num.partitions=1 >> > > > >> > > > num.recovery.threads.per.data.dir=1 >> > > > >> > > > *num.replica.fetchers=4* >> > > > >> > > > port=9092 >> > > > >> > > > socket.receive.buffer.bytes=1048576 >> > > > >> > > > socket.request.max.bytes=104857600 >> > > > >> > > > socket.send.buffer.bytes=1048576 >> > > > >> > > > zookeeper.connect=localhost:2181 >> > > > >> > > > zookeeper.connection.timeout.ms=6000 >> > > > >> > > > >> > > > *This is how I run the producer perf test:-* >> > > > >> > > > kafka-producer-perf-test.sh --broker-list >> > > > a.a.a.a:9092,b.b.b.b:9092,c.c.c.c:9092,d.d.d.d:9092,e.e.e.e:9092 >> > > --messages >> > > > 100000 --message-size 500 --topics temp --show-detailed-stats >> > --threads >> > > 5 >> > > > --request-num-acks -1 --batch-size 200 --request-timeout-ms 10000 >> > > > --compression-codec 0 >> > > > >> > > > *Problem:-* >> > > > >> > > > This test completes in under 15 seconds for me >> > > > >> > > > But, after this test, if I try writing to another topic which has 2 >> > > > partitions and 3 replicas, it is dead slow and the same script seems >> > > never >> > > > to finish because the slow ISR catch-up is still going on. >> > > > >> > > > *My inference:-* >> > > > I have noticed that for a topic with 1 partition and 3 replicas, the >> > ISR >> > > > shows only 1 broker id. >> > > > >> > > > Topic:temp PartitionCount:1 ReplicationFactor:3 Configs: >> > > > >> > > > Topic: temp Partition: 0 Leader: 5 Replicas: 5,1,2 Isr: 5 >> > > > >> > > > >> > > > I think it is because the data from the leader is not received in >> > broker >> > > > ids 1 and 2 >> > > > Also, I could confirm it from the data directory sizes for this >> topic. >> > > > Leader (5) has 20GB but replicas - 1 and 2 are still at 7GB >> > > > >> > > > *Doubts:-* >> > > > 1. But, I was running the kafka-producer-perf-test.sh with acks=-1, >> > which >> > > > means that all data must have been committed to all replicas. But, >> with >> > > the >> > > > replicas still at 7GB, it doesnt seem that acks=-1 is considered by >> the >> > > > producer. >> > > > >> > > > Am I missing something ? >> > > > >> > > > Regards, >> > > > Prabhjot >> > > > >> > > >> > >> > > > > -- > --------------------------------------------------------- > "There are only 10 types of people in the world: Those who understand > binary, and those who don't" >
