Hi Folks, Request your expertise on my doubt here.
*My setup:-* 5 node kafka cluster (4 cores, 8GB RAM) on RAID-6 (500 GB) Using Kafka 0.8.2.1 with modified ProducerPerformance.scala I've modified ProducerPerformance.scala to send custom ASCII data, instead of Byte Array of Zeroes *server.properties:-* broker.id=0 log.cleaner.enable=false log.dirs=/tmp/kafka-logs log.retention.check.interval.ms=300000 log.retention.hours=168 log.segment.bytes=1073741824 num.io.threads=8 num.network.threads=3 num.partitions=1 num.recovery.threads.per.data.dir=1 *num.replica.fetchers=4* port=9092 socket.receive.buffer.bytes=1048576 socket.request.max.bytes=104857600 socket.send.buffer.bytes=1048576 zookeeper.connect=localhost:2181 zookeeper.connection.timeout.ms=6000 *This is how I run the producer perf test:-* kafka-producer-perf-test.sh --broker-list a.a.a.a:9092,b.b.b.b:9092,c.c.c.c:9092,d.d.d.d:9092,e.e.e.e:9092 --messages 100000 --message-size 500 --topics temp --show-detailed-stats --threads 5 --request-num-acks -1 --batch-size 200 --request-timeout-ms 10000 --compression-codec 0 *Problem:-* This test completes in under 15 seconds for me But, after this test, if I try writing to another topic which has 2 partitions and 3 replicas, it is dead slow and the same script seems never to finish because the slow ISR catch-up is still going on. *My inference:-* I have noticed that for a topic with 1 partition and 3 replicas, the ISR shows only 1 broker id. Topic:temp PartitionCount:1 ReplicationFactor:3 Configs: Topic: temp Partition: 0 Leader: 5 Replicas: 5,1,2 Isr: 5 I think it is because the data from the leader is not received in broker ids 1 and 2 Also, I could confirm it from the data directory sizes for this topic. Leader (5) has 20GB but replicas - 1 and 2 are still at 7GB *Doubts:-* 1. But, I was running the kafka-producer-perf-test.sh with acks=-1, which means that all data must have been committed to all replicas. But, with the replicas still at 7GB, it doesnt seem that acks=-1 is considered by the producer. Am I missing something ? Regards, Prabhjot
