Yes, this should work. Expect lower throughput though. On Thu, Sep 3, 2015 at 12:52 PM, Prabhjot Bharaj <prabhbha...@gmail.com> wrote:
> Hi, > > Can I use sync for acks = -1? > > Regards, > Prabhjot > On Sep 3, 2015 11:49 PM, "Gwen Shapira" <g...@confluent.io> wrote: > > > The test uses the old producer (we should fix that), and since you don't > > specify --sync, it runs async. > > The old async producer simply sends data and doesn't wait for acks, so it > > is possible that the messages were never acked... > > > > On Thu, Sep 3, 2015 at 7:56 AM, Prabhjot Bharaj <prabhbha...@gmail.com> > > wrote: > > > > > Hi Folks, > > > > > > Request your expertise on my doubt here. > > > > > > *My setup:-* > > > > > > 5 node kafka cluster (4 cores, 8GB RAM) on RAID-6 (500 GB) > > > Using Kafka 0.8.2.1 with modified ProducerPerformance.scala > > > I've modified ProducerPerformance.scala to send custom ASCII data, > > instead > > > of Byte Array of Zeroes > > > > > > *server.properties:-* > > > > > > broker.id=0 > > > > > > log.cleaner.enable=false > > > > > > log.dirs=/tmp/kafka-logs > > > > > > log.retention.check.interval.ms=300000 > > > > > > log.retention.hours=168 > > > > > > log.segment.bytes=1073741824 > > > > > > num.io.threads=8 > > > > > > num.network.threads=3 > > > > > > num.partitions=1 > > > > > > num.recovery.threads.per.data.dir=1 > > > > > > *num.replica.fetchers=4* > > > > > > port=9092 > > > > > > socket.receive.buffer.bytes=1048576 > > > > > > socket.request.max.bytes=104857600 > > > > > > socket.send.buffer.bytes=1048576 > > > > > > zookeeper.connect=localhost:2181 > > > > > > zookeeper.connection.timeout.ms=6000 > > > > > > > > > *This is how I run the producer perf test:-* > > > > > > kafka-producer-perf-test.sh --broker-list > > > a.a.a.a:9092,b.b.b.b:9092,c.c.c.c:9092,d.d.d.d:9092,e.e.e.e:9092 > > --messages > > > 100000 --message-size 500 --topics temp --show-detailed-stats > --threads > > 5 > > > --request-num-acks -1 --batch-size 200 --request-timeout-ms 10000 > > > --compression-codec 0 > > > > > > *Problem:-* > > > > > > This test completes in under 15 seconds for me > > > > > > But, after this test, if I try writing to another topic which has 2 > > > partitions and 3 replicas, it is dead slow and the same script seems > > never > > > to finish because the slow ISR catch-up is still going on. > > > > > > *My inference:-* > > > I have noticed that for a topic with 1 partition and 3 replicas, the > ISR > > > shows only 1 broker id. > > > > > > Topic:temp PartitionCount:1 ReplicationFactor:3 Configs: > > > > > > Topic: temp Partition: 0 Leader: 5 Replicas: 5,1,2 Isr: 5 > > > > > > > > > I think it is because the data from the leader is not received in > broker > > > ids 1 and 2 > > > Also, I could confirm it from the data directory sizes for this topic. > > > Leader (5) has 20GB but replicas - 1 and 2 are still at 7GB > > > > > > *Doubts:-* > > > 1. But, I was running the kafka-producer-perf-test.sh with acks=-1, > which > > > means that all data must have been committed to all replicas. But, with > > the > > > replicas still at 7GB, it doesnt seem that acks=-1 is considered by the > > > producer. > > > > > > Am I missing something ? > > > > > > Regards, > > > Prabhjot > > > > > >
