That is right. And to determine whether you need multiple producers or not you can check io-wait-ratio on the producer, if it is close to 0 that means the ioThread is busy sending and hence you need multiple instances.
Guozhang On Tue, Jun 30, 2015 at 9:43 AM, Shushant Arora <shushantaror...@gmail.com> wrote: > In new java Producer is there only one ioThread irrespective of no of > brokers which send message from inmemory buffer to brokers sequentially? > > So if user need parallelism we should have multiple producer instances ? > > On Tue, Jun 30, 2015 at 10:02 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > ProducerSendThread and SyncProducer all come from the old Scala > producer, I > > thought you meant that producer not the new Java producer? > > > > Guozhang > > > > On Tue, Jun 30, 2015 at 9:09 AM, Shushant Arora < > shushantaror...@gmail.com > > > > > wrote: > > > > > According to code of org.apache.kafka.clients.producer.KafkaProducer. > > > > > > It has 1 ioThread > > > > > > private final Thread > > > < > > > > > > http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/Thread.java#Thread > > > > > > > ioThread; > > > > > > String < > > > > > > http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String > > > > > > > ioThreadName = "kafka-producer-network-thread" + (clientId.length > > > < > > > > > > http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String.length%28%29 > > > >() > > > > 0 ? " | " + clientId : ""); > > > > > > 210 < > > > > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#210 > > > > > > > > > > < > > > > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java# > > > > > > > > > > this.ioThread > > > < > > > > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0ioThread > > > > > > > = new KafkaThread > > > < > > > > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/common/utils/KafkaThread.java#KafkaThread > > > >(ioThreadName, > > > this.sender < > > > > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0sender > > > >, > > > true); > > > > > > 211 < > > > > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#211 > > > > > > > > > > < > > > > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java# > > > > > > > > > > this.ioThread > > > < > > > > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0ioThread > > > >.start > > > < > > > > > > http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/Thread.java#Thread.start%28%29 > > > >(); > > > > > > 212 < > > > > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#212 > > > > > > > > > > > > > is ioThread is the one responsible for sending message from inmemory > > buffer > > > to kafka broker . So where is SyncProducer got created and is > > SyncProducer > > > responsible for sending message from inmemory buffer to kafka broker > and > > > where is that got created? > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 30, 2015 at 9:16 PM, Guozhang Wang <wangg...@gmail.com> > > wrote: > > > > > > > One ProducerSendThread is always the case for producer, but inside > the > > > > thread there is one SyncProducer for each destination broker. I think > > > that > > > > blog may mis-understand the design a bit. > > > > > > > > Guozhang > > > > > > > > On Tue, Jun 30, 2015 at 1:45 AM, Shushant Arora < > > > shushantaror...@gmail.com > > > > > > > > > wrote: > > > > > > > > > According to > > > > > https://engineering.gnip.com/kafka-async-producer/ > > > > > There is one ProducerSendThread per broker. Is it changed from > > 0.8.2 ? > > > > to > > > > > use single send thread irrespective of broker or partitions. > > > > > > > > > > > > > > > On Mon, Jun 29, 2015 at 11:27 PM, Guozhang Wang < > wangg...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Shushant, > > > > > > > > > > > > The number of ProducerSendThread will be always one, regardless > of > > > the > > > > > > number of destination brokers, or the number of partitions. > > > > > > > > > > > > Guozhang > > > > > > > > > > > > On Mon, Jun 29, 2015 at 9:38 AM, Shushant Arora < > > > > > shushantaror...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi > > > > > > > > > > > > > > Does kafka async producer creates thread(ProducerSendThread) in > > > > > producer > > > > > > > memory based on no number of partitions or brokers in kafka > > cluster > > > > to > > > > > > > which it will write. > > > > > > > > > > > > > > If my cluster had 1000 partitions does each producer will have > > 1000 > > > > > > threads > > > > > > > running always ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > -- Guozhang > > > > > > > > > > > > > > > -- > > -- Guozhang > > > -- -- Guozhang
