In new java Producer is there only one ioThread irrespective of no of brokers which send message from inmemory buffer to brokers sequentially?
So if user need parallelism we should have multiple producer instances ? On Tue, Jun 30, 2015 at 10:02 PM, Guozhang Wang <wangg...@gmail.com> wrote: > ProducerSendThread and SyncProducer all come from the old Scala producer, I > thought you meant that producer not the new Java producer? > > Guozhang > > On Tue, Jun 30, 2015 at 9:09 AM, Shushant Arora <shushantaror...@gmail.com > > > wrote: > > > According to code of org.apache.kafka.clients.producer.KafkaProducer. > > > > It has 1 ioThread > > > > private final Thread > > < > > > http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/Thread.java#Thread > > > > > ioThread; > > > > String < > > > http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String > > > > > ioThreadName = "kafka-producer-network-thread" + (clientId.length > > < > > > http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/String.java#String.length%28%29 > > >() > > > 0 ? " | " + clientId : ""); > > > > 210 < > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#210 > > > > > > > < > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java# > > > > > > > this.ioThread > > < > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0ioThread > > > > > = new KafkaThread > > < > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/common/utils/KafkaThread.java#KafkaThread > > >(ioThreadName, > > this.sender < > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0sender > > >, > > true); > > > > 211 < > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#211 > > > > > > > < > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java# > > > > > > > this.ioThread > > < > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#KafkaProducer.0ioThread > > >.start > > < > > > http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b27/java/lang/Thread.java#Thread.start%28%29 > > >(); > > > > 212 < > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.kafka/kafka-clients/0.8.2.0/org/apache/kafka/clients/producer/KafkaProducer.java#212 > > > > > > > > > is ioThread is the one responsible for sending message from inmemory > buffer > > to kafka broker . So where is SyncProducer got created and is > SyncProducer > > responsible for sending message from inmemory buffer to kafka broker and > > where is that got created? > > > > > > > > > > > > > > On Tue, Jun 30, 2015 at 9:16 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > One ProducerSendThread is always the case for producer, but inside the > > > thread there is one SyncProducer for each destination broker. I think > > that > > > blog may mis-understand the design a bit. > > > > > > Guozhang > > > > > > On Tue, Jun 30, 2015 at 1:45 AM, Shushant Arora < > > shushantaror...@gmail.com > > > > > > > wrote: > > > > > > > According to > > > > https://engineering.gnip.com/kafka-async-producer/ > > > > There is one ProducerSendThread per broker. Is it changed from > 0.8.2 ? > > > to > > > > use single send thread irrespective of broker or partitions. > > > > > > > > > > > > On Mon, Jun 29, 2015 at 11:27 PM, Guozhang Wang <wangg...@gmail.com> > > > > wrote: > > > > > > > > > Hi Shushant, > > > > > > > > > > The number of ProducerSendThread will be always one, regardless of > > the > > > > > number of destination brokers, or the number of partitions. > > > > > > > > > > Guozhang > > > > > > > > > > On Mon, Jun 29, 2015 at 9:38 AM, Shushant Arora < > > > > shushantaror...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > Hi > > > > > > > > > > > > Does kafka async producer creates thread(ProducerSendThread) in > > > > producer > > > > > > memory based on no number of partitions or brokers in kafka > cluster > > > to > > > > > > which it will write. > > > > > > > > > > > > If my cluster had 1000 partitions does each producer will have > 1000 > > > > > threads > > > > > > running always ? > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > -- > -- Guozhang >
