It puts things on several internal queues. I'd benchmark what kind of rates
you're looking at - we handily do a few hundred thousand per second per
process with a 2GB JVM heap.
On Mon, May 23, 2016 at 1:31 PM, Joe San <codeintheo...@gmail.com> wrote:
> When you say Threadsafe, I assume that the calls to the send method is
> Synchronized or? If it is the case, I see this as a bottleneck. We have a
> high frequency system and the frequency at which the calls are made to the
> send method is very high. This was the reason why I came up with multiple
> instances of the producer!
>
> On Mon, May 23, 2016 at 2:22 PM, Tom Crayford <tcrayf...@heroku.com>
> wrote:
>
> > That's accurate. Why are you creating so many producers? The Kafka
> producer
> > is thread safe and *should* be shared to take advantage of batching, so
> I'd
> > recommend just having a single producer.
> >
> > Thanks
> >
> > Tom Crayford
> > Heroku Kafka
> >
> > On Mon, May 23, 2016 at 10:41 AM, Joe San <codeintheo...@gmail.com>
> wrote:
> >
> > > In one of our application, we have the following setting:
> > >
> > > # kafka configuration
> > > # ~~~~~
> > > kafka {
> > > # comma seperated list of brokers
> > > # for e.g., "localhost:9092,localhost:9032"
> > > brokers = "localhost:9092,localhost:9032"
> > > topic = "asset-computed-telemetry"
> > > isEnabled = true
> > > # for a detailed list of configuration options see
> > > # under New Producer Configs
> > > # http://kafka.apache.org/082/documentation.html#producerconfigs
> > > requestRequiredAcks = 1
> > > requestTimeout = 3.seconds
> > > bufferMemoryBytes = "33554432"
> > > blockOnBufferFull = false
> > > # setting this to 0 indicates that the producer will never
> > > # block and will just drop messages once the queue buffer to
> > > # kafka broker is full
> > > queueEnqueTimeoutMs = 0.seconds
> > > producerType = "async"
> > > messageSendMaxRetries = 1
> > > }
> > >
> > > As you can see from the configuration is that, we have a buffer of 33
> MB.
> > > Now I have one topic on my broker and this topic has 20 partitions. So
> > what
> > > I do in my producer application is that I create 10 instances of my
> > > Producer and I write to the topic.
> > >
> > > So each producer instance gets a copy of this configuration. Does this
> > mean
> > > that I will reserve 33 times 10 instances = 330 MB os space just for
> the
> > > buffer? What if I have more and more topics in the future? Will I use
> all
> > > the memory only for the buffer?
> > >
> >
>
