That's accurate. Why are you creating so many producers? The Kafka producer
is thread safe and *should* be shared to take advantage of batching, so I'd
recommend just having a single producer.
Thanks
Tom Crayford
Heroku Kafka
On Mon, May 23, 2016 at 10:41 AM, Joe San <codeintheo...@gmail.com> wrote:
> In one of our application, we have the following setting:
>
> # kafka configuration
> # ~~~~~
> kafka {
> # comma seperated list of brokers
> # for e.g., "localhost:9092,localhost:9032"
> brokers = "localhost:9092,localhost:9032"
> topic = "asset-computed-telemetry"
> isEnabled = true
> # for a detailed list of configuration options see
> # under New Producer Configs
> # http://kafka.apache.org/082/documentation.html#producerconfigs
> requestRequiredAcks = 1
> requestTimeout = 3.seconds
> bufferMemoryBytes = "33554432"
> blockOnBufferFull = false
> # setting this to 0 indicates that the producer will never
> # block and will just drop messages once the queue buffer to
> # kafka broker is full
> queueEnqueTimeoutMs = 0.seconds
> producerType = "async"
> messageSendMaxRetries = 1
> }
>
> As you can see from the configuration is that, we have a buffer of 33 MB.
> Now I have one topic on my broker and this topic has 20 partitions. So what
> I do in my producer application is that I create 10 instances of my
> Producer and I write to the topic.
>
> So each producer instance gets a copy of this configuration. Does this mean
> that I will reserve 33 times 10 instances = 330 MB os space just for the
> buffer? What if I have more and more topics in the future? Will I use all
> the memory only for the buffer?
>
