In one of our application, we have the following setting:
# kafka configuration
# ~~~~~
kafka {
# comma seperated list of brokers
# for e.g., "localhost:9092,localhost:9032"
brokers = "localhost:9092,localhost:9032"
topic = "asset-computed-telemetry"
isEnabled = true
# for a detailed list of configuration options see
# under New Producer Configs
# http://kafka.apache.org/082/documentation.html#producerconfigs
requestRequiredAcks = 1
requestTimeout = 3.seconds
bufferMemoryBytes = "33554432"
blockOnBufferFull = false
# setting this to 0 indicates that the producer will never
# block and will just drop messages once the queue buffer to
# kafka broker is full
queueEnqueTimeoutMs = 0.seconds
producerType = "async"
messageSendMaxRetries = 1
}
As you can see from the configuration is that, we have a buffer of 33 MB.
Now I have one topic on my broker and this topic has 20 partitions. So what
I do in my producer application is that I create 10 instances of my
Producer and I write to the topic.
So each producer instance gets a copy of this configuration. Does this mean
that I will reserve 33 times 10 instances = 330 MB os space just for the
buffer? What if I have more and more topics in the future? Will I use all
the memory only for the buffer?
