[
https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15642343#comment-15642343
]
Jun Yao commented on KAFKA-4385:
--------------------------------
Thanks, I tried changing 'retry.backoff.ms', it does help reducing the
duplicate requests. The thing is this "retry.backoff.ms" seems to be a global
setting that during my test it blocked other normal msgs (whose topic exist)
after the wrong msg.
Like below, msg0 is blocking all the msgs after it.
msg0 (topic does not exist)
msg1 (topic exist)
msg2 (topic exist)
msg3 (topic exist)
msg4 (topic exist)
In this case, as a user, I am more expecting fast failure (because I know keep
retrying metadata does not make sense, so I hope there will be one config for
me to just skip further metadata request on failure) rather than keep looping
the metadata request (slow down retrying is kind of similar),
for max.block.ms, this is also a global setting which will impact all, changing
it specifically for this reason seems a bit tricky to me.
for the option of "reuse the retries configuration for metadata refresh in
waitOnMetadata()", not sure if I understand you correctly or not, are you
suggesting to do the for loop with the general producer config of "retries"?
if that's what you suggest, I am not sure if you will be comfortable to do this
or not. I am more concerned to use this config.
As the global "retries" setting will have a bigger impact, couple of things
here, the default value is 0, then we may need to at least do "retries"+1.
meanwhile another concern is for normal case of
"auto.create.topics.enable=true", as you said topic creation is async, so we
may still want to do as much loop as possible within
"metadata.fetch.timeout.ms".
like for normal msgs we may want to set retries=2, but this may not be enough
for metadata request if topic is created automatically.
> producer is sending too many unnecessary meta data request if the meta data
> for a topic is not available and "auto.create.topics.enable" =false
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-4385
> URL: https://issues.apache.org/jira/browse/KAFKA-4385
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Yao
>
> All current kafka-client producer implementation (<= 0.10.1.0),
> When sending a msg to a topic, it will first check if meta data for this
> topic is available or not,
> when not available, it will set "metadata.requestUpdate()" and wait for meta
> data from brokers,
> The thing is inside "org.apache.kafka.clients.Metadata.awaitUpdate()", it's
> already doing a "while (this.version <= lastVersion)" loop waiting for new
> version response,
> So the loop inside
> "org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata() is not
> needed,
> When "auto.create.topics.enable" is false, sending msgs to a non-exist topic
> will trigger too many meta requests, everytime a metadata response is
> returned, because it does not contain the metadata for the topic, it's going
> to try again until TimeoutException is thrown;
> This is a waste and sometimes causes too much overhead when unexpected msgs
> are arrived.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
