[
https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872645#comment-15872645
]
Stevo Slavic commented on KAFKA-4385:
-------------------------------------
[~Jun Yao] [~ewencp] it would open up more options if producer was made aware
whether topic auto creation is enabled at all (e.g. through producer
configuration, or through producer on startup requesting configuration from
broker). Nevertheless, I think it would be enough to make model of metadata
cache in producer more rich, it should keep for every topic:
- last known metadata, but also
- last metadata request status error code
Then {{KafkaProducer}} thread waiting for metadata, at least on timeout could
check if there's no metadata and last error code is {{UnknownTopicOrPartition}}
to then throw {{UnknownTopicOrPartitionException}}, otherwise to throw
{{TimeoutException}}. For that no protocol change is needed. Maybe metadata
cache doesn't already keep last metadata request status error code per topic -
that and mentioned timeout handling logic change should be enough to be able to
distinguish between non-existing topic and timeout.
If {{KafkaProducer}} was aware of auto topic creation configuration
(enabled/disabled), then it could give up sooner, but that's optimization, with
arguable value while (at least option where configuration is obtained from
brokers) it might require protocol change. Somewhat related KAFKA-2948 already
makes more efficient one aspect - makes sure that topic metadata for
non-existing or no-longer-existing topics will eventually stop being retrieved.
> producer is sending too many unnecessary meta data request if the meta data
> for a topic is not available and "auto.create.topics.enable" =false
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-4385
> URL: https://issues.apache.org/jira/browse/KAFKA-4385
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.9.0.1
> Reporter: Jun Yao
>
> All current kafka-client producer implementation (<= 0.10.1.0),
> When sending a msg to a topic, it will first check if meta data for this
> topic is available or not,
> when not available, it will set "metadata.requestUpdate()" and wait for meta
> data from brokers,
> The thing is inside "org.apache.kafka.clients.Metadata.awaitUpdate()", it's
> already doing a "while (this.version <= lastVersion)" loop waiting for new
> version response,
> So the loop inside
> "org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata() is not
> needed,
> When "auto.create.topics.enable" is false, sending msgs to a non-exist topic
> will trigger too many meta requests, everytime a metadata response is
> returned, because it does not contain the metadata for the topic, it's going
> to try again until TimeoutException is thrown;
> This is a waste and sometimes causes too much overhead when unexpected msgs
> are arrived.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
