[
https://issues.apache.org/jira/browse/KAFKA-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17174105#comment-17174105
]
Luke Chen commented on KAFKA-10340:
-----------------------------------
Will improve the logging in this area.
Proposed to add 3 logs:
1. When the topic creation is disabled or topic is existed:
log.debug({color:#008000}"The topic creation setting is disabled or the topic
name {} is already created. " {color}+
{color:#008000}"If the topic doesn't exist, we'll rely on the
auto.create.topics.enable setting in broker side " {color}+
{color:#008000}"to see if the topic can be auto created or not"{color}, topic);
2. Before the producer send the record:
log.trace({color:#008000}"{} is going to send record to {}"{color},
WorkerSourceTask.{color:#000080}this{color}, topic);
3. If producer send record failed and topic is not existed and topic creation
is disabled:
log.info({color:#008000}"The reason of the error might be the server disabled
the " {color}+
{color:#008000}"auto.create.topics.enable setting, please check the broker
setting."{color});
> Source connectors should report error when trying to produce records to
> non-existent topics instead of hanging forever
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-10340
> URL: https://issues.apache.org/jira/browse/KAFKA-10340
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Reporter: Arjun Satish
> Assignee: Luke Chen
> Priority: Major
>
> Currently, a source connector will blindly attempt to write a record to a
> Kafka topic. When the topic does not exist, its creation is controlled by the
> {{auto.create.topics.enable}} config on the brokers. When auto.create is
> disabled, the producer.send() call on the Connect worker will hang
> indefinitely (due to the "infinite retries" configuration for said producer).
> In setups where this config is usually disabled, the source connector simply
> appears to hang and not produce any output.
> It is desirable to either log an info or an error message (or inform the user
> somehow) that the connector is simply stuck waiting for the destination topic
> to be created. When the worker has permissions to inspect the broker
> settings, it can use the {{listTopics}} and {{describeConfigs}} API in
> AdminClient to check if the topic exists, the broker can
> {{auto.create.topics.enable}} topics, and if these cases do not exist, either
> throw an error.
> With the recently merged
> [KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics],
> this becomes even more specific a corner case: when topic creation settings
> are enabled, the worker should handle the corner case where topic creation is
> disabled, {{auto.create.topics.enable}} is set to false and topic does not
> exist.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
