[
https://issues.apache.org/jira/browse/KAFKA-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish updated KAFKA-10340:
---------------------------------
Description:
Currently, a source connector will blindly attempt to write a record to a Kafka
topic. When the topic does not exist, its creation is controlled by the
{{auto.create.topics.enable}} config on the brokers. When auto.create is
disabled, the producer.send() call on the Connect worker will hang indefinitely
(due to the "infinite retries" configuration for said producer). In production
setups, where this config is usually disabled, the source connector simply
appears to hang and not produce any output.
It is desirable to either log an info or an error message (or inform the user
somehow) that the connector is simply stuck waiting for the destination topic
to be created. When the worker has permissions to inspect the broker settings,
it can use the {{listTopics}} and {{describeConfigs}} API in AdminClient to
check if the topic exists, the broker can {{auto.create.topics.enable}} topics,
and if these cases do not exist, either throw an error.
With the recently merged
[KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics],
this becomes even more specific a corner case: when topic creation settings
are enabled, the worker should handle the corner case where topic creation is
disabled, {{auto.create.topics.enable}} is set to false and topic does not
exist.
was:
Currently, a source connector will blindly attempt to write a record to a Kafka
topic. When the topic does not exist, its creation is controlled by ACLs and
the {{auto.create.topics.enable}} config on the brokers. When auto.create is
disabled, the producer.send() call on the Connect worker will hang indefinitely
(due to the "infinite retries" configuration for said producer). In production
setups, the config is usually disabled, and the source connector simply appears
to hang and not produce any output.
It is desirable to either log an info or an error message (or inform the user
somehow) that the connector is simply stuck waiting for the destination topic
to be created. When the worker has permissions to inspect the broker settings,
it can use the {{listTopics}} and {{describeConfigs}} API in AdminClient to
check if the topic exists, the broker can {{auto.create.topics.enable}} topics,
and if these cases do not exist, either throw an error.
With the recently merged
[KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics],
this becomes even more specific a corner case: when topic creation settings
are enabled, the worker should handle the corner case where topic creation is
disabled, {{auto.create.topics.enable}} is set to false and topic does not
exist.
> Source connectors should report error when trying to producer records to
> non-existent topics instead of hanging forever
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-10340
> URL: https://issues.apache.org/jira/browse/KAFKA-10340
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Reporter: Arjun Satish
> Priority: Major
>
> Currently, a source connector will blindly attempt to write a record to a
> Kafka topic. When the topic does not exist, its creation is controlled by the
> {{auto.create.topics.enable}} config on the brokers. When auto.create is
> disabled, the producer.send() call on the Connect worker will hang
> indefinitely (due to the "infinite retries" configuration for said producer).
> In production setups, where this config is usually disabled, the source
> connector simply appears to hang and not produce any output.
> It is desirable to either log an info or an error message (or inform the user
> somehow) that the connector is simply stuck waiting for the destination topic
> to be created. When the worker has permissions to inspect the broker
> settings, it can use the {{listTopics}} and {{describeConfigs}} API in
> AdminClient to check if the topic exists, the broker can
> {{auto.create.topics.enable}} topics, and if these cases do not exist, either
> throw an error.
> With the recently merged
> [KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics],
> this becomes even more specific a corner case: when topic creation settings
> are enabled, the worker should handle the corner case where topic creation is
> disabled, {{auto.create.topics.enable}} is set to false and topic does not
> exist.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
