[
https://issues.apache.org/jira/browse/KAFKA-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710972#comment-17710972
]
Chris Egerton commented on KAFKA-14746:
---------------------------------------
Hi Yash,
{quote}are you suggesting a KIP to modify the existing retry mechanism to not
cover exceptions thrown from connectors' taskConfigs method?
{quote}
Yep, exactly. But it's not high-priority and given Mickael's thoughts I don't
think we should move forward with it for now. The suggestion to at least
document this behavior is plenty (although we'll have to think about how to
handle the discrepancy in behavior between distributed and standalone modes).
{quote}Why would this require a KIP if this retry mechanism isn't part of the
public API?
{quote}
Some developers may have written connectors around this logic to get automatic
retries for free. Changing that behavior has the potential to break those
connectors. I also think that, even though we don't document this behavior in,
e.g., the {{taskConfigs}} Javadoc, it is still part of public API since it
directly affects how we interact with connectors.
> Throwing in Connector.taskConfigs in distributed mode generates a lot of logs
> -----------------------------------------------------------------------------
>
> Key: KAFKA-14746
> URL: https://issues.apache.org/jira/browse/KAFKA-14746
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Reporter: Mickael Maison
> Priority: Major
>
> If a Connector throws in its taskConfigs() method, the runtime ends up
> retrying using DistributedHerder.RECONFIGURE_CONNECTOR_TASKS_BACKOFF_MS which
> is a fixed value (250ms). For each retry, the runtime prints the connector
> configuration and the enriched configuration so this can quickly generate a
> lot of logs.
> There is some value in throwing in taskConfigs() as it allows to fail fast in
> case the connector is given bad credentials. For example this is what some of
> the Debezium connectors do:
> https://github.com/debezium/debezium/blob/main/debezium-connector-sqlserver/src/main/java/io/debezium/connector/sqlserver/SqlServerConnector.java#L56-L69
> The way Connectors are expected to work today is to instead always create
> tasks and let each task fail in case the configuration is wrong. We should
> document that and make it clear in the javadoc that throwing in taskConfigs
> is not recommended.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
