[
https://issues.apache.org/jira/browse/KAFKA-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994024#comment-16994024
]
ASF GitHub Bot commented on KAFKA-9228:
---------------------------------------
C0urante commented on pull request #7823: KAFKA-9228: Force reconfiguration of
tasks if converter or Kafka client configs are changed
URL: https://github.com/apache/kafka/pull/7823
[Jira](https://issues.apache.org/jira/browse/KAFKA-9228)
There's a subtle bug in the Connect framework that hasn't been a common
issue due to the patterns many connector developers follow for their
`Connector::taskConfigs` method (more on the cause of this is outlined in the
Jira issue). It's possible that reconfiguring a connector but only altering its
converters and/or Kafka clients will fail to propagate to the connector's tasks.
This change should fix that bug by caching the client and converter configs
for running connectors in the herder and, if a change is detected to any of
those configs, force a restart of the connector's tasks. This is done in both
the standalone and distributed herder.
A unit test is added for the shared converter/client config extraction
logic, and an integration test is added to confirm the fix which fails on the
current `trunk` but passes on the source branch of this PR.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Reconfigured converters and clients may not be propagated to connector tasks
> ----------------------------------------------------------------------------
>
> Key: KAFKA-9228
> URL: https://issues.apache.org/jira/browse/KAFKA-9228
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 2.3.0, 2.4.0, 2.3.1, 2.3.2
> Reporter: Chris Egerton
> Assignee: Chris Egerton
> Priority: Major
>
> If an existing connector is reconfigured but the only changes are to its
> converters and/or Kafka clients (enabled as of
> [KIP-458|https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy]),
> the changes will not propagate to its tasks unless the connector also
> generates task configs that differ from the existing task configs. Even after
> this point, if the connector tasks are reconfigured, they will still not pick
> up on the new converter and/or Kafka client configs.
> This is because the {{DistributedHerder}} only writes new task configurations
> to the connect config topic [if the connector-provided task configs differ
> from the task configs already in the config
> topic|https://github.com/apache/kafka/blob/e499c960e4f9cfc462f1a05a110d79ffa1c5b322/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1285-L1332],
> and neither of those contain converter or Kafka client configs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
