Frank, The inconsistentConnectors method is related to an extremely specific inconsistency that can happen when a worker writes some task configurations, and then disconnects without writing a following "commit tasks record" to the config topic. This is a hold-over from the early days of connect from before Kafka's transactions support, and is mostly an implementation detail. See the `KafkaConfigBackingStore::putTaskConfigs` and `KafkaConfigBackingStore::processTasksCommitRecord` for the relevant code. It is not expected that this method is in regular use, and is primarily for diagnostic purposes.
What the Strimzi issue seems to describe (and probably the issue you are facing) occurs at a higher level, when a worker is deciding whether to write new task configs at all. The relevant code is here: https://github.com/apache/kafka/blob/6e2b86597d9cd7c8b2019cffb895522deb63c93a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1918-L1931 In that snippet, new task configs generated by the connector are only written to the config topic if they differ from the current contents of the config topic. And this comparison is done on the post-transformation configurations, after ConfigProviders have been resolved. And critical for this bug, that resolution is done twice in quick succession, when the old and new configuration could evaluate to the same final result. The code snippet also shows why your workaround works: the other condition for writing all of the task configs to the config topic is that the number of configurations has changed. I believe this bug is captured in https://issues.apache.org/jira/browse/KAFKA-9228 but it has not progressed in some time. There is a potentially lower-impact workaround that involves adding a nonce to your connector configuration that changes each time you apply a new configuration to the connector, which most connectors will pass directly to their tasks. But this unfortunately does not work in general, as connectors could exclude the nonce when generating task configurations. I hope this gives some more insight to the behavior you're seeing. Thanks, Greg Harris On Fri, Feb 3, 2023 at 7:36 AM Frank Grimes <frankgrime...@yahoo.com.invalid> wrote: > Hi, we're investigating an issue where occasionally config changes don't > propagate to connectors/tasks. > > When this occurs, the only way to ensure that the configuration takes > effect is to resize the number of tasks back down to 1 and then resize back > up to the original number of tasks. > In searching for others who have been bitten by this scenario we found the > following thread on the Strimzi discussions pages: > https://github.com/strimzi/strimzi-kafka-operator/discussions/7738 > Both the symptoms and workaround described there match what we've > seen.We've been doing some digging into the Kafka Connect codebase to > better understand how config.storage.topic is consumed. > In the interest of brevity I won't repeat that entire thread of discussion > here. > However, I was wondering if anyone knows whether the JavaDoc suggestion on > ClusterConfigState.inconsistentConnectors() is actually implemented in the > clustered Worker code.i.e. "When a worker detects a connector in this > state, it should request that the connector regenerate its task > configurations." > The reason I ask is because I couldn't find any references to that API > call anywhere but in the KafkaConfigBackingStoreTest unit test cases. > Thanks! >
