Sergey Ivanov created KAFKA-16838:
-------------------------------------

             Summary: Kafka Connect loads old tasks from removed connectors
                 Key: KAFKA-16838
                 URL: https://issues.apache.org/jira/browse/KAFKA-16838
             Project: Kafka
          Issue Type: Bug
          Components: connect
    Affects Versions: 3.6.1, 3.5.1, 3.8.0
            Reporter: Sergey Ivanov


Hello,

When creating connector we faced an error from one of our ConfigProviders about 
not existing resource, but we didn't try to set that resource as config value:
{code:java}
[2024-05-24T12:08:24.362][ERROR][request_id= ][tenant_id= 
][thread=DistributedHerder-connect-1-1][class=org.apache.kafka.connect.runtime.distributed.DistributedHerder][method=lambda$reconfigureConnectorTasksWithExponentialBackoffRetries$44]
 [Worker clientId=connect-1, groupId=streaming-service_streaming_service] 
Failed to reconfigure connector's tasks (local-file-sink), retrying after 
backoff.
org.apache.kafka.common.config.ConfigException: Could not read properties from 
file /opt/kafka/provider.properties
 at 
org.apache.kafka.common.config.provider.FileConfigProvider.get(FileConfigProvider.java:98)
 at 
org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)
 at 
org.apache.kafka.connect.runtime.WorkerConfigTransformer.transform(WorkerConfigTransformer.java:58)
 at 
org.apache.kafka.connect.storage.ClusterConfigState.taskConfig(ClusterConfigState.java:181)
 at 
org.apache.kafka.connect.runtime.AbstractHerder.taskConfigsChanged(AbstractHerder.java:804)
 at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.publishConnectorTaskConfigs(DistributedHerder.java:2089)
 at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.reconfigureConnector(DistributedHerder.java:2082)
 at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.reconfigureConnectorTasksWithExponentialBackoffRetries(DistributedHerder.java:2025)
 at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$null$42(DistributedHerder.java:2038)
 at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.runRequest(DistributedHerder.java:2232)
 at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:470)
 at 
org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:371)
 at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
 at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
 at java.base/java.lang.Thread.run(Thread.java:840)
 {code}
After investigation we found out, that few months ago on that cloud there was 
the connector with the same name and another value for config provider. Then it 
was removed, but by some reason when we tried to create connector with the same 
name months ago AbstractHerder tried to update tasks from our previous connector

As an example I use FileConfigProvider, but actually any ConfigProvider is 
accceptable which could raise exception if something wrong with config (like 
result doesn't exist).

We continued our investigation and found the issue 
https://issues.apache.org/jira/browse/KAFKA-7745 that says Connect doesn't 
tombstone commit and task messages in the config topic of Kafka. As we remember 
config topic is `compact` *that means commit and tasks are stored every time 
(months, years after connector removing)* and impact further connector 
creations with the same name.

We didn't investigate reasons in ConfigClusterStore and how to avoid that 
issue, because would {+}like to ask{+}, probably it's better to fix KAFKA-7745 
and send tombstones for commit and task messages as connect does for connector 
and target messages?

I have synthetic TC to reproduce that error if needed.

 

This is linked with https://issues.apache.org/jira/browse/KAFKA-16837 but it's 
not the same issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to