Dave Sloan created KAFKA-14871:
----------------------------------
Summary: Kafka Connect - TTL not respected for Config
Provider-provided connector configurations
Key: KAFKA-14871
URL: https://issues.apache.org/jira/browse/KAFKA-14871
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 3.3.2
Reporter: Dave Sloan
When defining a configuration provider using environment variables (eg via
Docker), then a reload is scheduled according to the TTL of the returned
configuration.
Here is an example:
|Environment Variable|Value|
|CONNECT_CONFIG_PROVIDERS_VAULT_PARAM_VAULT_ENGINE_VERSION|2|
|CONNECT_CONFIG_PROVIDERS_VAULT_PARAM_VAULT_TOKEN|9c08104f-98b7-4bce-ab86-4a6c63897ec4|
|CONNECT_CONFIG_PROVIDERS|vault|
|CONNECT_CONFIG_PROVIDERS_VAULT_PARAM_DEFAULT_TTL|30000|
|CONNECT_CONFIG_PROVIDERS_VAULT_PARAM_VAULT_ADDR|http://vault:8200|
|CONNECT_CONFIG_PROVIDERS_VAULT_PARAM_VAULT_AUTH_METHOD|token|
|CONNECT_CONFIG_PROVIDERS_VAULT_PARAM_FILE_WRITE|false|
|CONNECT_CONFIG_PROVIDERS_VAULT_CLASS|io.lenses.connect.secrets.providers.VaultSecretProvider|
{code:java}
{
"name": "testSink",
"config": {
"topics": "vaultTest",
"name": "testSink",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"test.sink.secret.value":
"${vault:rotate-test/myVaultSecretPath:myVaultSecretKey}",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"tasks.max": 1,
}
}{code}
And this is what we see in the logs:
{code:java}
STDOUT: [2023-03-30 07:46:43,204] INFO Scheduling a restart of connector
testSink in 4908 ms
(org.apache.kafka.connect.runtime.WorkerConfigTransformer){code}
However when we try and achieve the same via connector properties, no restart
is scheduled
{code:java}
{
"name": "testSink",
"config": {
"topics": "vaultTest",
"name": "testSink",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"tasks.max": 1,
"connector.class": "io.lenses.connect.secrets.test.TestSinkConnector",
"test.sink.secret.value":
"${vault:rotate-test/myVaultSecretPath:myVaultSecretKey}",
"config.providers": "vault",
"config.providers.vault.class":
"io.lenses.connect.secrets.providers.VaultSecretProvider",
"config.providers.vault.param.vault.engineversion": 2,
"config.providers.vault.param.vault.token":
"9c08104f-98b7-4bce-ab86-4a6c63897ec4",
"config.providers.vault.param.default.ttl": 30000,
"config.providers.vault.param.vault.addr": "http://vault:8200";,
"config.providers.vault.param.vault.auth.method": "token",
"config.providers.vault.param.file.write": false
}
}{code}
Upon looking deeper into the code I can see that on line 239 of AbstractConfig
[https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java#L539]
{code:java}
ConfigTransformerResult result =
configTransformer.transform(indirectVariables);{code}
The result contains the TTLs. However these are not used.
Expectation:
TTLs should be used to schedule a restart of the connector so that the
behaviour is the same as if using environment properties.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
