Hi devs,
Some SQL Table properties contain sensitive data, like passwords that we do not
want to expose in the VVP ui to other users. Also, having them clear text in a
SQL statement is not secure. For example,
CREATE TABLE Orders (
`user` BIGINT,
product STRING,
order_time TIMESTAMP(3)
) WITH (
'connector' = 'kafka',
'properties.bootstrap.servers' = 'kafka-host-1:9093,kafka-host-2:9093',
'properties.security.protocol' = 'SSL',
'properties.ssl.key.password' = 'should-be-a-secret',
'properties.ssl.keystore.location' = '/tmp/secrets/my-keystore.jks',
'properties.ssl.keystore.password' = 'should-also-be-a-secret',
'properties.ssl.truststore.location' = '/tmp/secrets/my-truststore.jks',
'properties.ssl.truststore.password' = 'should-again-be-a-secret',
'scan.startup.mode' = 'earliest-offset'
);
I would like to bring up for a discussion a proposal to provide these secrets
values via environment variables since these can be populated from a K8s
configMap or secrets.
For implementing the SQL Table properties, the ConfigOption<T> class is used in
connectors and formatters. This class could be extended that it checks whether
the config-value contains certain tokens, like ‘${env-var-name}’. If it does,
it could fetch the value from the environment variable and use that to replace
that token in the config-value.
The above SQL statement would then look like,
CREATE TABLE Orders (
`user` BIGINT,
product STRING,
order_time TIMESTAMP(3)
) WITH (
'connector' = 'kafka',
'properties.bootstrap.servers' = 'kafka-host-1:9093,kafka-host-2:9093',
'properties.security.protocol' = 'SSL',
'properties.ssl.key.password' = '${secret_kafka_ssl_key_password}',
'properties.ssl.keystore.location' = '/tmp/secrets/my-keystore.jks',
'properties.ssl.keystore.password' =
'${secret_kafka_ssl_keystore_password}',
'properties.ssl.truststore.location' = '/tmp/secrets/my-truststore.jks',
'properties.ssl.truststore.password' =
'${secret_kafka_ssl_truststore_password}',
'scan.startup.mode' = 'earliest-offset'
);
For the purpose of secrets I don’t think you need any complex processing of
tokens but perhaps there are other usages as well. For instance,
'properties.bootstrap.servers' =
'kafka-${otap_env}-1:9093,kafka-${otap_env}-2:9093',
Because it is possible that (but I think unlikely) someone wants a property
value like ‘${not-an-env-var}’ you need to be able to escape this ’$’ token
like ‘$${not-an-env-var}’. This also means that in theory it would break
compatibility.
Looking forward for your feedback!
Best,
Fred Teunissen
-----------------------------------------------------------------
ATTENTION:
The information in this e-mail is confidential and only meant for the intended
recipient. If you are not the intended recipient, don't use or disclose it in
any way. Please let the sender know and delete the message immediately.
-----------------------------------------------------------------
