Hi Magesh,
Thanks for the feedback!
I will put together a PR to demonstrate what the implementation might look
like, as well as a reference FileConfigProvider.
1. The delayMs for a (potentially) scheduled reload is determined by the
ConfigProvider. For example, a (hypothetical) VaultConfigProvider, upon
contacting Vault for a particular secret, might also obtain a lease
duration indicating that the secret expires in 1 hour. The
VaultConfigProvider could then call scheduleConfigReload with delayMs set
to 3600000ms (1 hour). This would cause the Connector to restart in an
hour, forcing it to reload the configs and re-resolve all indirect
references.
2. Yes, the start() methods in SourceTask and SinkTask would get the
configs with all the indirect references resolved. Those config() methods
are for Connectors that want to get the latest configs (with all indirect
references re-resolved) at some time after start(). For example, if a Task
encountered some security exception because a secret expired, it could call
config() to get the config with the latest values. This is assuming that
the Task can gracefully recover from the security exception.
3. Yes, that is up to the ConfigProvider implementation and is out of
scope. If the ConfigProvider also needs some kind of secrets or other
data, it could possibly be passed in through the param properties
("config.providers.vault.param.auth=/run/myauth"). For example Docker
might generate the auth info for Vault in an in-memory tmpfs file that
could then be passed as a param.
Thanks,
Robert
On Tue, May 8, 2018 at 10:10 PM, Magesh Nandakumar <mage...@confluent.io>
wrote:
> Hi Robert,
>
> Thanks for the KIP. I think, this will be a great addition to the
> framework. I think, will be great if the KIP can elaborate a little bit
> more on how implementations would look like with an example.
> Also, would be good to provide a reference implementation as well.
>
> The other questions I had were
>
> 1. How would the framework get the delayMs for void scheduleConfigReload(
> long delayMs);
> 2. Would the start methods in SourceTask and SinkTask get the configs with
> all the indirect references resolved. If so, trying to understand
> the intent of the config() in SourceTaskContext and the SinkTaskContext
> 3. What if the provider itself needs some kind of secrets to be configured
> to connect to it? I assume that's out of scope for this proposal but wanted
> to clarify it.
>
> Thanks
> Magesh
>
> On Tue, May 8, 2018 at 1:52 PM, Robert Yokota <rayok...@gmail.com> wrote:
>
> > Hi,
> >
> > I would like to start a discussion for KIP-297 to externalize secrets
> from
> > Kafka Connect configurations. Any feedback is appreciated.
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 297%3A+Externalizing+Secrets+for+Connect+Configurations
> > >
> >
> > JIRA: <https://issues.apache.org/jira/browse/KAFKA-6886>
> >
> > Thanks in advance,
> > Robert
> >
>
