Hi Magesh,
I updated the KIP with a link to a PR for a working prototype. The
prototype does not yet use the Connect plugin machinery for class loader
isolation, but should give you an idea of what the final implementation
will look like. Here is the link:
https://github.com/apache/kafka/pull/4990/files.
I also added an example of a FileConfigProvider to the KIP.
Thanks,
Robert
On Wed, May 9, 2018 at 10:04 AM, Robert Yokota <rayok...@gmail.com> wrote:
> Hi Magesh,
>
> Thanks for the feedback!
>
> I will put together a PR to demonstrate what the implementation might look
> like, as well as a reference FileConfigProvider.
>
> 1. The delayMs for a (potentially) scheduled reload is determined by the
> ConfigProvider. For example, a (hypothetical) VaultConfigProvider, upon
> contacting Vault for a particular secret, might also obtain a lease
> duration indicating that the secret expires in 1 hour. The
> VaultConfigProvider could then call scheduleConfigReload with delayMs set
> to 3600000ms (1 hour). This would cause the Connector to restart in an
> hour, forcing it to reload the configs and re-resolve all indirect
> references.
>
> 2. Yes, the start() methods in SourceTask and SinkTask would get the
> configs with all the indirect references resolved. Those config() methods
> are for Connectors that want to get the latest configs (with all indirect
> references re-resolved) at some time after start(). For example, if a Task
> encountered some security exception because a secret expired, it could call
> config() to get the config with the latest values. This is assuming that
> the Task can gracefully recover from the security exception.
>
> 3. Yes, that is up to the ConfigProvider implementation and is out of
> scope. If the ConfigProvider also needs some kind of secrets or other
> data, it could possibly be passed in through the param properties
> ("config.providers.vault.param.auth=/run/myauth"). For example Docker
> might generate the auth info for Vault in an in-memory tmpfs file that
> could then be passed as a param.
>
> Thanks,
> Robert
>
>
>
> On Tue, May 8, 2018 at 10:10 PM, Magesh Nandakumar <mage...@confluent.io>
> wrote:
>
>> Hi Robert,
>>
>> Thanks for the KIP. I think, this will be a great addition to the
>> framework. I think, will be great if the KIP can elaborate a little bit
>> more on how implementations would look like with an example.
>> Also, would be good to provide a reference implementation as well.
>>
>> The other questions I had were
>>
>> 1. How would the framework get the delayMs for void scheduleConfigReload(
>> long delayMs);
>> 2. Would the start methods in SourceTask and SinkTask get the configs with
>> all the indirect references resolved. If so, trying to understand
>> the intent of the config() in SourceTaskContext and the SinkTaskContext
>> 3. What if the provider itself needs some kind of secrets to be configured
>> to connect to it? I assume that's out of scope for this proposal but
>> wanted
>> to clarify it.
>>
>> Thanks
>> Magesh
>>
>> On Tue, May 8, 2018 at 1:52 PM, Robert Yokota <rayok...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I would like to start a discussion for KIP-297 to externalize secrets
>> from
>> > Kafka Connect configurations. Any feedback is appreciated.
>> > <
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > 297%3A+Externalizing+Secrets+for+Connect+Configurations
>> > >
>> >
>> > JIRA: <https://issues.apache.org/jira/browse/KAFKA-6886>
>> >
>> > Thanks in advance,
>> > Robert
>> >
>>
>
>
