Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-21 Thread Gwen Shapira
Very, very minor, but given the aim to possibly reuse > > > > > elsewhere > > > > > > and the fact that it'll likely end up in the common packages > might > > > mean > > > > > > devs focused more on the common/core packa

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-21 Thread Guozhang Wang
focusing on the broker on where they think it should go > since I > > > > think > > > > > it would be very natural to extend this to security settings there. > > > > (Also, > > > > > I think ConfigData is left out of the list of new interfaces

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-18 Thread Randall Hauch
gt; > it would be very natural to extend this to security settings there. > > > (Also, > > > > I think ConfigData is left out of the list of new interfaces by > > accident, > > > > but I think it's clear what's being added anyway.) > > > &

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-18 Thread Rajini Sivaram
glanced past it, but we're not shipping any > ConfigProviders > > > out of the box? This mentions file and vault, but just as examples. > Just > > > want to make sure everyone knows up front that this is a pluggable API, > > but > > > you need to

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-18 Thread Robert Yokota
you need to add more jars to take advantage of it. I think this is fine > as > > I don't think there are truly common secrets provider > > formats/apis/protocols, just want to make sure it is clear. > > > > Thanks, > > Ewen > > > > On Thu, May 17, 2018 a

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-18 Thread Robert Yokota
t is clear. > > Thanks, > Ewen > > On Thu, May 17, 2018 at 6:19 PM Ted Yu wrote: > > > +1 > > Original message ----From: Magesh Nandakumar < > > mage...@confluent.io> Date: 5/17/18 6:05 PM (GMT-08:00) To: > > dev@kafka.apache.org Subje

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-18 Thread Rajini Sivaram
on secrets provider > formats/apis/protocols, just want to make sure it is clear. > > Thanks, > Ewen > > On Thu, May 17, 2018 at 6:19 PM Ted Yu wrote: > > > +1 > > -------- Original message ----From: Magesh Nandakumar < > > mage...@confluent.io> D

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-17 Thread Ewen Cheslack-Postava
nfluent.io> Date: 5/17/18 6:05 PM (GMT-08:00) To: > dev@kafka.apache.org Subject: Re: [VOTE] KIP-297: Externalizing Secrets > for Connect Configurations > Thanks Robert, this looks great > > +1 (non-binding) > > On Thu, May 17, 2018 at 5:35 PM, Colin McCabe wrote: > > > T

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-17 Thread Ted Yu
+1 Original message From: Magesh Nandakumar Date: 5/17/18 6:05 PM (GMT-08:00) To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations Thanks Robert, this looks great +1 (non-binding) On Thu, May 17, 2018 at 5:35 PM, Colin

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-17 Thread Magesh Nandakumar
Thanks Robert, this looks great +1 (non-binding) On Thu, May 17, 2018 at 5:35 PM, Colin McCabe wrote: > Thanks, Robert! > > +1 (non-binding) > > Colin > > > On Thu, May 17, 2018, at 14:15, Robert Yokota wrote: > > Hi Colin, > > > > I've changed the KIP to have a composite object returned from g

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-17 Thread Colin McCabe
Thanks, Robert! +1 (non-binding) Colin On Thu, May 17, 2018, at 14:15, Robert Yokota wrote: > Hi Colin, > > I've changed the KIP to have a composite object returned from get(). It's > probably the most straightforward option. Please let me know if you have > any other concerns. > > Thanks,

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-17 Thread Robert Yokota
Hi Colin, I've changed the KIP to have a composite object returned from get(). It's probably the most straightforward option. Please let me know if you have any other concerns. Thanks, Robert On Thu, May 17, 2018 at 11:44 AM, Robert Yokota wrote: > > > Hi Colin, > > My last response was not

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-17 Thread Robert Yokota
Hi Colin, My last response was not that clear, so let me back up and explain a bit more. Some secret managers, such as Vault (and maybe Keywhiz) have the notion of a lease duration or a TTL for a path. Every path can have a different TTL. This is period after which the value of the keys at the

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-17 Thread Colin McCabe
Hi Robert, Hmm. I thought that if you're using ConfigChangeCallback, you are relying on the ConfigProvider to make a callback to you when the configuration has changed. So isn't that always the "push model" (where the ConfigProvider pushes changes to Connect). If you want the "pull model" wh

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-16 Thread Robert Yokota
Hi Colin, > With regard to delayMs, can’t we just restart the > Connector when the keys are actually changed? Currently the VaultConfigProvider does not find out when values for keys have changed. You could do this with a poll model (with a background thread in the ConfigProvider), but since for

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-16 Thread Colin McCabe
Thanks, Robert. With regard to delayMs, can’t we just restart the Connector when the keys are actually changed? Or is the concern that this would lengthen the effective key rotation time? Can’t the user just configure a slightly shorter key rotation time to counteract this concern? Regards, Coli

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-16 Thread Robert Yokota
Hi Colin, Good questions. > As a clarification about the indirections, what if I have the connect configuration key foo set up as ${vault:bar}, and in Vault, have the bar key set to ${file:baz}? > Does connect get foo as the contents of the baz file? I would argue that it should not (and in gen

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-16 Thread Colin McCabe
Thanks, Robert. Looks good overall. As a clarification about the indirections, what if I have the connect configuration key foo set up as ${vault:bar}, and in Vault, have the bar key set to ${file:baz}? Does connect get foo as the contents of the baz file? I would argue that it should not (a

[VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-16 Thread Robert Yokota
Hello everyone, After a good round of discussions with excellent feedback and no major objections, I would like to start a vote on KIP-297 to externalize secrets from Kafka Connect configurations. My thanks in advance! KIP: < https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%3A+Externali