Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

On Mon, Apr 16, 2018, at 08:15, Ron Dagostino wrote: > Hi Rajini. I think a good, illustrative OAuth example is the situation > where a Kafka client (whether non-broker or broker, where the latter occurs > when OAUTHBEARER is the inter-broker SASL mechanism) needs to authenticate > to the token en

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Rajini. I think a good, illustrative OAuth example is the situation where a Kafka client (whether non-broker or broker, where the latter occurs when OAUTHBEARER is the inter-broker SASL mechanism) needs to authenticate to the token endpoint to retrieve an access token. There are actually 2 pot

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Ron, Thanks for the analysis, this is very useful. Reducing the feature to the minimum required for the scenarios helps (though I was hoping that the redact flag would one day help with improving SASL diagnostics, that can be for another day). In the past when users requested different passwor

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

I am unsure if substitution should be supported for just JAAS configs or if we should allow it for cluster/broker/consumer configs. What I think would be helpful would be to boil down this proposal to its most essential requirement, and I think the discussion has helped us arrive at what that look

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

We can limit substitution mechanism only for password config types and JAAS config. We may not want to use to for all config properties. On Sat, Apr 14, 2018 at 9:21 AM, Colin McCabe wrote: > On Fri, Apr 13, 2018, at 10:30, Rajini Sivaram wrote: > > Hi Colin, > > > > JAAS configuration can be pr

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

On Fri, Apr 13, 2018, at 10:30, Rajini Sivaram wrote: > Hi Colin, > > JAAS configuration can be provided in a separate file, but that has been > the cause of various problems in itself. The configuration option > `sasl.jaas.config` was added to overcome this. This is already a dynamic > configurat

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Thanks, Colin, for your comments. I appreciate the point that substitution can be over-applied. I did not have a specific requirement for the defaultKey= and fromValueOfKey modifiers; I included them because they intuitively felt like they would be useful and made the feature more powerful. Give

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Colin, JAAS configuration can be provided in a separate file, but that has been the cause of various problems in itself. The configuration option `sasl.jaas.config` was added to overcome this. This is already a dynamic configuration option stored in ZooKeeper since we allow listeners to be adde

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

I think we need to be a very careful here. Configuration complexity can get out of control very quickly. There are also some conflicting goals here. As much as possible, we want the configuration to be a full description of what the broker is going to do. If the configuration pulls in environ

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Ron, I think we should be able to process substitutions for both static JAAS configuration file as well as `sasl.jaas.config` property. We load the configuration using org.apache.kafka.common.security. JaasContext.loadXXXContext() and that would be a good place to do any substitution. The metho

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Rajini. Regarding processing the sasl.jaas.config value up-front, there are a couple of things that occur to me about it. First, the older way of storing the JAAS config in a separate file is still supported (and is at this time the prevalent mechanism on the broker side since sasl.jaas.config

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Ron, Thanks for the notes and KIP update. Handling `sasl.jaas.config` as a special case is fine, but it would be better if we can do any substitutions before we create a `Configuration` object rather than expect the login module to do the substitution. That way, we will have a consistent subst

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi folks. I updated KIP 269 to help clarify some of the issues mentioned previously. In particular, I added a new single-method UnderlyingValues interface to make it clear how data is to be provided to SubstitutableValues, and I added information about if/how the underlying values might be re-rea

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi folks. Here is a summary of where I think we stand on this KIP and what I believe it means for how we move forward. - There is some desire to use substitution more broadly beyond just JAAS module options. Specifically, cluster/producer/consumer config values such as ssl.keystore.pas

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Rajini. I've also been thinking about how sasl.jaas.config will be parsed. Something that is implicit in the current proposal needs to be made explicit if this is to be applied more broadly, and that is the fact that there is a namespacing of sorts going on. For example, in the current propos

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Ron, Thanks for the responses. For broader use as configs, opt-in makes sense to avoid any surprises and a global opt-in ought to be fine. If we do want to use this for all configs, a few things to consider: - How will sasl.jaas.config property will get parsed? This is essentially a co

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi folks. I think there are a couple of issues that were just raised in this thread. One is whether the ability to use PasswordCallback exists, and if so whether that impacts the applicability of this KIP to the SASL/OAUTHBEARER KIP-255. The second issue is related to how we might leverage this

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Yes, I was going to suggest that we should do this for all configs earlier, but was reluctant to do that since in its current form, there is a property enableSubstitution (in JAAS config at the moment) that indicates if substitution is to be performed. If enabled, all values in that config are cons

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi, Substitution mechanism can be useful to configure regular password configs liken ssl.keystore.password, ssl.truststore.password, etc. This is can be good alternative to previously proposed KIP-76 and will give more options to the user. https://cwiki.apache.org/confluence/display/KAFKA/KIP- 76

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Ron, For the password example, you could define a login CallbackHandler that processes PasswordCallback to provide passwords. We don't currently do this with PLAIN/SCRAM because login callback handlers were not configurable earlier and we haven't updated the login modules to do this. But that c

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Rajini. Thanks for the questions. I could see someone wanting to retrieve a password from a vended password vault solution (for example); that is the kind of scenario that the ability to add new substitutable types would be meant for. I do still consider this KIP 269 to be a prerequisite for

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Ron, Now that login callback handlers are configurable, is this KIP still a pre-req for OAuth? I was wondering whether we still need the ability to add new substitutable types or whether it would be sufficient to add the built-in ones to read from file etc. On Thu, Mar 29, 2018 at 6:48 AM, Ro

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi everyone. There have been no comments on this KIP, so I intend to put it to a vote next week if there are no comments that might entail changes between now and then. Please take a look in the meantime if you wish. Ron On Thu, Mar 15, 2018 at 2:36 PM, Ron Dagostino wrote: > Hi everyone. > >

[DISCUSS] KIP-269: Substitution Within Configuration Values

Hi everyone. I created KIP-269: Substitution Within Configuration Values (https://cwiki.apache.org/confluence/display/KAFKA/KIP+ 269+Substitution+Within+Configuration+Values