Hi Ron, For Kafka, we moved from hostname verification disabled-by-default to enabled-by-default under https://cwiki.apache.org/confluence/display/KAFKA/KIP-294+-+Enable+TLS+hostname+verification+by+default. So we have tested empty String for disabling hostname verification and we know that it works, not only in test environments, but also in actual deployments. Basically, "ssl.endpoint.identification.algorithm=" in the file results in a config value of empty String and not setting it results in the default value of HTTPS since the value is null.I would probably do the same for the ZK config. If you are creating certificates for ZooKeeper without hostnames, then you may be doing the same for Kafka too and in both cases, the config would be an empty string. This is insecure anyway and we would really expect this only in test environments. Not sure if it is worth making an exception for this in terms of consistent config names.
On Fri, Jan 17, 2020 at 2:37 PM Ron Dagostino <rndg...@gmail.com> wrote: > Hi again, Rajini. I've updated the KIP, and while doing it I became > concerned about using zookeeper.ssl.endpoint.identification.algorithm > for enabling/disabling hostname verification. The KIP reflects what > we decided, but upon reading it, I wonder if it is workable. Here's > what it says for this config: > > "Specifies whether to enable hostname verification in the ZooKeeper > TLS negotiation process, with (case-insensitively) "https" meaning > ZooKeeper hostname verification is enabled and an explicit blank value > meaning it is disabled (disabling it is only recommended for testing > purposes). Overrides any explicit "true" or "false" value set via the > <code>zookeeper.ssl.hostnameVerification</code> system property (true > implying https and false implying blank) and inherits the value of > <code>ssl.endpoint.identification.algorithm</code> (if any) if no > explicit value or non-value is set both here and via the system > property." > > I wonder if someone sets an explicit empty value > (zookeeper.ssl.endpoint.identification.algorithm=) in the config file, > will we be able to distinguish between that and a config file that > doesn't have it at all? And even if we can, this feels pretty forced > to me, like we are bending over backwards for consistency when > "zookeeper.ssl.hostname.verification.enable" is a natural fit. I > doubt this config is going to be used very much, so it really does > feel to me like we should just go with the natural fit and ditch the > compatibility and inheritance for this one config. > > Ron > > On Fri, Jan 17, 2020 at 8:07 AM Ron Dagostino <rndg...@gmail.com> wrote: > > > > Hi Rajini. I’ll submit a documentation PR to clarify the Kafka behavior > of #1 and will adopt the same config for ZK. I agree now we should inherit > for AclAuthorizer too — I was just stuck on the “no inheritance” idea more > than I realized, and while the ZK quorums are different that doesn’t > necessarily mean the same client identity wouldn’t be valid on both (and if > not then the configs can still be overridden). > > > > I’ll update the KIP today and let this version soak for a day or two, > and then I’ll start a vote if no other issues/comments arise. > > > > Ron > > > > > On Jan 17, 2020, at 6:08 AM, Rajini Sivaram <rajinisiva...@gmail.com> > wrote: > > > > > > Hi Ron, > > > > > > Unresolved item #1: Yes, Kafka disables hostname verification if > > > "ssl.endpoint.identification.algorithm" is an empty string. > > > > > > Unresolved item #2: Yes, those 9 plus hostname verification. If we do > > > decide to inherit Kafka configs, wouldn't we inherit these 10 in > > > AclAuthorizer as well? > > > > > > Unresolved item #3: Agree, following up on the discussion thread for > > > KIP-553. > > > > > > > > >> On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino <rndg...@gmail.com> > wrote: > > >> > > >> Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration > > >> value that turns it on is zookeeper.ssl.client.enable (which used to > > >> be called zookeeper.client.secure before we renamed everything to be > > >> consistent with Kafka's config naming style and eliminate confusion). > > >> The default value for this is false. Everything else related to TLS > > >> has no effect unless this is explicitly set to true, and this value > > >> does not inherit from anywhere, > > >> > > >> Ron > > >> > > >>> On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe <cmcc...@apache.org> > wrote: > > >>> > > >>>> On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote: > > >>>> Hi Ron/Colin, > > >>>> > > >>>> Thanks for the responses. I think we need to consider these cases: > > >>>> > > >>>> 1) The recommended approach: This is the one we document. Configs > > >>>> consistent with Kafka make sense here. It seems a lot more > confusing to > > >>>> configure `ssl.cipher.suites`, `ssl.enabled.protocols`, ` > > >>>> zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` > with > > >>>> multiple naming inconsistencies in server.properties. For users who > > >> have > > >>>> never used ZK system properties, this is clearly simpler. Another > > >> config to > > >>>> consider is `ssl.endpoint.identification.algorithm` which does the > same > > >>>> thing as `zookeeper.ssl.hostnameVerification` - if we are staying > > >>>> consistent, we should use the Kafka format for this too. I would > also > > >>>> consider using something like `zookeeper.client.ssl.enabled` instead > > >> of ` > > >>>> zookeeper.client.secure` since it is difficult to tell what `secure` > > >> means > > >>>> when you have options to configure Kerberos, ACLs and/or SSL. > > >>>> > > >>>> 2) Inheritance using prefixed configs as we do for other prefixed > > >> configs > > >>>> in Kafka: I think it may be useful for configs like `ssl.protocol` > and > > >> ` > > >>>> ssl.cipher.suites`. > > >>>> > > >>>> 3) Default values: Do we want to use Kafka defaults? The > inconsistency > > >> we > > >>>> have is `ssl.protocol/ssl.enabled.protocols` since we still have > > >> insecure > > >>>> protocols enabled for Kafka. We have KIP-553 ( > > >>>> > > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956 > > >> ) > > >>>> open to disable these. It is currently blocked since it also talks > > >> about > > >>>> enabling TLSv1.3 by default and we haven't done sufficient testing > for > > >>>> that. Since it will be good to disable the older insecure protocols > > >> anyway, > > >>>> perhaps we could do that without enabling TLSv1.3 by default in 2.5? > > >>> > > >>> Hi Rajini, > > >>> > > >>> Inheriting the SSL settings from Kafka would have the effect of > changing > > >> the defaults for almost everyone using SSL from non-SSL ZK to > SSL-based ZK, > > >> right? This seems to expand the scope of the KIP greatly, since it > moves > > >> it from being opt-in to opt-out. Are there any potential negative > > >> implications of this? > > >>> > > >>> best, > > >>> Colin > > >>> > > >>> > > >>>> > > >>>> 4) Backward compatibility for deployments which are using system > > >>>> properties. We shouldn't override system properties with Kafka > > >> defaults or > > >>>> inherited values. But we do want to override if user configures > > >> properties > > >>>> explicitly. In the current approach, this was simpler since we just > > >> had to > > >>>> set the configured values. But if we decide to inherit Kafka > configs, > > >> then > > >>>> we will need to check system properties and update accordingly. But > > >> that > > >>>> should be doable? > > >>>> > > >>>> On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cmcc...@apache.org> > > >> wrote: > > >>>> > > >>>>>> On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote: > > >>>>>> Hi Colin. Two things come to mind with respect to ZooKeeper > > >> camelCase > > >>>>>> style vs Kafka-style config names for ZooKeeper. First, I think > it > > >>>>>> would be desirable for the client configs and broker configs to be > > >>>>>> interoperable. For example, it feels like it would be convenient > > >> to > > >>>>>> be able to pass the broker's config file to the ZK Security > > >> Migrator > > >>>>>> tool. I think whatever we use (ZooKeeper camelCase style or Kafka > > >>>>>> style), I think we should use the same for both. > > >>>>>> > > >>>>>> The second thing to think about is the fact that ZooKeeper > > >>>>>> configuration inherently uses Java system properties as a first > > >> pass. > > >>>>>> So while we might switch to Kafka-style configs in the config file > > >>>>>> (e.g. zookeeper.ssl.trust.store.location), the > > >>>>>> org.apache.zookeeper.client.ZKClientConfig class (which is how TLS > > >>>>>> configs are set) is going to look for the camelCase > > >>>>>> zookeeper.ssl.trustStore.location system property and use any > value > > >>>>>> there. I think this could be very confusing for people. Granted, > > >> we > > >>>>>> are doing this so that people don't have to use system properties, > > >> but > > >>>>>> there is no way to turn off the use of system properties, so I > > >> think > > >>>>>> there would be pretty substantial potential for confusion. > > >>>>> > > >>>>> I don't think we document or recommend that anyone use system > > >> properties > > >>>>> to configure Zookeeper usage within Kafka. And I think the reason > > >> is > > >>>>> exactly the issue you pointed out -- it wouldn't be consistent with > > >> our > > >>>>> other configurations. > > >>>>> > > >>>>> best, > > >>>>> Colin > > >>>>> > > >>>>>> > > >>>>>> One idea is to migrate the system properties -- i.e. look to see > if > > >>>>>> zookeeper.ssl.trustStore.location is set and if it is clear out > the > > >>>>>> value there and move it under the > > >> zookeeper.ssl.trust.store.location > > >>>>>> system property. (I said it was an idea -- not necessarily a good > > >> one > > >>>>>> :-) > > >>>>>> > > >>>>>> Ron > > >>>>>> > > >>>>>> On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cmcc...@apache.org> > > >> wrote: > > >>>>>>> > > >>>>>>>> On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote: > > >>>>>>>> Thanks Colin and Rajini. > > >>>>>>>> > > >>>>>>>> I've updated the KIP to say that KIP-421 functionality is > > >> available > > >>>>> to > > >>>>>>>> encrypt sensitive configs like the ZK key store and trust store > > >>>>>>>> passwords. (I've also made it clear that the configs are not > > >>>>>>>> dynamically reconfigurable since dynamic values are stored in > > >> ZK and > > >>>>>>>> the data is needed to get to ZK in the first place). Colin, > > >> let me > > >>>>>>>> know if you think anything else needs to be said/done about it > > >> -- I > > >>>>>>>> wasn't sure if your comment above implies that there are > > >> additional > > >>>>>>>> direct actions that we need to take aside from this. > > >>>>>>>> > > >>>>>>>> I agree that making the brokers' ZooKeeper clients dynamic with > > >>>>>>>> respect to key and trust stores (e.g. via > > >>>>>>>> zookeeper.ssl.context.supplier.class) may increase the risk > > >> that this > > >>>>>>>> KIP may not land in time for 2.5. > > >>>>>>>> > > >>>>>>>> Regarding the config names for ZK being different than the > > >> ZooKeeper > > >>>>>>>> configs (e.g. camel-case keyStore/trustStore instead of > > >>>>>>>> keystore/truststore, ciphersuites instead of cipher.suites, > > >>>>>>>> enabledProtocols instead of enabled.protocols), I agree it is > > >> an > > >>>>>>>> issue. I tried to mitigate it by putting warning text in the > > >> config > > >>>>>>>> docs for these. Regarding configuration inheritance, I think > > >> what > > >>>>> you > > >>>>>>>> are saying is that for any configs where the broker has a clear > > >>>>>>>> semantic equivalent, the broker could fall back to the broker > > >> value > > >>>>> if > > >>>>>>>> the ZK value is not given. The potential list is: > > >>>>>>>> > > >>>>>>>> zookeeper.ssl.keyStore.location => ssl.keystore.location > > >>>>>>>> zookeeper.ssl.keyStore.password => ssl.keystore.password > > >>>>>>>> zookeeper.ssl.keyStore.type => ssl.keystore.type > > >>>>>>>> zookeeper.ssl.trustStore.location => ssl.truststore.location > > >>>>>>>> zookeeper.ssl.trustStore.password => ssl.truststore.password > > >>>>>>>> zookeeper.ssl.trustStore.type => ssl.truststore.type > > >>>>>>>> zookeeper.ssl.ciphersuites => ssl.cipher.suites > > >>>>>>>> > > >>>>>>>> Note that there are two configs that look the same based on > > >> their key > > >>>>>>>> names but whose allowable values differ: > > >>>>>>>> > > >>>>>>>> zookeeper.ssl.protocol(Default: TLSv1.2) => > > >> ssl.protocol(Default: > > >>>>> TLS) > > >>>>>>>> zookeeper.ssl.enabledProtocols(Default: value of protocol > > >> property) = > > >>>>>>>> ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1) > > >>>>>>>> > > >>>>>>>> Not sure what the right answer is, but hopefully this detail > > >> will > > >>>>> help > > >>>>>>>> get us there. > > >>>>>>>> > > >>>>>>>> Ron > > >>>>>>> > > >>>>>>> I think, on balance, I agree with Rajini that it would be nice > > >> to make > > >>>>> the configs look more like Kafka configs. We don't have camel-case > > >> in > > >>>>> configuration keys, so we should avoid it here. > > >>>>>>> > > >>>>>>> best, > > >>>>>>> Colin > > >>>>>>> > > >>>>>>>> > > >>>>>>>> On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe < > > >> cmcc...@apache.org> > > >>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote: > > >>>>>>>>>> Hi Ron, > > >>>>>>>>>> > > >>>>>>>>>> Thanks for the detailed explanation, sounds good to me. > > >>>>>>>>>> > > >>>>>>>>>> A few more questions: > > >>>>>>>>>> > > >>>>>>>>>> 1) At the moment, all sensitive broker configs including > > >>>>>>>>>> keystore/truststore passwords can be stored encrypted in > > >>>>> ZooKeeper prior to > > >>>>>>>>>> starting up brokers. We are now adding SSL > > >> keystore/truststore > > >>>>> passwords > > >>>>>>>>>> for ZooKeeper client that cannot be stored in ZooKeeper > > >> since > > >>>>> you need > > >>>>>>>>>> these to connect to ZK. We should perhaps document that > > >> these > > >>>>> configs can > > >>>>>>>>>> be encrypted using KIP-421. > > >>>>>>>>> > > >>>>>>>>> That's a good point. Previously, we didn't have ZK > > >> on-the-wire > > >>>>> security support. However, now that we do, sending sensitive > > >> keystore and > > >>>>> truststore passwords to ZK in cleartext should be deprecated in > > >> favor of > > >>>>> using KIP-421. > > >>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> 2) We can dynamically update keystores and trust stores > > >> used by > > >>>>> brokers > > >>>>>>>>>> without restarting the broker. Can we support this easily > > >> for ZK > > >>>>> clients > > >>>>>>>>>> created by the broker, for example by adding our own > > >>>>>>>>>> `zookeeper.ssl.context.supplier.class`? > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> Hmm. That might be better to consider in a follow-up, since > > >> the > > >>>>> deadline for 2.5 KIPs is coming up? > > >>>>>>>>> > > >>>>>>>>> best, > > >>>>>>>>> Colin > > >>>>>>>>> > > >>>>>>>>>> 3) Looks like we are using config names that directly map > > >> to ZK > > >>>>> configs. > > >>>>>>>>>> Have we considered using equivalent Kafka config names with > > >>>>> prefixes, > > >>>>>>>>>> perhaps with inheritance from the non-prefixed names? Not > > >> sure > > >>>>> if this is a > > >>>>>>>>>> good idea, but perhaps worth documenting in Rejected > > >>>>> Alternatives at least. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe < > > >> cmcc...@apache.org> > > >>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> Hi Ron, > > >>>>>>>>>>> > > >>>>>>>>>>> Thanks for the explanation. I guess thinking about it a > > >>>>> little bit more, > > >>>>>>>>>>> we should just add --zk-tls-config-file to all of these > > >>>>> commands. > > >>>>>>>>>>> > > >>>>>>>>>>> We will be removing this option (plus ZK in general) from > > >>>>> these commands > > >>>>>>>>>>> in the next major release, but ZK is still supported in > > >> 2.5, > > >>>>> so we should > > >>>>>>>>>>> just do the logical thing. (The exception is > > >>>>> ZkSecurityMigrator, which > > >>>>>>>>>>> will stay). > > >>>>>>>>>>> > > >>>>>>>>>>> best, > > >>>>>>>>>>> Colin > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>>> On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote: > > >>>>>>>>>>>> Hi Colin. > > >>>>>>>>>>>> > > >>>>>>>>>>>> <<< It seems like this [--zk-tls-config-file > > >> information] > > >>>>> could just > > >>>>>>>>>>> appear > > >>>>>>>>>>>> in a configuration file, which all of these tools > > >> already > > >>>>> accept (I > > >>>>>>>>>>> think) > > >>>>>>>>>>>> > > >>>>>>>>>>>> ZkSecurityMigrator has no such property file facility; > > >>>>> adding a > > >>>>>>>>>>>> "--zk-tls-config-file" parameter is exactly for this > > >>>>> purpose. If we add > > >>>>>>>>>>>> that to ZkSecurityMigrator then it is trivial to add > > >> it to > > >>>>> other commands > > >>>>>>>>>>>> (the same code is simply reused; it ends up being just > > >> a few > > >>>>> extra > > >>>>>>>>>>> lines). > > >>>>>>>>>>>> I do not see any parameter in the other two commands to > > >>>>> adjust the ZK > > >>>>>>>>>>>> connection; ConfigCommand accepts a "--command-config" > > >> flag, > > >>>>> but > > >>>>>>>>>>> according > > >>>>>>>>>>>> to the code "This is used only with --bootstrap-server > > >>>>> option for > > >>>>>>>>>>>> describing and altering broker configs." > > >>>>>>>>>>>> > > >>>>>>>>>>>> I do agree there would be no need to add > > >>>>> "--zk-tls-config-file" to > > >>>>>>>>>>>> ReassignPartitionsCommand if its direct ZK > > >> connectivity is > > >>>>> replaced in > > >>>>>>>>>>> time > > >>>>>>>>>>>> for the next release. > > >>>>>>>>>>>> > > >>>>>>>>>>>> ConfigCommand supports the "--bootstrap-server" option > > >> and > > >>>>> will have its > > >>>>>>>>>>>> direct ZooKeeper access formally deprecated as per > > >> KIP-555, > > >>>>> but the > > >>>>>>>>>>> special > > >>>>>>>>>>>> use case of bootstrapping a ZooKeeper ensemble with > > >>>>> encrypted credentials > > >>>>>>>>>>>> prior to starting Kafka will still exist, so it feels > > >> like > > >>>>> while > > >>>>>>>>>>>> "--zk-tls-config-file" would never be used except for > > >> this > > >>>>> use case it > > >>>>>>>>>>>> could probably still be added for this particular > > >> situation. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Ron > > >>>>>>>>>>>> > > >>>>>>>>>>>> P.S. I responded on 1/6 but I just discovered that e, > > >> ail > > >>>>> (and 3 more I > > >>>>>>>>>>>> sent) did not go through. I am trying to get emails > > >> through > > >>>>> now to move > > >>>>>>>>>>>> this discussion forward. > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe < > > >>>>> cmcc...@apache.org> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>>>> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote: > > >>>>>>>>>>>>>> Hi everyone. I would like to make the following > > >> changes > > >>>>> to the KIP. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> MOTIVATION: > > >>>>>>>>>>>>>> Include a statement that it will be difficult in > > >> the > > >>>>> short term to > > >>>>>>>>>>>>>> deprecate direct Zookeeper communication in > > >>>>> kafka-configs.{sh, bat} > > >>>>>>>>>>>>> (which > > >>>>>>>>>>>>>> invoke kafka.admin.ConfigCommand) because > > >> bootstrapping > > >>>>> a Kafka > > >>>>>>>>>>> cluster > > >>>>>>>>>>>>>> with encrypted passwords in Zookeeper is an > > >>>>> explicitly-supported use > > >>>>>>>>>>>>> case; > > >>>>>>>>>>>>>> therefore it is in scope to be able to securely > > >>>>> configure the CLI > > >>>>>>>>>>> tools > > >>>>>>>>>>>>>> that still leverage non-deprecated direct Zookeeper > > >>>>> communication > > >>>>>>>>>>> for TLS > > >>>>>>>>>>>>>> (the other 2 tools are > > >> kafka-reassign-partitions.{sh, > > >>>>> bat} and > > >>>>>>>>>>>>>> zookeeper-security-migration.sh). > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Hi Ron, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Thanks for the KIP. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> About deprecations: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> * zookeeper-security-migration: clearly, deprecating > > >> ZK > > >>>>> access in this > > >>>>>>>>>>> one > > >>>>>>>>>>>>> would not make sense, since it would defeat the whole > > >>>>> point of the > > >>>>>>>>>>> tool :) > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> * kafka-reassign-partitions: ZK access should be > > >>>>> deprecated here. > > >>>>>>>>>>> KIP-455 > > >>>>>>>>>>>>> implementation has dragged a bit, but this should > > >> get done > > >>>>> soon. > > >>>>>>>>>>> Certainly > > >>>>>>>>>>>>> before the next release. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> * kafka-configs: I think ZK access should be > > >> deprecated > > >>>>> here as well. > > >>>>>>>>>>> I > > >>>>>>>>>>>>> agree there is a super-special bootstrapping case > > >> here, > > >>>>> but that should > > >>>>>>>>>>>>> have its own tool, not use kafka-configs. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> I will post a separate KIP for this, though. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> GOALS: > > >>>>>>>>>>>>>> Support the secure configuration of TLS-encrypted > > >>>>> communication > > >>>>>>>>>>> between > > >>>>>>>>>>>>>> Zookeeper and: > > >>>>>>>>>>>>>> a) Kafka brokers > > >>>>>>>>>>>>>> b) The three CLI tools mentioned above that still > > >>>>> support direct, > > >>>>>>>>>>>>>> non-deprecated communication to Zookeeper > > >>>>>>>>>>>>>> It is explicitly out-of-scope to deprecate any > > >> direct > > >>>>> Zookeeper > > >>>>>>>>>>>>>> communication in CLI tools as part of this KIP; > > >> such > > >>>>> work will occur > > >>>>>>>>>>> in > > >>>>>>>>>>>>>> future KIPs instead. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> PUBLIC INTERFACES: > > >>>>>>>>>>>>>> 1) The following new broker configurations will be > > >>>>> recognized. > > >>>>>>>>>>>>>> zookeeper.client.secure (default value = false, > > >> for > > >>>>> backwards > > >>>>>>>>>>>>>> compatibility) > > >>>>>>>>>>>>>> zookeeper.clientCnxnSocket > > >>>>>>>>>>>>>> zookeeper.ssl.keyStore.location > > >>>>>>>>>>>>>> zookeeper.ssl.keyStore.password > > >>>>>>>>>>>>>> zookeeper.ssl.trustStore.location > > >>>>>>>>>>>>>> zookeeper.ssl.trustStore.password > > >>>>>>>>>>>>>> It will be an error for any of the last 5 values > > >> to be > > >>>>> left > > >>>>>>>>>>> unspecified > > >>>>>>>>>>>>> if > > >>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> 2) In addition, the > > >>>>> kafka.security.authorizer.AclAuthorizer class > > >>>>>>>>>>>>> supports > > >>>>>>>>>>>>>> the ability to connect to a different Zookeeper > > >> instance > > >>>>> than the > > >>>>>>>>>>> one the > > >>>>>>>>>>>>>> brokers use. We therefore also add the following > > >>>>> optional configs, > > >>>>>>>>>>> which > > >>>>>>>>>>>>>> override the corresponding ones from above when > > >> present: > > >>>>>>>>>>>>>> authorizer.zookeeper.client.secure > > >>>>>>>>>>>>>> authorizer.zookeeper.clientCnxnSocket > > >>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.location > > >>>>>>>>>>>>>> authorizer.zookeeper.ssl.keyStore.password > > >>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.location > > >>>>>>>>>>>>>> authorizer.zookeeper.ssl.trustStore.password > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> 3) The three CLI tools mentioned above will > > >> support a new > > >>>>>>>>>>>>> --zk-tls-config-file > > >>>>>>>>>>>>>> <String: Zookeeper TLS configuration file path>" > > >>>>> option. The > > >>>>>>>>>>> following > > >>>>>>>>>>>>>> properties will be recognized in that file, and > > >>>>> unrecognized > > >>>>>>>>>>> properties > > >>>>>>>>>>>>>> will be ignored to allow the possibility of > > >> pointing > > >>>>>>>>>>> zk-tls-config-file > > >>>>>>>>>>>>> at > > >>>>>>>>>>>>>> the broker's config file. > > >>>>>>>>>>>>>> zookeeper.client.secure (default value = false) > > >>>>>>>>>>>>>> zookeeper.clientCnxnSocket > > >>>>>>>>>>>>>> zookeeper.ssl.keyStore.location > > >>>>>>>>>>>>>> zookeeper.ssl.keyStore.password > > >>>>>>>>>>>>>> zookeeper.ssl.trustStore.location > > >>>>>>>>>>>>>> zookeeper.ssl.trustStore.password > > >>>>>>>>>>>>>> It will be an error for any of the last 5 values > > >> to be > > >>>>> left > > >>>>>>>>>>> unspecified > > >>>>>>>>>>>>> if > > >>>>>>>>>>>>>> zookeeper.client.secure is explicitly set to true. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Do we really need a --zk-tls-config-file flag? It > > >> seems > > >>>>> like this > > >>>>>>>>>>> could > > >>>>>>>>>>>>> just appear in a configuration file, which all of > > >> these > > >>>>> tools already > > >>>>>>>>>>>>> accept (I think). > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> best, > > >>>>>>>>>>>>> Colin > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Ron > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino < > > >>>>> rndg...@gmail.com> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Hi everyone. Let's get this discussion going > > >> again > > >>>>> now that Kafka > > >>>>>>>>>>> 2.4 > > >>>>>>>>>>>>>>> with Zookeeper 3.5.6 is out. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> First, regarding the KIP number, the other KIP > > >> that > > >>>>> was using this > > >>>>>>>>>>>>> number > > >>>>>>>>>>>>>>> moved to KIP 534, so KIP 515 remains the correct > > >>>>> number for this > > >>>>>>>>>>>>>>> discussion. I've updated the Kafka Improvement > > >>>>> Proposal page to > > >>>>>>>>>>> list > > >>>>>>>>>>>>> this > > >>>>>>>>>>>>>>> KIP in the 515 slot, so we're all set there. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Regarding the actual issues under discussion, I > > >> think > > >>>>> there are > > >>>>>>>>>>> some > > >>>>>>>>>>>>>>> things we should clarify. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> 1) It is possible to use TLS connectivity to > > >> Zookeeper > > >>>>> from Apache > > >>>>>>>>>>>>> Kafka > > >>>>>>>>>>>>>>> 2.4 -- the problem is that configuration > > >> information > > >>>>> has to be > > >>>>>>>>>>> passed > > >>>>>>>>>>>>> via > > >>>>>>>>>>>>>>> system properties as "-D" command line options > > >> on the > > >>>>> java > > >>>>>>>>>>> invocation > > >>>>>>>>>>>>> of > > >>>>>>>>>>>>>>> the broker, and those are not secure (anyone with > > >>>>> access to the box > > >>>>>>>>>>>>> can see > > >>>>>>>>>>>>>>> the command line used to invoke the broker); the > > >>>>> configuration > > >>>>>>>>>>> includes > > >>>>>>>>>>>>>>> sensitive information (e.g. a keystore > > >> password), so > > >>>>> we need a > > >>>>>>>>>>> secure > > >>>>>>>>>>>>>>> mechanism for passing the configuration values. > > >> I > > >>>>> believe the real > > >>>>>>>>>>>>>>> motivation for this KIP is to harden the > > >> configuration > > >>>>> mechanism > > >>>>>>>>>>> for > > >>>>>>>>>>>>>>> Zookeeper TLS connectivity. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> 2) I believe the list of CLI tools that continue > > >> to > > >>>>> use direct > > >>>>>>>>>>>>> Zookeeper > > >>>>>>>>>>>>>>> connectivity in a non-deprecated fashion is: > > >>>>>>>>>>>>>>> a) > > >>>>>>>>>>> > > >> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator > > >>>>>>>>>>>>>>> b) > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>> > > >> > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand > > >>>>>>>>>>>>>>> c) kafka-configs.{sh, > > >> bat}/kafka.admin.ConfigCommand > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> 3) I believe Kafka.admin.ConfigCommand presents a > > >>>>> conundrum > > >>>>>>>>>>> because it > > >>>>>>>>>>>>>>> explicitly states in a comment that a supported > > >> use > > >>>>> case is > > >>>>>>>>>>>>> bootstrapping a > > >>>>>>>>>>>>>>> Kafka cluster with encrypted passwords in > > >> Zookeeper > > >>>>> (see > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>> > > >> > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65 > > >>>>>>>>>>>>> ). > > >>>>>>>>>>>>>>> This means it will be especially difficult to > > >>>>> deprecate this > > >>>>>>>>>>> particular > > >>>>>>>>>>>>>>> direct Zookeeper connectivity without a different > > >>>>> storage > > >>>>>>>>>>> mechanism for > > >>>>>>>>>>>>>>> dynamic configuration values being available > > >> (i.e. the > > >>>>> self-managed > > >>>>>>>>>>>>> quorum > > >>>>>>>>>>>>>>> referred to in KIP-500). > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> I think it would be easier and simpler to harden > > >> the > > >>>>> Zookeeper TLS > > >>>>>>>>>>>>>>> configuration in both Kafka and in CLI tools > > >> compared > > >>>>> to trying to > > >>>>>>>>>>>>>>> deprecate the direct Zookeeper connectivity in > > >> the > > >>>>> above 3 CLI > > >>>>>>>>>>> tools -- > > >>>>>>>>>>>>>>> especially when ConfigCommand has no obvious > > >>>>> short-term path to > > >>>>>>>>>>>>> deprecation. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Regarding how to harden the TLS configuration, > > >> adding > > >>>>> the > > >>>>>>>>>>> appropriate > > >>>>>>>>>>>>>>> configs to Kafka is an obvious choice for the > > >>>>> brokers. Not that > > >>>>>>>>>>> these > > >>>>>>>>>>>>>>> values cannot be dynamically reconfigurable > > >> because > > >>>>> the values > > >>>>>>>>>>> would be > > >>>>>>>>>>>>>>> required to connect to Zookeeper via TLS in the > > >> first > > >>>>> place. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Regarding how to harden the TLS configuration > > >> for the > > >>>>> 3 remaining > > >>>>>>>>>>> CLI > > >>>>>>>>>>>>>>> tools, it would be relatively straightforward to > > >> add > > >>>>> support for a > > >>>>>>>>>>>>>>> "--zk-tls-config-file <String: Zookeeper TLS > > >>>>> configuration system > > >>>>>>>>>>>>>>> properties file path>" option to each one. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Thoughts? > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Ron > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes > > >> < > > >>>>>>>>>>> pere.ur...@gmail.com > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> True, the number is duplicated. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> do you know how can we get the problem fix? I > > >> was > > >>>>> not aware, I > > >>>>>>>>>>>>> missed, > > >>>>>>>>>>>>>>>> sorry the need to add the KIP to the table. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> -- Pere > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Missatge de Jorge Esteban Quilcate Otoya < > > >>>>>>>>>>> quilcate.jo...@gmail.com> > > >>>>>>>>>>>>> del > > >>>>>>>>>>>>>>>> dia > > >>>>>>>>>>>>>>>> dt., 3 de set. 2019 a les 13:43: > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Hi Pere, > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Have you add your KIP to the list here > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > >>>>>>>>>>>>>>>> ? > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> I found the KIP number assigned to another. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón > > >> Bayes < > > >>>>>>>>>>>>> pere.ur...@gmail.com> > > >>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Thanks for your time Harsha, > > >>>>>>>>>>>>>>>>>> anyone else with comments? looking > > >> forward to > > >>>>> hearing from > > >>>>>>>>>>> you. > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Stupid question: when do you move from > > >> discussion > > >>>>> to vote? > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani < > > >> ka...@harsha.io> > > >>>>> del dia > > >>>>>>>>>>> dv., 30 > > >>>>>>>>>>>>>>>> d’ag. > > >>>>>>>>>>>>>>>>>> 2019 a les 21:59: > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> Thanks Pere. KIP looks good to me. > > >>>>>>>>>>>>>>>>>>> -Harsha > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:05 AM, Pere > > >> Urbón > > >>>>> Bayes < > > >>>>>>>>>>>>>>>>>> pere.ur...@gmail.com> > > >>>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Not really, > > >>>>>>>>>>>>>>>>>>>> my idea is to keep the JAAS parameter, > > >> so > > >>>>> people don't see > > >>>>>>>>>>>>> major > > >>>>>>>>>>>>>>>>>>>> changes. But if you pass a properties > > >> file, > > >>>>> then this takes > > >>>>>>>>>>>>>>>> precedence > > >>>>>>>>>>>>>>>>>> over > > >>>>>>>>>>>>>>>>>>>> the other, with the idea that you can do > > >> sasl > > >>>>> as well with > > >>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>> properties > > >>>>>>>>>>>>>>>>>>>> files. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Makes sense? > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> -- Pere > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Missatge de Harsha Chintalapani < > > >>>>> ka...@harsha.io> del dia > > >>>>>>>>>>> dv., > > >>>>>>>>>>>>> 30 > > >>>>>>>>>>>>>>>>>> d’ag. > > >>>>>>>>>>>>>>>>>>>> 2019 a les 19:00: > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Hi Pere, > > >>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. Enabling > > >> SSL for > > >>>>> zookeeper > > >>>>>>>>>>> for > > >>>>>>>>>>>>> Kafka > > >>>>>>>>>>>>>>>>>> makes > > >>>>>>>>>>>>>>>>>>>>> sense. > > >>>>>>>>>>>>>>>>>>>>> "The changes are planned to be > > >> introduced in a > > >>>>> compatible > > >>>>>>>>>>> way, > > >>>>>>>>>>>>> by > > >>>>>>>>>>>>>>>>>>>>> keeping the current JAAS variable > > >> precedence." > > >>>>>>>>>>>>>>>>>>>>> Can you elaborate a bit here. If the user > > >>>>> configures a JAAS > > >>>>>>>>>>>>> file > > >>>>>>>>>>>>>>>> with > > >>>>>>>>>>>>>>>>>>>>> Client section it will take precedence > > >> over > > >>>>> zookeeper SSL > > >>>>>>>>>>>>> configs? > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>>>>>>>>>> Harsha > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> On Fri, Aug 30, 2019 at 7:50 AM, Pere > > >> Urbón > > >>>>> Bayes < > > >>>>>>>>>>>>>>>>>> pere.ur...@gmail.com> > > >>>>>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> Hi, > > >>>>>>>>>>>>>>>>>>>>>> quick question, I saw in another mail > > >> that > > >>>>> 2.4 release is > > >>>>>>>>>>>>> planned > > >>>>>>>>>>>>>>>> for > > >>>>>>>>>>>>>>>>>>>>>> September. I think it would be really > > >> awesome > > >>>>> to have > > >>>>>>>>>>> this for > > >>>>>>>>>>>>>>>> this > > >>>>>>>>>>>>>>>>>>>>>> release, do you think we can make it? > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> -- Pere > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> Missatge de Pere Urbón Bayes < > > >>>>> pere.ur...@gmail.com> del > > >>>>>>>>>>> dia > > >>>>>>>>>>>>> dj., > > >>>>>>>>>>>>>>>> 29 > > >>>>>>>>>>>>>>>>>>>>>> d’ag. 2019 a les 20:10: > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> Hi, > > >>>>>>>>>>>>>>>>>>>>>> this is my first KIP for a change in > > >> Apache > > >>>>> Kafka, so I'm > > >>>>>>>>>>>>> really > > >>>>>>>>>>>>>>>> need > > >>>>>>>>>>>>>>>>>>>>>> to the process. Looking forward to > > >> hearing > > >>>>> from you and > > >>>>>>>>>>> learn > > >>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>> best > > >>>>>>>>>>>>>>>>>>>>>> ropes here. > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> I would like to propose this KIP-515 to > > >>>>> enable the > > >>>>>>>>>>>>>>>> ZookeeperClients > > >>>>>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>>>>>> take full advantage of the TLS > > >> communication > > >>>>> in the new > > >>>>>>>>>>>>> Zookeeper > > >>>>>>>>>>>>>>>>>> 3.5.5. > > >>>>>>>>>>>>>>>>>>>>>> Specially interesting it the Zookeeper > > >>>>> Security Migration, > > >>>>>>>>>>>>> that > > >>>>>>>>>>>>>>>>>> without > > >>>>>>>>>>>>>>>>>>>>>> this change will not work with TLS, > > >> disabling > > >>>>> users to use > > >>>>>>>>>>>>> ACLs > > >>>>>>>>>>>>>>>> when > > >>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>>>> Zookeeper cluster use TLS. > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> link: > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> Looking forward to hearing from you on > > >> this, > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> /cheers > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes > > >>>>>>>>>>>>>>>>>>>>>> Software Architect > > >>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com > > >>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon > > >>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/ > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes > > >>>>>>>>>>>>>>>>>>>>>> Software Architect > > >>>>>>>>>>>>>>>>>>>>>> http://www.purbon.com > > >>>>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon > > >>>>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/ > > >>>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>>>>>> Pere Urbon-Bayes > > >>>>>>>>>>>>>>>>>>>> Software Architect > > >>>>>>>>>>>>>>>>>>>> http://www.purbon.com > > >>>>>>>>>>>>>>>>>>>> https://twitter.com/purbon > > >>>>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/ > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>>>> Pere Urbon-Bayes > > >>>>>>>>>>>>>>>>>> Software Architect > > >>>>>>>>>>>>>>>>>> http://www.purbon.com > > >>>>>>>>>>>>>>>>>> https://twitter.com/purbon > > >>>>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/ > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>> Pere Urbon-Bayes > > >>>>>>>>>>>>>>>> Software Architect > > >>>>>>>>>>>>>>>> http://www.purbon.com > > >>>>>>>>>>>>>>>> https://twitter.com/purbon > > >>>>>>>>>>>>>>>> https://www.linkedin.com/in/purbon/ > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >> > > >> > >
