Hi Ron, Unresolved item #1: Yes, Kafka disables hostname verification if "ssl.endpoint.identification.algorithm" is an empty string.
Unresolved item #2: Yes, those 9 plus hostname verification. If we do decide to inherit Kafka configs, wouldn't we inherit these 10 in AclAuthorizer as well? Unresolved item #3: Agree, following up on the discussion thread for KIP-553. On Fri, Jan 17, 2020 at 3:29 AM Ron Dagostino <rndg...@gmail.com> wrote: > Hi Colin. TLS to ZooKeeper will remain an opt-in. The configuration > value that turns it on is zookeeper.ssl.client.enable (which used to > be called zookeeper.client.secure before we renamed everything to be > consistent with Kafka's config naming style and eliminate confusion). > The default value for this is false. Everything else related to TLS > has no effect unless this is explicitly set to true, and this value > does not inherit from anywhere, > > Ron > > On Thu, Jan 16, 2020 at 5:39 PM Colin McCabe <cmcc...@apache.org> wrote: > > > > On Thu, Jan 16, 2020, at 02:58, Rajini Sivaram wrote: > > > Hi Ron/Colin, > > > > > > Thanks for the responses. I think we need to consider these cases: > > > > > > 1) The recommended approach: This is the one we document. Configs > > > consistent with Kafka make sense here. It seems a lot more confusing to > > > configure `ssl.cipher.suites`, `ssl.enabled.protocols`, ` > > > zookeeper.ssl.ciphersuites` and `zookeeper.ssl.enabledProtocols` with > > > multiple naming inconsistencies in server.properties. For users who > have > > > never used ZK system properties, this is clearly simpler. Another > config to > > > consider is `ssl.endpoint.identification.algorithm` which does the same > > > thing as `zookeeper.ssl.hostnameVerification` - if we are staying > > > consistent, we should use the Kafka format for this too. I would also > > > consider using something like `zookeeper.client.ssl.enabled` instead > of ` > > > zookeeper.client.secure` since it is difficult to tell what `secure` > means > > > when you have options to configure Kerberos, ACLs and/or SSL. > > > > > > 2) Inheritance using prefixed configs as we do for other prefixed > configs > > > in Kafka: I think it may be useful for configs like `ssl.protocol` and > ` > > > ssl.cipher.suites`. > > > > > > 3) Default values: Do we want to use Kafka defaults? The inconsistency > we > > > have is `ssl.protocol/ssl.enabled.protocols` since we still have > insecure > > > protocols enabled for Kafka. We have KIP-553 ( > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956 > ) > > > open to disable these. It is currently blocked since it also talks > about > > > enabling TLSv1.3 by default and we haven't done sufficient testing for > > > that. Since it will be good to disable the older insecure protocols > anyway, > > > perhaps we could do that without enabling TLSv1.3 by default in 2.5? > > > > Hi Rajini, > > > > Inheriting the SSL settings from Kafka would have the effect of changing > the defaults for almost everyone using SSL from non-SSL ZK to SSL-based ZK, > right? This seems to expand the scope of the KIP greatly, since it moves > it from being opt-in to opt-out. Are there any potential negative > implications of this? > > > > best, > > Colin > > > > > > > > > > 4) Backward compatibility for deployments which are using system > > > properties. We shouldn't override system properties with Kafka > defaults or > > > inherited values. But we do want to override if user configures > properties > > > explicitly. In the current approach, this was simpler since we just > had to > > > set the configured values. But if we decide to inherit Kafka configs, > then > > > we will need to check system properties and update accordingly. But > that > > > should be doable? > > > > > > On Wed, Jan 15, 2020 at 9:51 PM Colin McCabe <cmcc...@apache.org> > wrote: > > > > > > > On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote: > > > > > Hi Colin. Two things come to mind with respect to ZooKeeper > camelCase > > > > > style vs Kafka-style config names for ZooKeeper. First, I think it > > > > > would be desirable for the client configs and broker configs to be > > > > > interoperable. For example, it feels like it would be convenient > to > > > > > be able to pass the broker's config file to the ZK Security > Migrator > > > > > tool. I think whatever we use (ZooKeeper camelCase style or Kafka > > > > > style), I think we should use the same for both. > > > > > > > > > > The second thing to think about is the fact that ZooKeeper > > > > > configuration inherently uses Java system properties as a first > pass. > > > > > So while we might switch to Kafka-style configs in the config file > > > > > (e.g. zookeeper.ssl.trust.store.location), the > > > > > org.apache.zookeeper.client.ZKClientConfig class (which is how TLS > > > > > configs are set) is going to look for the camelCase > > > > > zookeeper.ssl.trustStore.location system property and use any value > > > > > there. I think this could be very confusing for people. Granted, > we > > > > > are doing this so that people don't have to use system properties, > but > > > > > there is no way to turn off the use of system properties, so I > think > > > > > there would be pretty substantial potential for confusion. > > > > > > > > I don't think we document or recommend that anyone use system > properties > > > > to configure Zookeeper usage within Kafka. And I think the reason > is > > > > exactly the issue you pointed out -- it wouldn't be consistent with > our > > > > other configurations. > > > > > > > > best, > > > > Colin > > > > > > > > > > > > > > One idea is to migrate the system properties -- i.e. look to see if > > > > > zookeeper.ssl.trustStore.location is set and if it is clear out the > > > > > value there and move it under the > zookeeper.ssl.trust.store.location > > > > > system property. (I said it was an idea -- not necessarily a good > one > > > > > :-) > > > > > > > > > > Ron > > > > > > > > > > On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe <cmcc...@apache.org> > wrote: > > > > > > > > > > > > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote: > > > > > > > Thanks Colin and Rajini. > > > > > > > > > > > > > > I've updated the KIP to say that KIP-421 functionality is > available > > > > to > > > > > > > encrypt sensitive configs like the ZK key store and trust store > > > > > > > passwords. (I've also made it clear that the configs are not > > > > > > > dynamically reconfigurable since dynamic values are stored in > ZK and > > > > > > > the data is needed to get to ZK in the first place). Colin, > let me > > > > > > > know if you think anything else needs to be said/done about it > -- I > > > > > > > wasn't sure if your comment above implies that there are > additional > > > > > > > direct actions that we need to take aside from this. > > > > > > > > > > > > > > I agree that making the brokers' ZooKeeper clients dynamic with > > > > > > > respect to key and trust stores (e.g. via > > > > > > > zookeeper.ssl.context.supplier.class) may increase the risk > that this > > > > > > > KIP may not land in time for 2.5. > > > > > > > > > > > > > > Regarding the config names for ZK being different than the > ZooKeeper > > > > > > > configs (e.g. camel-case keyStore/trustStore instead of > > > > > > > keystore/truststore, ciphersuites instead of cipher.suites, > > > > > > > enabledProtocols instead of enabled.protocols), I agree it is > an > > > > > > > issue. I tried to mitigate it by putting warning text in the > config > > > > > > > docs for these. Regarding configuration inheritance, I think > what > > > > you > > > > > > > are saying is that for any configs where the broker has a clear > > > > > > > semantic equivalent, the broker could fall back to the broker > value > > > > if > > > > > > > the ZK value is not given. The potential list is: > > > > > > > > > > > > > > zookeeper.ssl.keyStore.location => ssl.keystore.location > > > > > > > zookeeper.ssl.keyStore.password => ssl.keystore.password > > > > > > > zookeeper.ssl.keyStore.type => ssl.keystore.type > > > > > > > zookeeper.ssl.trustStore.location => ssl.truststore.location > > > > > > > zookeeper.ssl.trustStore.password => ssl.truststore.password > > > > > > > zookeeper.ssl.trustStore.type => ssl.truststore.type > > > > > > > zookeeper.ssl.ciphersuites => ssl.cipher.suites > > > > > > > > > > > > > > Note that there are two configs that look the same based on > their key > > > > > > > names but whose allowable values differ: > > > > > > > > > > > > > > zookeeper.ssl.protocol(Default: TLSv1.2) => > ssl.protocol(Default: > > > > TLS) > > > > > > > zookeeper.ssl.enabledProtocols(Default: value of protocol > property) = > > > > > > > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1) > > > > > > > > > > > > > > Not sure what the right answer is, but hopefully this detail > will > > > > help > > > > > > > get us there. > > > > > > > > > > > > > > Ron > > > > > > > > > > > > I think, on balance, I agree with Rajini that it would be nice > to make > > > > the configs look more like Kafka configs. We don't have camel-case > in > > > > configuration keys, so we should avoid it here. > > > > > > > > > > > > best, > > > > > > Colin > > > > > > > > > > > > > > > > > > > > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe < > cmcc...@apache.org> > > > > wrote: > > > > > > > > > > > > > > > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote: > > > > > > > > > Hi Ron, > > > > > > > > > > > > > > > > > > Thanks for the detailed explanation, sounds good to me. > > > > > > > > > > > > > > > > > > A few more questions: > > > > > > > > > > > > > > > > > > 1) At the moment, all sensitive broker configs including > > > > > > > > > keystore/truststore passwords can be stored encrypted in > > > > ZooKeeper prior to > > > > > > > > > starting up brokers. We are now adding SSL > keystore/truststore > > > > passwords > > > > > > > > > for ZooKeeper client that cannot be stored in ZooKeeper > since > > > > you need > > > > > > > > > these to connect to ZK. We should perhaps document that > these > > > > configs can > > > > > > > > > be encrypted using KIP-421. > > > > > > > > > > > > > > > > That's a good point. Previously, we didn't have ZK > on-the-wire > > > > security support. However, now that we do, sending sensitive > keystore and > > > > truststore passwords to ZK in cleartext should be deprecated in > favor of > > > > using KIP-421. > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We can dynamically update keystores and trust stores > used by > > > > brokers > > > > > > > > > without restarting the broker. Can we support this easily > for ZK > > > > clients > > > > > > > > > created by the broker, for example by adding our own > > > > > > > > > `zookeeper.ssl.context.supplier.class`? > > > > > > > > > > > > > > > > > > > > > > > > > Hmm. That might be better to consider in a follow-up, since > the > > > > deadline for 2.5 KIPs is coming up? > > > > > > > > > > > > > > > > best, > > > > > > > > Colin > > > > > > > > > > > > > > > > > 3) Looks like we are using config names that directly map > to ZK > > > > configs. > > > > > > > > > Have we considered using equivalent Kafka config names with > > > > prefixes, > > > > > > > > > perhaps with inheritance from the non-prefixed names? Not > sure > > > > if this is a > > > > > > > > > good idea, but perhaps worth documenting in Rejected > > > > Alternatives at least. > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe < > cmcc...@apache.org> > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Ron, > > > > > > > > > > > > > > > > > > > > Thanks for the explanation. I guess thinking about it a > > > > little bit more, > > > > > > > > > > we should just add --zk-tls-config-file to all of these > > > > commands. > > > > > > > > > > > > > > > > > > > > We will be removing this option (plus ZK in general) from > > > > these commands > > > > > > > > > > in the next major release, but ZK is still supported in > 2.5, > > > > so we should > > > > > > > > > > just do the logical thing. (The exception is > > > > ZkSecurityMigrator, which > > > > > > > > > > will stay). > > > > > > > > > > > > > > > > > > > > best, > > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote: > > > > > > > > > > > Hi Colin. > > > > > > > > > > > > > > > > > > > > > > <<< It seems like this [--zk-tls-config-file > information] > > > > could just > > > > > > > > > > appear > > > > > > > > > > > in a configuration file, which all of these tools > already > > > > accept (I > > > > > > > > > > think) > > > > > > > > > > > > > > > > > > > > > > ZkSecurityMigrator has no such property file facility; > > > > adding a > > > > > > > > > > > "--zk-tls-config-file" parameter is exactly for this > > > > purpose. If we add > > > > > > > > > > > that to ZkSecurityMigrator then it is trivial to add > it to > > > > other commands > > > > > > > > > > > (the same code is simply reused; it ends up being just > a few > > > > extra > > > > > > > > > > lines). > > > > > > > > > > > I do not see any parameter in the other two commands to > > > > adjust the ZK > > > > > > > > > > > connection; ConfigCommand accepts a "--command-config" > flag, > > > > but > > > > > > > > > > according > > > > > > > > > > > to the code "This is used only with --bootstrap-server > > > > option for > > > > > > > > > > > describing and altering broker configs." > > > > > > > > > > > > > > > > > > > > > > I do agree there would be no need to add > > > > "--zk-tls-config-file" to > > > > > > > > > > > ReassignPartitionsCommand if its direct ZK > connectivity is > > > > replaced in > > > > > > > > > > time > > > > > > > > > > > for the next release. > > > > > > > > > > > > > > > > > > > > > > ConfigCommand supports the "--bootstrap-server" option > and > > > > will have its > > > > > > > > > > > direct ZooKeeper access formally deprecated as per > KIP-555, > > > > but the > > > > > > > > > > special > > > > > > > > > > > use case of bootstrapping a ZooKeeper ensemble with > > > > encrypted credentials > > > > > > > > > > > prior to starting Kafka will still exist, so it feels > like > > > > while > > > > > > > > > > > "--zk-tls-config-file" would never be used except for > this > > > > use case it > > > > > > > > > > > could probably still be added for this particular > situation. > > > > > > > > > > > > > > > > > > > > > > Ron > > > > > > > > > > > > > > > > > > > > > > P.S. I responded on 1/6 but I just discovered that e, > ail > > > > (and 3 more I > > > > > > > > > > > sent) did not go through. I am trying to get emails > through > > > > now to move > > > > > > > > > > > this discussion forward. > > > > > > > > > > > > > > > > > > > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe < > > > > cmcc...@apache.org> wrote: > > > > > > > > > > > > > > > > > > > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote: > > > > > > > > > > > > > Hi everyone. I would like to make the following > changes > > > > to the KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > MOTIVATION: > > > > > > > > > > > > > Include a statement that it will be difficult in > the > > > > short term to > > > > > > > > > > > > > deprecate direct Zookeeper communication in > > > > kafka-configs.{sh, bat} > > > > > > > > > > > > (which > > > > > > > > > > > > > invoke kafka.admin.ConfigCommand) because > bootstrapping > > > > a Kafka > > > > > > > > > > cluster > > > > > > > > > > > > > with encrypted passwords in Zookeeper is an > > > > explicitly-supported use > > > > > > > > > > > > case; > > > > > > > > > > > > > therefore it is in scope to be able to securely > > > > configure the CLI > > > > > > > > > > tools > > > > > > > > > > > > > that still leverage non-deprecated direct Zookeeper > > > > communication > > > > > > > > > > for TLS > > > > > > > > > > > > > (the other 2 tools are > kafka-reassign-partitions.{sh, > > > > bat} and > > > > > > > > > > > > > zookeeper-security-migration.sh). > > > > > > > > > > > > > > > > > > > > > > > > Hi Ron, > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP. > > > > > > > > > > > > > > > > > > > > > > > > About deprecations: > > > > > > > > > > > > > > > > > > > > > > > > * zookeeper-security-migration: clearly, deprecating > ZK > > > > access in this > > > > > > > > > > one > > > > > > > > > > > > would not make sense, since it would defeat the whole > > > > point of the > > > > > > > > > > tool :) > > > > > > > > > > > > > > > > > > > > > > > > * kafka-reassign-partitions: ZK access should be > > > > deprecated here. > > > > > > > > > > KIP-455 > > > > > > > > > > > > implementation has dragged a bit, but this should > get done > > > > soon. > > > > > > > > > > Certainly > > > > > > > > > > > > before the next release. > > > > > > > > > > > > > > > > > > > > > > > > * kafka-configs: I think ZK access should be > deprecated > > > > here as well. > > > > > > > > > > I > > > > > > > > > > > > agree there is a super-special bootstrapping case > here, > > > > but that should > > > > > > > > > > > > have its own tool, not use kafka-configs. > > > > > > > > > > > > > > > > > > > > > > > > I will post a separate KIP for this, though. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > GOALS: > > > > > > > > > > > > > Support the secure configuration of TLS-encrypted > > > > communication > > > > > > > > > > between > > > > > > > > > > > > > Zookeeper and: > > > > > > > > > > > > > a) Kafka brokers > > > > > > > > > > > > > b) The three CLI tools mentioned above that still > > > > support direct, > > > > > > > > > > > > > non-deprecated communication to Zookeeper > > > > > > > > > > > > > It is explicitly out-of-scope to deprecate any > direct > > > > Zookeeper > > > > > > > > > > > > > communication in CLI tools as part of this KIP; > such > > > > work will occur > > > > > > > > > > in > > > > > > > > > > > > > future KIPs instead. > > > > > > > > > > > > > > > > > > > > > > > > > > PUBLIC INTERFACES: > > > > > > > > > > > > > 1) The following new broker configurations will be > > > > recognized. > > > > > > > > > > > > > zookeeper.client.secure (default value = false, > for > > > > backwards > > > > > > > > > > > > > compatibility) > > > > > > > > > > > > > zookeeper.clientCnxnSocket > > > > > > > > > > > > > zookeeper.ssl.keyStore.location > > > > > > > > > > > > > zookeeper.ssl.keyStore.password > > > > > > > > > > > > > zookeeper.ssl.trustStore.location > > > > > > > > > > > > > zookeeper.ssl.trustStore.password > > > > > > > > > > > > > It will be an error for any of the last 5 values > to be > > > > left > > > > > > > > > > unspecified > > > > > > > > > > > > if > > > > > > > > > > > > > zookeeper.client.secure is explicitly set to true. > > > > > > > > > > > > > > > > > > > > > > > > > > 2) In addition, the > > > > kafka.security.authorizer.AclAuthorizer class > > > > > > > > > > > > supports > > > > > > > > > > > > > the ability to connect to a different Zookeeper > instance > > > > than the > > > > > > > > > > one the > > > > > > > > > > > > > brokers use. We therefore also add the following > > > > optional configs, > > > > > > > > > > which > > > > > > > > > > > > > override the corresponding ones from above when > present: > > > > > > > > > > > > > authorizer.zookeeper.client.secure > > > > > > > > > > > > > authorizer.zookeeper.clientCnxnSocket > > > > > > > > > > > > > authorizer.zookeeper.ssl.keyStore.location > > > > > > > > > > > > > authorizer.zookeeper.ssl.keyStore.password > > > > > > > > > > > > > authorizer.zookeeper.ssl.trustStore.location > > > > > > > > > > > > > authorizer.zookeeper.ssl.trustStore.password > > > > > > > > > > > > > > > > > > > > > > > > > > 3) The three CLI tools mentioned above will > support a new > > > > > > > > > > > > --zk-tls-config-file > > > > > > > > > > > > > <String: Zookeeper TLS configuration file path>" > > > > option. The > > > > > > > > > > following > > > > > > > > > > > > > properties will be recognized in that file, and > > > > unrecognized > > > > > > > > > > properties > > > > > > > > > > > > > will be ignored to allow the possibility of > pointing > > > > > > > > > > zk-tls-config-file > > > > > > > > > > > > at > > > > > > > > > > > > > the broker's config file. > > > > > > > > > > > > > zookeeper.client.secure (default value = false) > > > > > > > > > > > > > zookeeper.clientCnxnSocket > > > > > > > > > > > > > zookeeper.ssl.keyStore.location > > > > > > > > > > > > > zookeeper.ssl.keyStore.password > > > > > > > > > > > > > zookeeper.ssl.trustStore.location > > > > > > > > > > > > > zookeeper.ssl.trustStore.password > > > > > > > > > > > > > It will be an error for any of the last 5 values > to be > > > > left > > > > > > > > > > unspecified > > > > > > > > > > > > if > > > > > > > > > > > > > zookeeper.client.secure is explicitly set to true. > > > > > > > > > > > > > > > > > > > > > > > > Do we really need a --zk-tls-config-file flag? It > seems > > > > like this > > > > > > > > > > could > > > > > > > > > > > > just appear in a configuration file, which all of > these > > > > tools already > > > > > > > > > > > > accept (I think). > > > > > > > > > > > > > > > > > > > > > > > > best, > > > > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ron > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino < > > > > rndg...@gmail.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone. Let's get this discussion going > again > > > > now that Kafka > > > > > > > > > > 2.4 > > > > > > > > > > > > > > with Zookeeper 3.5.6 is out. > > > > > > > > > > > > > > > > > > > > > > > > > > > > First, regarding the KIP number, the other KIP > that > > > > was using this > > > > > > > > > > > > number > > > > > > > > > > > > > > moved to KIP 534, so KIP 515 remains the correct > > > > number for this > > > > > > > > > > > > > > discussion. I've updated the Kafka Improvement > > > > Proposal page to > > > > > > > > > > list > > > > > > > > > > > > this > > > > > > > > > > > > > > KIP in the 515 slot, so we're all set there. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the actual issues under discussion, I > think > > > > there are > > > > > > > > > > some > > > > > > > > > > > > > > things we should clarify. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1) It is possible to use TLS connectivity to > Zookeeper > > > > from Apache > > > > > > > > > > > > Kafka > > > > > > > > > > > > > > 2.4 -- the problem is that configuration > information > > > > has to be > > > > > > > > > > passed > > > > > > > > > > > > via > > > > > > > > > > > > > > system properties as "-D" command line options > on the > > > > java > > > > > > > > > > invocation > > > > > > > > > > > > of > > > > > > > > > > > > > > the broker, and those are not secure (anyone with > > > > access to the box > > > > > > > > > > > > can see > > > > > > > > > > > > > > the command line used to invoke the broker); the > > > > configuration > > > > > > > > > > includes > > > > > > > > > > > > > > sensitive information (e.g. a keystore > password), so > > > > we need a > > > > > > > > > > secure > > > > > > > > > > > > > > mechanism for passing the configuration values. > I > > > > believe the real > > > > > > > > > > > > > > motivation for this KIP is to harden the > configuration > > > > mechanism > > > > > > > > > > for > > > > > > > > > > > > > > Zookeeper TLS connectivity. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2) I believe the list of CLI tools that continue > to > > > > use direct > > > > > > > > > > > > Zookeeper > > > > > > > > > > > > > > connectivity in a non-deprecated fashion is: > > > > > > > > > > > > > > a) > > > > > > > > > > > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator > > > > > > > > > > > > > > b) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand > > > > > > > > > > > > > > c) kafka-configs.{sh, > bat}/kafka.admin.ConfigCommand > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a > > > > conundrum > > > > > > > > > > because it > > > > > > > > > > > > > > explicitly states in a comment that a supported > use > > > > case is > > > > > > > > > > > > bootstrapping a > > > > > > > > > > > > > > Kafka cluster with encrypted passwords in > Zookeeper > > > > (see > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65 > > > > > > > > > > > > ). > > > > > > > > > > > > > > This means it will be especially difficult to > > > > deprecate this > > > > > > > > > > particular > > > > > > > > > > > > > > direct Zookeeper connectivity without a different > > > > storage > > > > > > > > > > mechanism for > > > > > > > > > > > > > > dynamic configuration values being available > (i.e. the > > > > self-managed > > > > > > > > > > > > quorum > > > > > > > > > > > > > > referred to in KIP-500). > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it would be easier and simpler to harden > the > > > > Zookeeper TLS > > > > > > > > > > > > > > configuration in both Kafka and in CLI tools > compared > > > > to trying to > > > > > > > > > > > > > > deprecate the direct Zookeeper connectivity in > the > > > > above 3 CLI > > > > > > > > > > tools -- > > > > > > > > > > > > > > especially when ConfigCommand has no obvious > > > > short-term path to > > > > > > > > > > > > deprecation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding how to harden the TLS configuration, > adding > > > > the > > > > > > > > > > appropriate > > > > > > > > > > > > > > configs to Kafka is an obvious choice for the > > > > brokers. Not that > > > > > > > > > > these > > > > > > > > > > > > > > values cannot be dynamically reconfigurable > because > > > > the values > > > > > > > > > > would be > > > > > > > > > > > > > > required to connect to Zookeeper via TLS in the > first > > > > place. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding how to harden the TLS configuration > for the > > > > 3 remaining > > > > > > > > > > CLI > > > > > > > > > > > > > > tools, it would be relatively straightforward to > add > > > > support for a > > > > > > > > > > > > > > "--zk-tls-config-file <String: Zookeeper TLS > > > > configuration system > > > > > > > > > > > > > > properties file path>" option to each one. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ron > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes > < > > > > > > > > > > pere.ur...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> True, the number is duplicated. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> do you know how can we get the problem fix? I > was > > > > not aware, I > > > > > > > > > > > > missed, > > > > > > > > > > > > > >> sorry the need to add the KIP to the table. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> -- Pere > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Missatge de Jorge Esteban Quilcate Otoya < > > > > > > > > > > quilcate.jo...@gmail.com> > > > > > > > > > > > > del > > > > > > > > > > > > > >> dia > > > > > > > > > > > > > >> dt., 3 de set. 2019 a les 13:43: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Hi Pere, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > Have you add your KIP to the list here > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > > > > > > > >> ? > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > I found the KIP number assigned to another. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón > Bayes < > > > > > > > > > > > > pere.ur...@gmail.com> > > > > > > > > > > > > > >> > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> >> Thanks for your time Harsha, > > > > > > > > > > > > > >> >> anyone else with comments? looking > forward to > > > > hearing from > > > > > > > > > > you. > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> Stupid question: when do you move from > discussion > > > > to vote? > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> Missatge de Harsha Chintalapani < > ka...@harsha.io> > > > > del dia > > > > > > > > > > dv., 30 > > > > > > > > > > > > > >> d’ag. > > > > > > > > > > > > > >> >> 2019 a les 21:59: > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > Thanks Pere. KIP looks good to me. > > > > > > > > > > > > > >> >> > -Harsha > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere > Urbón > > > > Bayes < > > > > > > > > > > > > > >> >> pere.ur...@gmail.com> > > > > > > > > > > > > > >> >> > wrote: > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> >> Not really, > > > > > > > > > > > > > >> >> >> my idea is to keep the JAAS parameter, > so > > > > people don't see > > > > > > > > > > > > major > > > > > > > > > > > > > >> >> >> changes. But if you pass a properties > file, > > > > then this takes > > > > > > > > > > > > > >> precedence > > > > > > > > > > > > > >> >> over > > > > > > > > > > > > > >> >> >> the other, with the idea that you can do > sasl > > > > as well with > > > > > > > > > > the > > > > > > > > > > > > > >> >> properties > > > > > > > > > > > > > >> >> >> files. > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> >> Makes sense? > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> >> -- Pere > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> >> Missatge de Harsha Chintalapani < > > > > ka...@harsha.io> del dia > > > > > > > > > > dv., > > > > > > > > > > > > 30 > > > > > > > > > > > > > >> >> d’ag. > > > > > > > > > > > > > >> >> >> 2019 a les 19:00: > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> >>> Hi Pere, > > > > > > > > > > > > > >> >> >>> Thanks for the KIP. Enabling > SSL for > > > > zookeeper > > > > > > > > > > for > > > > > > > > > > > > Kafka > > > > > > > > > > > > > >> >> makes > > > > > > > > > > > > > >> >> >>> sense. > > > > > > > > > > > > > >> >> >>> "The changes are planned to be > introduced in a > > > > compatible > > > > > > > > > > way, > > > > > > > > > > > > by > > > > > > > > > > > > > >> >> >>> keeping the current JAAS variable > precedence." > > > > > > > > > > > > > >> >> >>> Can you elaborate a bit here. If the user > > > > configures a JAAS > > > > > > > > > > > > file > > > > > > > > > > > > > >> with > > > > > > > > > > > > > >> >> >>> Client section it will take precedence > over > > > > zookeeper SSL > > > > > > > > > > > > configs? > > > > > > > > > > > > > >> >> >>> > > > > > > > > > > > > > >> >> >>> Thanks, > > > > > > > > > > > > > >> >> >>> Harsha > > > > > > > > > > > > > >> >> >>> > > > > > > > > > > > > > >> >> >>> > > > > > > > > > > > > > >> >> >>> > > > > > > > > > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere > Urbón > > > > Bayes < > > > > > > > > > > > > > >> >> pere.ur...@gmail.com> > > > > > > > > > > > > > >> >> >>> wrote: > > > > > > > > > > > > > >> >> >>> > > > > > > > > > > > > > >> >> >>>> Hi, > > > > > > > > > > > > > >> >> >>>> quick question, I saw in another mail > that > > > > 2.4 release is > > > > > > > > > > > > planned > > > > > > > > > > > > > >> for > > > > > > > > > > > > > >> >> >>>> September. I think it would be really > awesome > > > > to have > > > > > > > > > > this for > > > > > > > > > > > > > >> this > > > > > > > > > > > > > >> >> >>>> release, do you think we can make it? > > > > > > > > > > > > > >> >> >>>> > > > > > > > > > > > > > >> >> >>>> -- Pere > > > > > > > > > > > > > >> >> >>>> > > > > > > > > > > > > > >> >> >>>> Missatge de Pere Urbón Bayes < > > > > pere.ur...@gmail.com> del > > > > > > > > > > dia > > > > > > > > > > > > dj., > > > > > > > > > > > > > >> 29 > > > > > > > > > > > > > >> >> >>>> d’ag. 2019 a les 20:10: > > > > > > > > > > > > > >> >> >>>> > > > > > > > > > > > > > >> >> >>>> Hi, > > > > > > > > > > > > > >> >> >>>> this is my first KIP for a change in > Apache > > > > Kafka, so I'm > > > > > > > > > > > > really > > > > > > > > > > > > > >> need > > > > > > > > > > > > > >> >> >>>> to the process. Looking forward to > hearing > > > > from you and > > > > > > > > > > learn > > > > > > > > > > > > the > > > > > > > > > > > > > >> >> best > > > > > > > > > > > > > >> >> >>>> ropes here. > > > > > > > > > > > > > >> >> >>>> > > > > > > > > > > > > > >> >> >>>> I would like to propose this KIP-515 to > > > > enable the > > > > > > > > > > > > > >> ZookeeperClients > > > > > > > > > > > > > >> >> to > > > > > > > > > > > > > >> >> >>>> take full advantage of the TLS > communication > > > > in the new > > > > > > > > > > > > Zookeeper > > > > > > > > > > > > > >> >> 3.5.5. > > > > > > > > > > > > > >> >> >>>> Specially interesting it the Zookeeper > > > > Security Migration, > > > > > > > > > > > > that > > > > > > > > > > > > > >> >> without > > > > > > > > > > > > > >> >> >>>> this change will not work with TLS, > disabling > > > > users to use > > > > > > > > > > > > ACLs > > > > > > > > > > > > > >> when > > > > > > > > > > > > > >> >> the > > > > > > > > > > > > > >> >> >>>> Zookeeper cluster use TLS. > > > > > > > > > > > > > >> >> >>>> > > > > > > > > > > > > > >> >> >>>> link: > > > > > > > > > > > > > >> >> >>>> > > > > > > > > > > > > > >> >> >>>> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > > > > > > > > > > > > >> >> >>>> > > > > > > > > > > > > > >> >> >>>> Looking forward to hearing from you on > this, > > > > > > > > > > > > > >> >> >>>> > > > > > > > > > > > > > >> >> >>>> /cheers > > > > > > > > > > > > > >> >> >>>> > > > > > > > > > > > > > >> >> >>>> -- > > > > > > > > > > > > > >> >> >>>> Pere Urbon-Bayes > > > > > > > > > > > > > >> >> >>>> Software Architect > > > > > > > > > > > > > >> >> >>>> http://www.purbon.com > > > > > > > > > > > > > >> >> >>>> https://twitter.com/purbon > > > > > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/ > > > > > > > > > > > > > >> >> >>>> > > > > > > > > > > > > > >> >> >>>> -- > > > > > > > > > > > > > >> >> >>>> Pere Urbon-Bayes > > > > > > > > > > > > > >> >> >>>> Software Architect > > > > > > > > > > > > > >> >> >>>> http://www.purbon.com > > > > > > > > > > > > > >> >> >>>> https://twitter.com/purbon > > > > > > > > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/ > > > > > > > > > > > > > >> >> >>>> > > > > > > > > > > > > > >> >> >>> > > > > > > > > > > > > > >> >> >>> > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> >> -- > > > > > > > > > > > > > >> >> >> Pere Urbon-Bayes > > > > > > > > > > > > > >> >> >> Software Architect > > > > > > > > > > > > > >> >> >> http://www.purbon.com > > > > > > > > > > > > > >> >> >> https://twitter.com/purbon > > > > > > > > > > > > > >> >> >> https://www.linkedin.com/in/purbon/ > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> -- > > > > > > > > > > > > > >> >> Pere Urbon-Bayes > > > > > > > > > > > > > >> >> Software Architect > > > > > > > > > > > > > >> >> http://www.purbon.com > > > > > > > > > > > > > >> >> https://twitter.com/purbon > > > > > > > > > > > > > >> >> https://www.linkedin.com/in/purbon/ > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> -- > > > > > > > > > > > > > >> Pere Urbon-Bayes > > > > > > > > > > > > > >> Software Architect > > > > > > > > > > > > > >> http://www.purbon.com > > > > > > > > > > > > > >> https://twitter.com/purbon > > > > > > > > > > > > > >> https://www.linkedin.com/in/purbon/ > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
