On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote: > Hi Ron, > > Thanks for the detailed explanation, sounds good to me. > > A few more questions: > > 1) At the moment, all sensitive broker configs including > keystore/truststore passwords can be stored encrypted in ZooKeeper prior to > starting up brokers. We are now adding SSL keystore/truststore passwords > for ZooKeeper client that cannot be stored in ZooKeeper since you need > these to connect to ZK. We should perhaps document that these configs can > be encrypted using KIP-421.
That's a good point. Previously, we didn't have ZK on-the-wire security support. However, now that we do, sending sensitive keystore and truststore passwords to ZK in cleartext should be deprecated in favor of using KIP-421. > > 2) We can dynamically update keystores and trust stores used by brokers > without restarting the broker. Can we support this easily for ZK clients > created by the broker, for example by adding our own > `zookeeper.ssl.context.supplier.class`? > Hmm. That might be better to consider in a follow-up, since the deadline for 2.5 KIPs is coming up? best, Colin > 3) Looks like we are using config names that directly map to ZK configs. > Have we considered using equivalent Kafka config names with prefixes, > perhaps with inheritance from the non-prefixed names? Not sure if this is a > good idea, but perhaps worth documenting in Rejected Alternatives at least. > > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cmcc...@apache.org> wrote: > > > Hi Ron, > > > > Thanks for the explanation. I guess thinking about it a little bit more, > > we should just add --zk-tls-config-file to all of these commands. > > > > We will be removing this option (plus ZK in general) from these commands > > in the next major release, but ZK is still supported in 2.5, so we should > > just do the logical thing. (The exception is ZkSecurityMigrator, which > > will stay). > > > > best, > > Colin > > > > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote: > > > Hi Colin. > > > > > > <<< It seems like this [--zk-tls-config-file information] could just > > appear > > > in a configuration file, which all of these tools already accept (I > > think) > > > > > > ZkSecurityMigrator has no such property file facility; adding a > > > "--zk-tls-config-file" parameter is exactly for this purpose. If we add > > > that to ZkSecurityMigrator then it is trivial to add it to other commands > > > (the same code is simply reused; it ends up being just a few extra > > lines). > > > I do not see any parameter in the other two commands to adjust the ZK > > > connection; ConfigCommand accepts a "--command-config" flag, but > > according > > > to the code "This is used only with --bootstrap-server option for > > > describing and altering broker configs." > > > > > > I do agree there would be no need to add "--zk-tls-config-file" to > > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in > > time > > > for the next release. > > > > > > ConfigCommand supports the "--bootstrap-server" option and will have its > > > direct ZooKeeper access formally deprecated as per KIP-555, but the > > special > > > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials > > > prior to starting Kafka will still exist, so it feels like while > > > "--zk-tls-config-file" would never be used except for this use case it > > > could probably still be added for this particular situation. > > > > > > Ron > > > > > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I > > > sent) did not go through. I am trying to get emails through now to move > > > this discussion forward. > > > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cmcc...@apache.org> wrote: > > > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote: > > > > > Hi everyone. I would like to make the following changes to the KIP. > > > > > > > > > > MOTIVATION: > > > > > Include a statement that it will be difficult in the short term to > > > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat} > > > > (which > > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka > > cluster > > > > > with encrypted passwords in Zookeeper is an explicitly-supported use > > > > case; > > > > > therefore it is in scope to be able to securely configure the CLI > > tools > > > > > that still leverage non-deprecated direct Zookeeper communication > > for TLS > > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and > > > > > zookeeper-security-migration.sh). > > > > > > > > Hi Ron, > > > > > > > > Thanks for the KIP. > > > > > > > > About deprecations: > > > > > > > > * zookeeper-security-migration: clearly, deprecating ZK access in this > > one > > > > would not make sense, since it would defeat the whole point of the > > tool :) > > > > > > > > * kafka-reassign-partitions: ZK access should be deprecated here. > > KIP-455 > > > > implementation has dragged a bit, but this should get done soon. > > Certainly > > > > before the next release. > > > > > > > > * kafka-configs: I think ZK access should be deprecated here as well. > > I > > > > agree there is a super-special bootstrapping case here, but that should > > > > have its own tool, not use kafka-configs. > > > > > > > > I will post a separate KIP for this, though. > > > > > > > > > > > > > > GOALS: > > > > > Support the secure configuration of TLS-encrypted communication > > between > > > > > Zookeeper and: > > > > > a) Kafka brokers > > > > > b) The three CLI tools mentioned above that still support direct, > > > > > non-deprecated communication to Zookeeper > > > > > It is explicitly out-of-scope to deprecate any direct Zookeeper > > > > > communication in CLI tools as part of this KIP; such work will occur > > in > > > > > future KIPs instead. > > > > > > > > > > PUBLIC INTERFACES: > > > > > 1) The following new broker configurations will be recognized. > > > > > zookeeper.client.secure (default value = false, for backwards > > > > > compatibility) > > > > > zookeeper.clientCnxnSocket > > > > > zookeeper.ssl.keyStore.location > > > > > zookeeper.ssl.keyStore.password > > > > > zookeeper.ssl.trustStore.location > > > > > zookeeper.ssl.trustStore.password > > > > > It will be an error for any of the last 5 values to be left > > unspecified > > > > if > > > > > zookeeper.client.secure is explicitly set to true. > > > > > > > > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class > > > > supports > > > > > the ability to connect to a different Zookeeper instance than the > > one the > > > > > brokers use. We therefore also add the following optional configs, > > which > > > > > override the corresponding ones from above when present: > > > > > authorizer.zookeeper.client.secure > > > > > authorizer.zookeeper.clientCnxnSocket > > > > > authorizer.zookeeper.ssl.keyStore.location > > > > > authorizer.zookeeper.ssl.keyStore.password > > > > > authorizer.zookeeper.ssl.trustStore.location > > > > > authorizer.zookeeper.ssl.trustStore.password > > > > > > > > > > 3) The three CLI tools mentioned above will support a new > > > > --zk-tls-config-file > > > > > <String: Zookeeper TLS configuration file path>" option. The > > following > > > > > properties will be recognized in that file, and unrecognized > > properties > > > > > will be ignored to allow the possibility of pointing > > zk-tls-config-file > > > > at > > > > > the broker's config file. > > > > > zookeeper.client.secure (default value = false) > > > > > zookeeper.clientCnxnSocket > > > > > zookeeper.ssl.keyStore.location > > > > > zookeeper.ssl.keyStore.password > > > > > zookeeper.ssl.trustStore.location > > > > > zookeeper.ssl.trustStore.password > > > > > It will be an error for any of the last 5 values to be left > > unspecified > > > > if > > > > > zookeeper.client.secure is explicitly set to true. > > > > > > > > Do we really need a --zk-tls-config-file flag? It seems like this > > could > > > > just appear in a configuration file, which all of these tools already > > > > accept (I think). > > > > > > > > best, > > > > Colin > > > > > > > > > > > > > > > > > > Ron > > > > > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rndg...@gmail.com> > > wrote: > > > > > > > > > > > Hi everyone. Let's get this discussion going again now that Kafka > > 2.4 > > > > > > with Zookeeper 3.5.6 is out. > > > > > > > > > > > > First, regarding the KIP number, the other KIP that was using this > > > > number > > > > > > moved to KIP 534, so KIP 515 remains the correct number for this > > > > > > discussion. I've updated the Kafka Improvement Proposal page to > > list > > > > this > > > > > > KIP in the 515 slot, so we're all set there. > > > > > > > > > > > > Regarding the actual issues under discussion, I think there are > > some > > > > > > things we should clarify. > > > > > > > > > > > > 1) It is possible to use TLS connectivity to Zookeeper from Apache > > > > Kafka > > > > > > 2.4 -- the problem is that configuration information has to be > > passed > > > > via > > > > > > system properties as "-D" command line options on the java > > invocation > > > > of > > > > > > the broker, and those are not secure (anyone with access to the box > > > > can see > > > > > > the command line used to invoke the broker); the configuration > > includes > > > > > > sensitive information (e.g. a keystore password), so we need a > > secure > > > > > > mechanism for passing the configuration values. I believe the real > > > > > > motivation for this KIP is to harden the configuration mechanism > > for > > > > > > Zookeeper TLS connectivity. > > > > > > > > > > > > 2) I believe the list of CLI tools that continue to use direct > > > > Zookeeper > > > > > > connectivity in a non-deprecated fashion is: > > > > > > a) > > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator > > > > > > b) > > > > > > > > > > > > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand > > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand > > > > > > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum > > because it > > > > > > explicitly states in a comment that a supported use case is > > > > bootstrapping a > > > > > > Kafka cluster with encrypted passwords in Zookeeper (see > > > > > > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65 > > > > ). > > > > > > This means it will be especially difficult to deprecate this > > particular > > > > > > direct Zookeeper connectivity without a different storage > > mechanism for > > > > > > dynamic configuration values being available (i.e. the self-managed > > > > quorum > > > > > > referred to in KIP-500). > > > > > > > > > > > > I think it would be easier and simpler to harden the Zookeeper TLS > > > > > > configuration in both Kafka and in CLI tools compared to trying to > > > > > > deprecate the direct Zookeeper connectivity in the above 3 CLI > > tools -- > > > > > > especially when ConfigCommand has no obvious short-term path to > > > > deprecation. > > > > > > > > > > > > Regarding how to harden the TLS configuration, adding the > > appropriate > > > > > > configs to Kafka is an obvious choice for the brokers. Not that > > these > > > > > > values cannot be dynamically reconfigurable because the values > > would be > > > > > > required to connect to Zookeeper via TLS in the first place. > > > > > > > > > > > > Regarding how to harden the TLS configuration for the 3 remaining > > CLI > > > > > > tools, it would be relatively straightforward to add support for a > > > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system > > > > > > properties file path>" option to each one. > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > Ron > > > > > > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes < > > pere.ur...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > >> True, the number is duplicated. > > > > > >> > > > > > >> do you know how can we get the problem fix? I was not aware, I > > > > missed, > > > > > >> sorry the need to add the KIP to the table. > > > > > >> > > > > > >> -- Pere > > > > > >> > > > > > >> Missatge de Jorge Esteban Quilcate Otoya < > > quilcate.jo...@gmail.com> > > > > del > > > > > >> dia > > > > > >> dt., 3 de set. 2019 a les 13:43: > > > > > >> > > > > > >> > Hi Pere, > > > > > >> > > > > > > >> > Have you add your KIP to the list here > > > > > >> > > > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > >> ? > > > > > >> > > > > > > >> > I found the KIP number assigned to another. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes < > > > > pere.ur...@gmail.com> > > > > > >> > wrote: > > > > > >> > > > > > > >> >> Thanks for your time Harsha, > > > > > >> >> anyone else with comments? looking forward to hearing from > > you. > > > > > >> >> > > > > > >> >> Stupid question: when do you move from discussion to vote? > > > > > >> >> > > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia > > dv., 30 > > > > > >> d’ag. > > > > > >> >> 2019 a les 21:59: > > > > > >> >> > > > > > >> >> > Thanks Pere. KIP looks good to me. > > > > > >> >> > -Harsha > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes < > > > > > >> >> pere.ur...@gmail.com> > > > > > >> >> > wrote: > > > > > >> >> > > > > > > >> >> >> Not really, > > > > > >> >> >> my idea is to keep the JAAS parameter, so people don't see > > > > major > > > > > >> >> >> changes. But if you pass a properties file, then this takes > > > > > >> precedence > > > > > >> >> over > > > > > >> >> >> the other, with the idea that you can do sasl as well with > > the > > > > > >> >> properties > > > > > >> >> >> files. > > > > > >> >> >> > > > > > >> >> >> Makes sense? > > > > > >> >> >> > > > > > >> >> >> -- Pere > > > > > >> >> >> > > > > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia > > dv., > > > > 30 > > > > > >> >> d’ag. > > > > > >> >> >> 2019 a les 19:00: > > > > > >> >> >> > > > > > >> >> >>> Hi Pere, > > > > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper > > for > > > > Kafka > > > > > >> >> makes > > > > > >> >> >>> sense. > > > > > >> >> >>> "The changes are planned to be introduced in a compatible > > way, > > > > by > > > > > >> >> >>> keeping the current JAAS variable precedence." > > > > > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS > > > > file > > > > > >> with > > > > > >> >> >>> Client section it will take precedence over zookeeper SSL > > > > configs? > > > > > >> >> >>> > > > > > >> >> >>> Thanks, > > > > > >> >> >>> Harsha > > > > > >> >> >>> > > > > > >> >> >>> > > > > > >> >> >>> > > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes < > > > > > >> >> pere.ur...@gmail.com> > > > > > >> >> >>> wrote: > > > > > >> >> >>> > > > > > >> >> >>>> Hi, > > > > > >> >> >>>> quick question, I saw in another mail that 2.4 release is > > > > planned > > > > > >> for > > > > > >> >> >>>> September. I think it would be really awesome to have > > this for > > > > > >> this > > > > > >> >> >>>> release, do you think we can make it? > > > > > >> >> >>>> > > > > > >> >> >>>> -- Pere > > > > > >> >> >>>> > > > > > >> >> >>>> Missatge de Pere Urbón Bayes <pere.ur...@gmail.com> del > > dia > > > > dj., > > > > > >> 29 > > > > > >> >> >>>> d’ag. 2019 a les 20:10: > > > > > >> >> >>>> > > > > > >> >> >>>> Hi, > > > > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm > > > > really > > > > > >> need > > > > > >> >> >>>> to the process. Looking forward to hearing from you and > > learn > > > > the > > > > > >> >> best > > > > > >> >> >>>> ropes here. > > > > > >> >> >>>> > > > > > >> >> >>>> I would like to propose this KIP-515 to enable the > > > > > >> ZookeeperClients > > > > > >> >> to > > > > > >> >> >>>> take full advantage of the TLS communication in the new > > > > Zookeeper > > > > > >> >> 3.5.5. > > > > > >> >> >>>> Specially interesting it the Zookeeper Security Migration, > > > > that > > > > > >> >> without > > > > > >> >> >>>> this change will not work with TLS, disabling users to use > > > > ACLs > > > > > >> when > > > > > >> >> the > > > > > >> >> >>>> Zookeeper cluster use TLS. > > > > > >> >> >>>> > > > > > >> >> >>>> link: > > > > > >> >> >>>> > > > > > >> >> >>>> > > > > > >> >> > > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > > > > >> >> >>>> > > > > > >> >> >>>> Looking forward to hearing from you on this, > > > > > >> >> >>>> > > > > > >> >> >>>> /cheers > > > > > >> >> >>>> > > > > > >> >> >>>> -- > > > > > >> >> >>>> Pere Urbon-Bayes > > > > > >> >> >>>> Software Architect > > > > > >> >> >>>> http://www.purbon.com > > > > > >> >> >>>> https://twitter.com/purbon > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/ > > > > > >> >> >>>> > > > > > >> >> >>>> -- > > > > > >> >> >>>> Pere Urbon-Bayes > > > > > >> >> >>>> Software Architect > > > > > >> >> >>>> http://www.purbon.com > > > > > >> >> >>>> https://twitter.com/purbon > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/ > > > > > >> >> >>>> > > > > > >> >> >>> > > > > > >> >> >>> > > > > > >> >> >> > > > > > >> >> >> -- > > > > > >> >> >> Pere Urbon-Bayes > > > > > >> >> >> Software Architect > > > > > >> >> >> http://www.purbon.com > > > > > >> >> >> https://twitter.com/purbon > > > > > >> >> >> https://www.linkedin.com/in/purbon/ > > > > > >> >> >> > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> -- > > > > > >> >> Pere Urbon-Bayes > > > > > >> >> Software Architect > > > > > >> >> http://www.purbon.com > > > > > >> >> https://twitter.com/purbon > > > > > >> >> https://www.linkedin.com/in/purbon/ > > > > > >> >> > > > > > >> > > > > > > >> > > > > > >> -- > > > > > >> Pere Urbon-Bayes > > > > > >> Software Architect > > > > > >> http://www.purbon.com > > > > > >> https://twitter.com/purbon > > > > > >> https://www.linkedin.com/in/purbon/ > > > > > >> > > > > > > > > > > > > > > > > > > > > >