Hi Rajini. Thanks for pointing out kafka-acls as another bootstrap use case. I will update the KIP to mention it. (BTW, I sent an email on 1/8 stating that I had updated the KIP to reflect the current understanding and discussion, but that email did not go through -- the KIP has undergone a major rewrite, so interested parties should definitely take a look).
It is easy to support the bootstrap use case as I described previously, so I think we should support it in both kafka-acls and kafka-configs. Regarding SASL + SSL simultaneously, yes, the two can be used together in ZooKeeper. I will be sure to state this in any documentation, and I will update the KIP to fully describe how this works, including the principal that is used and whether there is any way to transform the DN in the certificate. Here is a quick summary: 1) ZooKeeper 3.5.6 requires a client certificate. There is a sever-side config "ssl.clientAuth" that the code recognizes (case-insensitively: want/need/none are the valid options), but this config has no effect in 3.5.6 (https://issues.apache.org/jira/browse/ZOOKEEPER-3674). I built from source recently and confirmed that it worked in the 3.6 SNAPSHOT, but that version is not yet released. 2) ZooKeeper will associate multiple identities with any session that authenticates multiple ways (e.g. SSL and SASL). A client is authorized if it has at least 1 of the identities present in an ACL. 3) The SSL identity is the DN, and that can only be changed by implementing and using a custom ZK authentication provider that overrides the method protected String getClientId(X509Certificate clientCert). Given the above, if clients want to use client certificate authentication with SASL, then there really isn't any difficulty -- brokers and ZK security migrator will be identified by both their respective DNs and the SASL identity, and at least the SASL identity will be common to all clients as it is today, so it continues to work fine. If clients want to use client certificate authentication without SASL (which is I believe what Pere intended in the original KIP discussion), then either all brokers and ZK security migrator will have to present the same DN (with different SANs for the different hostnames) or ZooKeeper will have to be running with a custom authProvider that maps the different DNs to a single, common identity (this second approach is more difficult and seems to be less secure than the first one that uses SANs since it creates the possibility of the DN mapping being leveraged in a malicious way). Since I am catching up on emails I sent that did not go through, I'll also include the following two comments here: 1) The ZooKeeper project's command line interface has no facility for accepting client TLS configuration except via Java system properties. I've opened ZooKeeper issue https://issues.apache.org/jira/browse/ZOOKEEPER-3689 for this, but in the meantime we can address it in Kafka's zookeeper-shell.{bat,sh} scripts by invoking a new, Kafka-specific subclass of org.apache.zookeeper.ZooKeeperMain as opposed to org.apache.zookeeper.ZooKeeperMain itself. We would simply look for a "-zk-tls-config-file <file>" argument pair, create the TLS client config if specified, and pass the arguments without the config information to the superclass; then we would override the connectToZK(String) method to use any created TLS client config. While the subclass unfortunately needs to be in the same org.apache.zookeeper package due to the package-private run() method in ZooKeeperMain that it has to call, it is working fine (confirmed via a system test). 2) Regarding Thomas Zhou's question on 1/10 about changing the way client certificates are loaded ( https://lists.apache.org/thread.html/r3bf6c58b052bf47bc99c360dff86a8bd38c35a89a7387ba2a5f16dee%40%3Cdev.kafka.apache.org%3E), when I updated the KIP I included all of the ZooKeeper-supported client configurations related to TLS (not just keystore/trsustore locations and passwords). There is no intention to support anything beyond what ZooKeeper supports, but the "zookeeper.ssl.context.supplier.class" configuration is included and is useful for this particular use case. Ron On Tue, Jan 14, 2020 at 10:48 AM Rajini Sivaram <rajinisiva...@gmail.com> wrote: > Resending this note which wasn't delivered when it was sent on Jan 6th: > > Hi Ron, > > Thanks for taking over this KIP. Is it possible to use SASL/GSSAPI with SSL > for ZooKeeper like we do in Kafka? If not, we need to describe that in the > migration section. We should also document the principals used for > authorization when SSL is used and any additional configs that may be used > for transforming the DN from the certificate. > > For the CLI tools, the two that use ZooKeeper for bootstrapping are > kafka-configs.sh and kafka-acls.sh. In both cases, we support direct > connectivity to ZooKeeper in order to store configs/ACLs prior to starting > brokers. Are we going to support SSL for both? > > Regards, > > Rajini > > On Tue, Jan 14, 2020 at 3:39 PM Ron Dagostino <rndg...@gmail.com> wrote: > > > Hi Colin. > > > > <<< It seems like this [--zk-tls-config-file information] could just > appear > > in a configuration file, which all of these tools already accept (I > think) > > > > ZkSecurityMigrator has no such property file facility; adding a > > "--zk-tls-config-file" parameter is exactly for this purpose. If we add > > that to ZkSecurityMigrator then it is trivial to add it to other commands > > (the same code is simply reused; it ends up being just a few extra > lines). > > I do not see any parameter in the other two commands to adjust the ZK > > connection; ConfigCommand accepts a "--command-config" flag, but > according > > to the code "This is used only with --bootstrap-server option for > > describing and altering broker configs." > > > > I do agree there would be no need to add "--zk-tls-config-file" to > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in > time > > for the next release. > > > > ConfigCommand supports the "--bootstrap-server" option and will have its > > direct ZooKeeper access formally deprecated as per KIP-555, but the > special > > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials > > prior to starting Kafka will still exist, so it feels like while > > "--zk-tls-config-file" would never be used except for this use case it > > could probably still be added for this particular situation. > > > > Ron > > > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I > > sent) did not go through. I am trying to get emails through now to move > > this discussion forward. > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cmcc...@apache.org> wrote: > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote: > > > > Hi everyone. I would like to make the following changes to the KIP. > > > > > > > > MOTIVATION: > > > > Include a statement that it will be difficult in the short term to > > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat} > > > (which > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka > cluster > > > > with encrypted passwords in Zookeeper is an explicitly-supported use > > > case; > > > > therefore it is in scope to be able to securely configure the CLI > tools > > > > that still leverage non-deprecated direct Zookeeper communication for > > TLS > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and > > > > zookeeper-security-migration.sh). > > > > > > Hi Ron, > > > > > > Thanks for the KIP. > > > > > > About deprecations: > > > > > > * zookeeper-security-migration: clearly, deprecating ZK access in this > > one > > > would not make sense, since it would defeat the whole point of the tool > > :) > > > > > > * kafka-reassign-partitions: ZK access should be deprecated here. > > KIP-455 > > > implementation has dragged a bit, but this should get done soon. > > Certainly > > > before the next release. > > > > > > * kafka-configs: I think ZK access should be deprecated here as well. > I > > > agree there is a super-special bootstrapping case here, but that should > > > have its own tool, not use kafka-configs. > > > > > > I will post a separate KIP for this, though. > > > > > > > > > > > GOALS: > > > > Support the secure configuration of TLS-encrypted communication > between > > > > Zookeeper and: > > > > a) Kafka brokers > > > > b) The three CLI tools mentioned above that still support direct, > > > > non-deprecated communication to Zookeeper > > > > It is explicitly out-of-scope to deprecate any direct Zookeeper > > > > communication in CLI tools as part of this KIP; such work will occur > in > > > > future KIPs instead. > > > > > > > > PUBLIC INTERFACES: > > > > 1) The following new broker configurations will be recognized. > > > > zookeeper.client.secure (default value = false, for backwards > > > > compatibility) > > > > zookeeper.clientCnxnSocket > > > > zookeeper.ssl.keyStore.location > > > > zookeeper.ssl.keyStore.password > > > > zookeeper.ssl.trustStore.location > > > > zookeeper.ssl.trustStore.password > > > > It will be an error for any of the last 5 values to be left > unspecified > > > if > > > > zookeeper.client.secure is explicitly set to true. > > > > > > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class > > > supports > > > > the ability to connect to a different Zookeeper instance than the one > > the > > > > brokers use. We therefore also add the following optional configs, > > which > > > > override the corresponding ones from above when present: > > > > authorizer.zookeeper.client.secure > > > > authorizer.zookeeper.clientCnxnSocket > > > > authorizer.zookeeper.ssl.keyStore.location > > > > authorizer.zookeeper.ssl.keyStore.password > > > > authorizer.zookeeper.ssl.trustStore.location > > > > authorizer.zookeeper.ssl.trustStore.password > > > > > > > > 3) The three CLI tools mentioned above will support a new > > > --zk-tls-config-file > > > > <String: Zookeeper TLS configuration file path>" option. The > following > > > > properties will be recognized in that file, and unrecognized > properties > > > > will be ignored to allow the possibility of pointing > zk-tls-config-file > > > at > > > > the broker's config file. > > > > zookeeper.client.secure (default value = false) > > > > zookeeper.clientCnxnSocket > > > > zookeeper.ssl.keyStore.location > > > > zookeeper.ssl.keyStore.password > > > > zookeeper.ssl.trustStore.location > > > > zookeeper.ssl.trustStore.password > > > > It will be an error for any of the last 5 values to be left > unspecified > > > if > > > > zookeeper.client.secure is explicitly set to true. > > > > > > Do we really need a --zk-tls-config-file flag? It seems like this > could > > > just appear in a configuration file, which all of these tools already > > > accept (I think). > > > > > > best, > > > Colin > > > > > > > > > > > > > > Ron > > > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rndg...@gmail.com> > > wrote: > > > > > > > > > Hi everyone. Let's get this discussion going again now that Kafka > > 2.4 > > > > > with Zookeeper 3.5.6 is out. > > > > > > > > > > First, regarding the KIP number, the other KIP that was using this > > > number > > > > > moved to KIP 534, so KIP 515 remains the correct number for this > > > > > discussion. I've updated the Kafka Improvement Proposal page to > list > > > this > > > > > KIP in the 515 slot, so we're all set there. > > > > > > > > > > Regarding the actual issues under discussion, I think there are > some > > > > > things we should clarify. > > > > > > > > > > 1) It is possible to use TLS connectivity to Zookeeper from Apache > > > Kafka > > > > > 2.4 -- the problem is that configuration information has to be > passed > > > via > > > > > system properties as "-D" command line options on the java > invocation > > > of > > > > > the broker, and those are not secure (anyone with access to the box > > > can see > > > > > the command line used to invoke the broker); the configuration > > includes > > > > > sensitive information (e.g. a keystore password), so we need a > secure > > > > > mechanism for passing the configuration values. I believe the real > > > > > motivation for this KIP is to harden the configuration mechanism > for > > > > > Zookeeper TLS connectivity. > > > > > > > > > > 2) I believe the list of CLI tools that continue to use direct > > > Zookeeper > > > > > connectivity in a non-deprecated fashion is: > > > > > a) > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator > > > > > b) > > > > > > > > > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand > > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand > > > > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum because > > it > > > > > explicitly states in a comment that a supported use case is > > > bootstrapping a > > > > > Kafka cluster with encrypted passwords in Zookeeper (see > > > > > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65 > > > ). > > > > > This means it will be especially difficult to deprecate this > > particular > > > > > direct Zookeeper connectivity without a different storage mechanism > > for > > > > > dynamic configuration values being available (i.e. the self-managed > > > quorum > > > > > referred to in KIP-500). > > > > > > > > > > I think it would be easier and simpler to harden the Zookeeper TLS > > > > > configuration in both Kafka and in CLI tools compared to trying to > > > > > deprecate the direct Zookeeper connectivity in the above 3 CLI > tools > > -- > > > > > especially when ConfigCommand has no obvious short-term path to > > > deprecation. > > > > > > > > > > Regarding how to harden the TLS configuration, adding the > appropriate > > > > > configs to Kafka is an obvious choice for the brokers. Not that > > these > > > > > values cannot be dynamically reconfigurable because the values > would > > be > > > > > required to connect to Zookeeper via TLS in the first place. > > > > > > > > > > Regarding how to harden the TLS configuration for the 3 remaining > CLI > > > > > tools, it would be relatively straightforward to add support for a > > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system > > > > > properties file path>" option to each one. > > > > > > > > > > Thoughts? > > > > > > > > > > Ron > > > > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes < > > pere.ur...@gmail.com > > > > > > > > > wrote: > > > > > > > > > >> True, the number is duplicated. > > > > >> > > > > >> do you know how can we get the problem fix? I was not aware, I > > > missed, > > > > >> sorry the need to add the KIP to the table. > > > > >> > > > > >> -- Pere > > > > >> > > > > >> Missatge de Jorge Esteban Quilcate Otoya < > quilcate.jo...@gmail.com> > > > del > > > > >> dia > > > > >> dt., 3 de set. 2019 a les 13:43: > > > > >> > > > > >> > Hi Pere, > > > > >> > > > > > >> > Have you add your KIP to the list here > > > > >> > > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > >> ? > > > > >> > > > > > >> > I found the KIP number assigned to another. > > > > >> > > > > > >> > > > > > >> > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes < > > > pere.ur...@gmail.com> > > > > >> > wrote: > > > > >> > > > > > >> >> Thanks for your time Harsha, > > > > >> >> anyone else with comments? looking forward to hearing from > > you. > > > > >> >> > > > > >> >> Stupid question: when do you move from discussion to vote? > > > > >> >> > > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., > > 30 > > > > >> d’ag. > > > > >> >> 2019 a les 21:59: > > > > >> >> > > > > >> >> > Thanks Pere. KIP looks good to me. > > > > >> >> > -Harsha > > > > >> >> > > > > > >> >> > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes < > > > > >> >> pere.ur...@gmail.com> > > > > >> >> > wrote: > > > > >> >> > > > > > >> >> >> Not really, > > > > >> >> >> my idea is to keep the JAAS parameter, so people don't see > > > major > > > > >> >> >> changes. But if you pass a properties file, then this takes > > > > >> precedence > > > > >> >> over > > > > >> >> >> the other, with the idea that you can do sasl as well with > the > > > > >> >> properties > > > > >> >> >> files. > > > > >> >> >> > > > > >> >> >> Makes sense? > > > > >> >> >> > > > > >> >> >> -- Pere > > > > >> >> >> > > > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia > > dv., > > > 30 > > > > >> >> d’ag. > > > > >> >> >> 2019 a les 19:00: > > > > >> >> >> > > > > >> >> >>> Hi Pere, > > > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper > for > > > Kafka > > > > >> >> makes > > > > >> >> >>> sense. > > > > >> >> >>> "The changes are planned to be introduced in a compatible > > way, > > > by > > > > >> >> >>> keeping the current JAAS variable precedence." > > > > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS > > > file > > > > >> with > > > > >> >> >>> Client section it will take precedence over zookeeper SSL > > > configs? > > > > >> >> >>> > > > > >> >> >>> Thanks, > > > > >> >> >>> Harsha > > > > >> >> >>> > > > > >> >> >>> > > > > >> >> >>> > > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes < > > > > >> >> pere.ur...@gmail.com> > > > > >> >> >>> wrote: > > > > >> >> >>> > > > > >> >> >>>> Hi, > > > > >> >> >>>> quick question, I saw in another mail that 2.4 release is > > > planned > > > > >> for > > > > >> >> >>>> September. I think it would be really awesome to have this > > for > > > > >> this > > > > >> >> >>>> release, do you think we can make it? > > > > >> >> >>>> > > > > >> >> >>>> -- Pere > > > > >> >> >>>> > > > > >> >> >>>> Missatge de Pere Urbón Bayes <pere.ur...@gmail.com> del > dia > > > dj., > > > > >> 29 > > > > >> >> >>>> d’ag. 2019 a les 20:10: > > > > >> >> >>>> > > > > >> >> >>>> Hi, > > > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm > > > really > > > > >> need > > > > >> >> >>>> to the process. Looking forward to hearing from you and > > learn > > > the > > > > >> >> best > > > > >> >> >>>> ropes here. > > > > >> >> >>>> > > > > >> >> >>>> I would like to propose this KIP-515 to enable the > > > > >> ZookeeperClients > > > > >> >> to > > > > >> >> >>>> take full advantage of the TLS communication in the new > > > Zookeeper > > > > >> >> 3.5.5. > > > > >> >> >>>> Specially interesting it the Zookeeper Security Migration, > > > that > > > > >> >> without > > > > >> >> >>>> this change will not work with TLS, disabling users to use > > > ACLs > > > > >> when > > > > >> >> the > > > > >> >> >>>> Zookeeper cluster use TLS. > > > > >> >> >>>> > > > > >> >> >>>> link: > > > > >> >> >>>> > > > > >> >> >>>> > > > > >> >> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication > > > > >> >> >>>> > > > > >> >> >>>> Looking forward to hearing from you on this, > > > > >> >> >>>> > > > > >> >> >>>> /cheers > > > > >> >> >>>> > > > > >> >> >>>> -- > > > > >> >> >>>> Pere Urbon-Bayes > > > > >> >> >>>> Software Architect > > > > >> >> >>>> http://www.purbon.com > > > > >> >> >>>> https://twitter.com/purbon > > > > >> >> >>>> https://www.linkedin.com/in/purbon/ > > > > >> >> >>>> > > > > >> >> >>>> -- > > > > >> >> >>>> Pere Urbon-Bayes > > > > >> >> >>>> Software Architect > > > > >> >> >>>> http://www.purbon.com > > > > >> >> >>>> https://twitter.com/purbon > > > > >> >> >>>> https://www.linkedin.com/in/purbon/ > > > > >> >> >>>> > > > > >> >> >>> > > > > >> >> >>> > > > > >> >> >> > > > > >> >> >> -- > > > > >> >> >> Pere Urbon-Bayes > > > > >> >> >> Software Architect > > > > >> >> >> http://www.purbon.com > > > > >> >> >> https://twitter.com/purbon > > > > >> >> >> https://www.linkedin.com/in/purbon/ > > > > >> >> >> > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> -- > > > > >> >> Pere Urbon-Bayes > > > > >> >> Software Architect > > > > >> >> http://www.purbon.com > > > > >> >> https://twitter.com/purbon > > > > >> >> https://www.linkedin.com/in/purbon/ > > > > >> >> > > > > >> > > > > > >> > > > > >> -- > > > > >> Pere Urbon-Bayes > > > > >> Software Architect > > > > >> http://www.purbon.com > > > > >> https://twitter.com/purbon > > > > >> https://www.linkedin.com/in/purbon/ > > > > >> > > > > > > > > > > > > > > >
