Hi Ron,
Thanks for the detailed explanation, sounds good to me.
A few more questions:
1) At the moment, all sensitive broker configs including
keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
starting up brokers. We are now adding SSL keystore/truststore passwords
for ZooKeeper client that cannot be stored in ZooKeeper since you need
these to connect to ZK. We should perhaps document that these configs can
be encrypted using KIP-421.
2) We can dynamically update keystores and trust stores used by brokers
without restarting the broker. Can we support this easily for ZK clients
created by the broker, for example by adding our own
`zookeeper.ssl.context.supplier.class`?
3) Looks like we are using config names that directly map to ZK configs.
Have we considered using equivalent Kafka config names with prefixes,
perhaps with inheritance from the non-prefixed names? Not sure if this is a
good idea, but perhaps worth documenting in Rejected Alternatives at least.
On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cmcc...@apache.org> wrote:
> Hi Ron,
>
> Thanks for the explanation. I guess thinking about it a little bit more,
> we should just add --zk-tls-config-file to all of these commands.
>
> We will be removing this option (plus ZK in general) from these commands
> in the next major release, but ZK is still supported in 2.5, so we should
> just do the logical thing. (The exception is ZkSecurityMigrator, which
> will stay).
>
> best,
> Colin
>
>
> On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > Hi Colin.
> >
> > <<< It seems like this [--zk-tls-config-file information] could just
> appear
> > in a configuration file, which all of these tools already accept (I
> think)
> >
> > ZkSecurityMigrator has no such property file facility; adding a
> > "--zk-tls-config-file" parameter is exactly for this purpose. If we add
> > that to ZkSecurityMigrator then it is trivial to add it to other commands
> > (the same code is simply reused; it ends up being just a few extra
> lines).
> > I do not see any parameter in the other two commands to adjust the ZK
> > connection; ConfigCommand accepts a "--command-config" flag, but
> according
> > to the code "This is used only with --bootstrap-server option for
> > describing and altering broker configs."
> >
> > I do agree there would be no need to add "--zk-tls-config-file" to
> > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> time
> > for the next release.
> >
> > ConfigCommand supports the "--bootstrap-server" option and will have its
> > direct ZooKeeper access formally deprecated as per KIP-555, but the
> special
> > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > prior to starting Kafka will still exist, so it feels like while
> > "--zk-tls-config-file" would never be used except for this use case it
> > could probably still be added for this particular situation.
> >
> > Ron
> >
> > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > sent) did not go through. I am trying to get emails through now to move
> > this discussion forward.
> >
> > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cmcc...@apache.org> wrote:
> >
> > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > Hi everyone. I would like to make the following changes to the KIP.
> > > >
> > > > MOTIVATION:
> > > > Include a statement that it will be difficult in the short term to
> > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > (which
> > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> cluster
> > > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > > case;
> > > > therefore it is in scope to be able to securely configure the CLI
> tools
> > > > that still leverage non-deprecated direct Zookeeper communication
> for TLS
> > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > zookeeper-security-migration.sh).
> > >
> > > Hi Ron,
> > >
> > > Thanks for the KIP.
> > >
> > > About deprecations:
> > >
> > > * zookeeper-security-migration: clearly, deprecating ZK access in this
> one
> > > would not make sense, since it would defeat the whole point of the
> tool :)
> > >
> > > * kafka-reassign-partitions: ZK access should be deprecated here.
> KIP-455
> > > implementation has dragged a bit, but this should get done soon.
> Certainly
> > > before the next release.
> > >
> > > * kafka-configs: I think ZK access should be deprecated here as well.
> I
> > > agree there is a super-special bootstrapping case here, but that should
> > > have its own tool, not use kafka-configs.
> > >
> > > I will post a separate KIP for this, though.
> > >
> > > >
> > > > GOALS:
> > > > Support the secure configuration of TLS-encrypted communication
> between
> > > > Zookeeper and:
> > > > a) Kafka brokers
> > > > b) The three CLI tools mentioned above that still support direct,
> > > > non-deprecated communication to Zookeeper
> > > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > > communication in CLI tools as part of this KIP; such work will occur
> in
> > > > future KIPs instead.
> > > >
> > > > PUBLIC INTERFACES:
> > > > 1) The following new broker configurations will be recognized.
> > > > zookeeper.client.secure (default value = false, for backwards
> > > > compatibility)
> > > > zookeeper.clientCnxnSocket
> > > > zookeeper.ssl.keyStore.location
> > > > zookeeper.ssl.keyStore.password
> > > > zookeeper.ssl.trustStore.location
> > > > zookeeper.ssl.trustStore.password
> > > > It will be an error for any of the last 5 values to be left
> unspecified
> > > if
> > > > zookeeper.client.secure is explicitly set to true.
> > > >
> > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer class
> > > supports
> > > > the ability to connect to a different Zookeeper instance than the
> one the
> > > > brokers use. We therefore also add the following optional configs,
> which
> > > > override the corresponding ones from above when present:
> > > > authorizer.zookeeper.client.secure
> > > > authorizer.zookeeper.clientCnxnSocket
> > > > authorizer.zookeeper.ssl.keyStore.location
> > > > authorizer.zookeeper.ssl.keyStore.password
> > > > authorizer.zookeeper.ssl.trustStore.location
> > > > authorizer.zookeeper.ssl.trustStore.password
> > > >
> > > > 3) The three CLI tools mentioned above will support a new
> > > --zk-tls-config-file
> > > > <String: Zookeeper TLS configuration file path>" option. The
> following
> > > > properties will be recognized in that file, and unrecognized
> properties
> > > > will be ignored to allow the possibility of pointing
> zk-tls-config-file
> > > at
> > > > the broker's config file.
> > > > zookeeper.client.secure (default value = false)
> > > > zookeeper.clientCnxnSocket
> > > > zookeeper.ssl.keyStore.location
> > > > zookeeper.ssl.keyStore.password
> > > > zookeeper.ssl.trustStore.location
> > > > zookeeper.ssl.trustStore.password
> > > > It will be an error for any of the last 5 values to be left
> unspecified
> > > if
> > > > zookeeper.client.secure is explicitly set to true.
> > >
> > > Do we really need a --zk-tls-config-file flag? It seems like this
> could
> > > just appear in a configuration file, which all of these tools already
> > > accept (I think).
> > >
> > > best,
> > > Colin
> > >
> > >
> > > >
> > > > Ron
> > > >
> > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rndg...@gmail.com>
> wrote:
> > > >
> > > > > Hi everyone. Let's get this discussion going again now that Kafka
> 2.4
> > > > > with Zookeeper 3.5.6 is out.
> > > > >
> > > > > First, regarding the KIP number, the other KIP that was using this
> > > number
> > > > > moved to KIP 534, so KIP 515 remains the correct number for this
> > > > > discussion. I've updated the Kafka Improvement Proposal page to
> list
> > > this
> > > > > KIP in the 515 slot, so we're all set there.
> > > > >
> > > > > Regarding the actual issues under discussion, I think there are
> some
> > > > > things we should clarify.
> > > > >
> > > > > 1) It is possible to use TLS connectivity to Zookeeper from Apache
> > > Kafka
> > > > > 2.4 -- the problem is that configuration information has to be
> passed
> > > via
> > > > > system properties as "-D" command line options on the java
> invocation
> > > of
> > > > > the broker, and those are not secure (anyone with access to the box
> > > can see
> > > > > the command line used to invoke the broker); the configuration
> includes
> > > > > sensitive information (e.g. a keystore password), so we need a
> secure
> > > > > mechanism for passing the configuration values. I believe the real
> > > > > motivation for this KIP is to harden the configuration mechanism
> for
> > > > > Zookeeper TLS connectivity.
> > > > >
> > > > > 2) I believe the list of CLI tools that continue to use direct
> > > Zookeeper
> > > > > connectivity in a non-deprecated fashion is:
> > > > > a)
> zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > b)
> > > > >
> > >
> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > >
> > > > > 3) I believe Kafka.admin.ConfigCommand presents a conundrum
> because it
> > > > > explicitly states in a comment that a supported use case is
> > > bootstrapping a
> > > > > Kafka cluster with encrypted passwords in Zookeeper (see
> > > > >
> > >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > ).
> > > > > This means it will be especially difficult to deprecate this
> particular
> > > > > direct Zookeeper connectivity without a different storage
> mechanism for
> > > > > dynamic configuration values being available (i.e. the self-managed
> > > quorum
> > > > > referred to in KIP-500).
> > > > >
> > > > > I think it would be easier and simpler to harden the Zookeeper TLS
> > > > > configuration in both Kafka and in CLI tools compared to trying to
> > > > > deprecate the direct Zookeeper connectivity in the above 3 CLI
> tools --
> > > > > especially when ConfigCommand has no obvious short-term path to
> > > deprecation.
> > > > >
> > > > > Regarding how to harden the TLS configuration, adding the
> appropriate
> > > > > configs to Kafka is an obvious choice for the brokers. Not that
> these
> > > > > values cannot be dynamically reconfigurable because the values
> would be
> > > > > required to connect to Zookeeper via TLS in the first place.
> > > > >
> > > > > Regarding how to harden the TLS configuration for the 3 remaining
> CLI
> > > > > tools, it would be relatively straightforward to add support for a
> > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > > > > properties file path>" option to each one.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Ron
> > > > >
> > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <
> pere.ur...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> True, the number is duplicated.
> > > > >>
> > > > >> do you know how can we get the problem fix? I was not aware, I
> > > missed,
> > > > >> sorry the need to add the KIP to the table.
> > > > >>
> > > > >> -- Pere
> > > > >>
> > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com>
> > > del
> > > > >> dia
> > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > >>
> > > > >> > Hi Pere,
> > > > >> >
> > > > >> > Have you add your KIP to the list here
> > > > >> >
> > > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > >> ?
> > > > >> >
> > > > >> > I found the KIP number assigned to another.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <
> > > pere.ur...@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> >> Thanks for your time Harsha,
> > > > >> >> anyone else with comments? looking forward to hearing from
> you.
> > > > >> >>
> > > > >> >> Stupid question: when do you move from discussion to vote?
> > > > >> >>
> > > > >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> dv., 30
> > > > >> d’ag.
> > > > >> >> 2019 a les 21:59:
> > > > >> >>
> > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > >> >> > -Harsha
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> > > > >> >> pere.ur...@gmail.com>
> > > > >> >> > wrote:
> > > > >> >> >
> > > > >> >> >> Not really,
> > > > >> >> >> my idea is to keep the JAAS parameter, so people don't see
> > > major
> > > > >> >> >> changes. But if you pass a properties file, then this takes
> > > > >> precedence
> > > > >> >> over
> > > > >> >> >> the other, with the idea that you can do sasl as well with
> the
> > > > >> >> properties
> > > > >> >> >> files.
> > > > >> >> >>
> > > > >> >> >> Makes sense?
> > > > >> >> >>
> > > > >> >> >> -- Pere
> > > > >> >> >>
> > > > >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia
> dv.,
> > > 30
> > > > >> >> d’ag.
> > > > >> >> >> 2019 a les 19:00:
> > > > >> >> >>
> > > > >> >> >>> Hi Pere,
> > > > >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper
> for
> > > Kafka
> > > > >> >> makes
> > > > >> >> >>> sense.
> > > > >> >> >>> "The changes are planned to be introduced in a compatible
> way,
> > > by
> > > > >> >> >>> keeping the current JAAS variable precedence."
> > > > >> >> >>> Can you elaborate a bit here. If the user configures a JAAS
> > > file
> > > > >> with
> > > > >> >> >>> Client section it will take precedence over zookeeper SSL
> > > configs?
> > > > >> >> >>>
> > > > >> >> >>> Thanks,
> > > > >> >> >>> Harsha
> > > > >> >> >>>
> > > > >> >> >>>
> > > > >> >> >>>
> > > > >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> > > > >> >> pere.ur...@gmail.com>
> > > > >> >> >>> wrote:
> > > > >> >> >>>
> > > > >> >> >>>> Hi,
> > > > >> >> >>>> quick question, I saw in another mail that 2.4 release is
> > > planned
> > > > >> for
> > > > >> >> >>>> September. I think it would be really awesome to have
> this for
> > > > >> this
> > > > >> >> >>>> release, do you think we can make it?
> > > > >> >> >>>>
> > > > >> >> >>>> -- Pere
> > > > >> >> >>>>
> > > > >> >> >>>> Missatge de Pere Urbón Bayes <pere.ur...@gmail.com> del
> dia
> > > dj.,
> > > > >> 29
> > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > >> >> >>>>
> > > > >> >> >>>> Hi,
> > > > >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm
> > > really
> > > > >> need
> > > > >> >> >>>> to the process. Looking forward to hearing from you and
> learn
> > > the
> > > > >> >> best
> > > > >> >> >>>> ropes here.
> > > > >> >> >>>>
> > > > >> >> >>>> I would like to propose this KIP-515 to enable the
> > > > >> ZookeeperClients
> > > > >> >> to
> > > > >> >> >>>> take full advantage of the TLS communication in the new
> > > Zookeeper
> > > > >> >> 3.5.5.
> > > > >> >> >>>> Specially interesting it the Zookeeper Security Migration,
> > > that
> > > > >> >> without
> > > > >> >> >>>> this change will not work with TLS, disabling users to use
> > > ACLs
> > > > >> when
> > > > >> >> the
> > > > >> >> >>>> Zookeeper cluster use TLS.
> > > > >> >> >>>>
> > > > >> >> >>>> link:
> > > > >> >> >>>>
> > > > >> >> >>>>
> > > > >> >>
> > > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > >> >> >>>>
> > > > >> >> >>>> Looking forward to hearing from you on this,
> > > > >> >> >>>>
> > > > >> >> >>>> /cheers
> > > > >> >> >>>>
> > > > >> >> >>>> --
> > > > >> >> >>>> Pere Urbon-Bayes
> > > > >> >> >>>> Software Architect
> > > > >> >> >>>> http://www.purbon.com
> > > > >> >> >>>> https://twitter.com/purbon
> > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > >> >> >>>>
> > > > >> >> >>>> --
> > > > >> >> >>>> Pere Urbon-Bayes
> > > > >> >> >>>> Software Architect
> > > > >> >> >>>> http://www.purbon.com
> > > > >> >> >>>> https://twitter.com/purbon
> > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > >> >> >>>>
> > > > >> >> >>>
> > > > >> >> >>>
> > > > >> >> >>
> > > > >> >> >> --
> > > > >> >> >> Pere Urbon-Bayes
> > > > >> >> >> Software Architect
> > > > >> >> >> http://www.purbon.com
> > > > >> >> >> https://twitter.com/purbon
> > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > >> >> >>
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >> >> --
> > > > >> >> Pere Urbon-Bayes
> > > > >> >> Software Architect
> > > > >> >> http://www.purbon.com
> > > > >> >> https://twitter.com/purbon
> > > > >> >> https://www.linkedin.com/in/purbon/
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >> --
> > > > >> Pere Urbon-Bayes
> > > > >> Software Architect
> > > > >> http://www.purbon.com
> > > > >> https://twitter.com/purbon
> > > > >> https://www.linkedin.com/in/purbon/
> > > > >>
> > > > >
> > > >
> > >
> >
>
