On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> Hi everyone. I would like to make the following changes to the KIP.
>
> MOTIVATION:
> Include a statement that it will be difficult in the short term to
> deprecate direct Zookeeper communication in kafka-configs.{sh, bat} (which
> invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
> with encrypted passwords in Zookeeper is an explicitly-supported use case;
> therefore it is in scope to be able to securely configure the CLI tools
> that still leverage non-deprecated direct Zookeeper communication for TLS
> (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> zookeeper-security-migration.sh).
Hi Ron,
Thanks for the KIP.
About deprecations:
* zookeeper-security-migration: clearly, deprecating ZK access in this one
would not make sense, since it would defeat the whole point of the tool :)
* kafka-reassign-partitions: ZK access should be deprecated here. KIP-455
implementation has dragged a bit, but this should get done soon. Certainly
before the next release.
* kafka-configs: I think ZK access should be deprecated here as well. I agree
there is a super-special bootstrapping case here, but that should have its own
tool, not use kafka-configs.
I will post a separate KIP for this, though.
>
> GOALS:
> Support the secure configuration of TLS-encrypted communication between
> Zookeeper and:
> a) Kafka brokers
> b) The three CLI tools mentioned above that still support direct,
> non-deprecated communication to Zookeeper
> It is explicitly out-of-scope to deprecate any direct Zookeeper
> communication in CLI tools as part of this KIP; such work will occur in
> future KIPs instead.
>
> PUBLIC INTERFACES:
> 1) The following new broker configurations will be recognized.
> zookeeper.client.secure (default value = false, for backwards
> compatibility)
> zookeeper.clientCnxnSocket
> zookeeper.ssl.keyStore.location
> zookeeper.ssl.keyStore.password
> zookeeper.ssl.trustStore.location
> zookeeper.ssl.trustStore.password
> It will be an error for any of the last 5 values to be left unspecified if
> zookeeper.client.secure is explicitly set to true.
>
> 2) In addition, the kafka.security.authorizer.AclAuthorizer class supports
> the ability to connect to a different Zookeeper instance than the one the
> brokers use. We therefore also add the following optional configs, which
> override the corresponding ones from above when present:
> authorizer.zookeeper.client.secure
> authorizer.zookeeper.clientCnxnSocket
> authorizer.zookeeper.ssl.keyStore.location
> authorizer.zookeeper.ssl.keyStore.password
> authorizer.zookeeper.ssl.trustStore.location
> authorizer.zookeeper.ssl.trustStore.password
>
> 3) The three CLI tools mentioned above will support a new --zk-tls-config-file
> <String: Zookeeper TLS configuration file path>" option. The following
> properties will be recognized in that file, and unrecognized properties
> will be ignored to allow the possibility of pointing zk-tls-config-file at
> the broker's config file.
> zookeeper.client.secure (default value = false)
> zookeeper.clientCnxnSocket
> zookeeper.ssl.keyStore.location
> zookeeper.ssl.keyStore.password
> zookeeper.ssl.trustStore.location
> zookeeper.ssl.trustStore.password
> It will be an error for any of the last 5 values to be left unspecified if
> zookeeper.client.secure is explicitly set to true.
Do we really need a --zk-tls-config-file flag? It seems like this could just
appear in a configuration file, which all of these tools already accept (I
think).
best,
Colin
>
> Ron
>
> On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rndg...@gmail.com> wrote:
>
> > Hi everyone. Let's get this discussion going again now that Kafka 2.4
> > with Zookeeper 3.5.6 is out.
> >
> > First, regarding the KIP number, the other KIP that was using this number
> > moved to KIP 534, so KIP 515 remains the correct number for this
> > discussion. I've updated the Kafka Improvement Proposal page to list this
> > KIP in the 515 slot, so we're all set there.
> >
> > Regarding the actual issues under discussion, I think there are some
> > things we should clarify.
> >
> > 1) It is possible to use TLS connectivity to Zookeeper from Apache Kafka
> > 2.4 -- the problem is that configuration information has to be passed via
> > system properties as "-D" command line options on the java invocation of
> > the broker, and those are not secure (anyone with access to the box can see
> > the command line used to invoke the broker); the configuration includes
> > sensitive information (e.g. a keystore password), so we need a secure
> > mechanism for passing the configuration values. I believe the real
> > motivation for this KIP is to harden the configuration mechanism for
> > Zookeeper TLS connectivity.
> >
> > 2) I believe the list of CLI tools that continue to use direct Zookeeper
> > connectivity in a non-deprecated fashion is:
> > a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > b)
> > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> >
> > 3) I believe Kafka.admin.ConfigCommand presents a conundrum because it
> > explicitly states in a comment that a supported use case is bootstrapping a
> > Kafka cluster with encrypted passwords in Zookeeper (see
> > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65).
> > This means it will be especially difficult to deprecate this particular
> > direct Zookeeper connectivity without a different storage mechanism for
> > dynamic configuration values being available (i.e. the self-managed quorum
> > referred to in KIP-500).
> >
> > I think it would be easier and simpler to harden the Zookeeper TLS
> > configuration in both Kafka and in CLI tools compared to trying to
> > deprecate the direct Zookeeper connectivity in the above 3 CLI tools --
> > especially when ConfigCommand has no obvious short-term path to deprecation.
> >
> > Regarding how to harden the TLS configuration, adding the appropriate
> > configs to Kafka is an obvious choice for the brokers. Not that these
> > values cannot be dynamically reconfigurable because the values would be
> > required to connect to Zookeeper via TLS in the first place.
> >
> > Regarding how to harden the TLS configuration for the 3 remaining CLI
> > tools, it would be relatively straightforward to add support for a
> > "--zk-tls-config-file <String: Zookeeper TLS configuration system
> > properties file path>" option to each one.
> >
> > Thoughts?
> >
> > Ron
> >
> > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <pere.ur...@gmail.com>
> > wrote:
> >
> >> True, the number is duplicated.
> >>
> >> do you know how can we get the problem fix? I was not aware, I missed,
> >> sorry the need to add the KIP to the table.
> >>
> >> -- Pere
> >>
> >> Missatge de Jorge Esteban Quilcate Otoya <quilcate.jo...@gmail.com> del
> >> dia
> >> dt., 3 de set. 2019 a les 13:43:
> >>
> >> > Hi Pere,
> >> >
> >> > Have you add your KIP to the list here
> >> >
> >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >> ?
> >> >
> >> > I found the KIP number assigned to another.
> >> >
> >> >
> >> >
> >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <pere.ur...@gmail.com>
> >> > wrote:
> >> >
> >> >> Thanks for your time Harsha,
> >> >> anyone else with comments? looking forward to hearing from you.
> >> >>
> >> >> Stupid question: when do you move from discussion to vote?
> >> >>
> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
> >> d’ag.
> >> >> 2019 a les 21:59:
> >> >>
> >> >> > Thanks Pere. KIP looks good to me.
> >> >> > -Harsha
> >> >> >
> >> >> >
> >> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> >> >> pere.ur...@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> >> Not really,
> >> >> >> my idea is to keep the JAAS parameter, so people don't see major
> >> >> >> changes. But if you pass a properties file, then this takes
> >> precedence
> >> >> over
> >> >> >> the other, with the idea that you can do sasl as well with the
> >> >> properties
> >> >> >> files.
> >> >> >>
> >> >> >> Makes sense?
> >> >> >>
> >> >> >> -- Pere
> >> >> >>
> >> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
> >> >> d’ag.
> >> >> >> 2019 a les 19:00:
> >> >> >>
> >> >> >>> Hi Pere,
> >> >> >>> Thanks for the KIP. Enabling SSL for zookeeper for Kafka
> >> >> makes
> >> >> >>> sense.
> >> >> >>> "The changes are planned to be introduced in a compatible way, by
> >> >> >>> keeping the current JAAS variable precedence."
> >> >> >>> Can you elaborate a bit here. If the user configures a JAAS file
> >> with
> >> >> >>> Client section it will take precedence over zookeeper SSL configs?
> >> >> >>>
> >> >> >>> Thanks,
> >> >> >>> Harsha
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> >> >> pere.ur...@gmail.com>
> >> >> >>> wrote:
> >> >> >>>
> >> >> >>>> Hi,
> >> >> >>>> quick question, I saw in another mail that 2.4 release is planned
> >> for
> >> >> >>>> September. I think it would be really awesome to have this for
> >> this
> >> >> >>>> release, do you think we can make it?
> >> >> >>>>
> >> >> >>>> -- Pere
> >> >> >>>>
> >> >> >>>> Missatge de Pere Urbón Bayes <pere.ur...@gmail.com> del dia dj.,
> >> 29
> >> >> >>>> d’ag. 2019 a les 20:10:
> >> >> >>>>
> >> >> >>>> Hi,
> >> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm really
> >> need
> >> >> >>>> to the process. Looking forward to hearing from you and learn the
> >> >> best
> >> >> >>>> ropes here.
> >> >> >>>>
> >> >> >>>> I would like to propose this KIP-515 to enable the
> >> ZookeeperClients
> >> >> to
> >> >> >>>> take full advantage of the TLS communication in the new Zookeeper
> >> >> 3.5.5.
> >> >> >>>> Specially interesting it the Zookeeper Security Migration, that
> >> >> without
> >> >> >>>> this change will not work with TLS, disabling users to use ACLs
> >> when
> >> >> the
> >> >> >>>> Zookeeper cluster use TLS.
> >> >> >>>>
> >> >> >>>> link:
> >> >> >>>>
> >> >> >>>>
> >> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> >> >> >>>>
> >> >> >>>> Looking forward to hearing from you on this,
> >> >> >>>>
> >> >> >>>> /cheers
> >> >> >>>>
> >> >> >>>> --
> >> >> >>>> Pere Urbon-Bayes
> >> >> >>>> Software Architect
> >> >> >>>> http://www.purbon.com
> >> >> >>>> https://twitter.com/purbon
> >> >> >>>> https://www.linkedin.com/in/purbon/
> >> >> >>>>
> >> >> >>>> --
> >> >> >>>> Pere Urbon-Bayes
> >> >> >>>> Software Architect
> >> >> >>>> http://www.purbon.com
> >> >> >>>> https://twitter.com/purbon
> >> >> >>>> https://www.linkedin.com/in/purbon/
> >> >> >>>>
> >> >> >>>
> >> >> >>>
> >> >> >>
> >> >> >> --
> >> >> >> Pere Urbon-Bayes
> >> >> >> Software Architect
> >> >> >> http://www.purbon.com
> >> >> >> https://twitter.com/purbon
> >> >> >> https://www.linkedin.com/in/purbon/
> >> >> >>
> >> >> >
> >> >> >
> >> >>
> >> >> --
> >> >> Pere Urbon-Bayes
> >> >> Software Architect
> >> >> http://www.purbon.com
> >> >> https://twitter.com/purbon
> >> >> https://www.linkedin.com/in/purbon/
> >> >>
> >> >
> >>
> >> --
> >> Pere Urbon-Bayes
> >> Software Architect
> >> http://www.purbon.com
> >> https://twitter.com/purbon
> >> https://www.linkedin.com/in/purbon/
> >>
> >
>
