Hi everyone. I would like to make the following changes to the KIP.
MOTIVATION:
Include a statement that it will be difficult in the short term to
deprecate direct Zookeeper communication in kafka-configs.{sh, bat} (which
invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka cluster
with encrypted passwords in Zookeeper is an explicitly-supported use case;
therefore it is in scope to be able to securely configure the CLI tools
that still leverage non-deprecated direct Zookeeper communication for TLS
(the other 2 tools are kafka-reassign-partitions.{sh, bat} and
zookeeper-security-migration.sh).
GOALS:
Support the secure configuration of TLS-encrypted communication between
Zookeeper and:
a) Kafka brokers
b) The three CLI tools mentioned above that still support direct,
non-deprecated communication to Zookeeper
It is explicitly out-of-scope to deprecate any direct Zookeeper
communication in CLI tools as part of this KIP; such work will occur in
future KIPs instead.
PUBLIC INTERFACES:
1) The following new broker configurations will be recognized.
zookeeper.client.secure (default value = false, for backwards
compatibility)
zookeeper.clientCnxnSocket
zookeeper.ssl.keyStore.location
zookeeper.ssl.keyStore.password
zookeeper.ssl.trustStore.location
zookeeper.ssl.trustStore.password
It will be an error for any of the last 5 values to be left unspecified if
zookeeper.client.secure is explicitly set to true.
2) In addition, the kafka.security.authorizer.AclAuthorizer class supports
the ability to connect to a different Zookeeper instance than the one the
brokers use. We therefore also add the following optional configs, which
override the corresponding ones from above when present:
authorizer.zookeeper.client.secure
authorizer.zookeeper.clientCnxnSocket
authorizer.zookeeper.ssl.keyStore.location
authorizer.zookeeper.ssl.keyStore.password
authorizer.zookeeper.ssl.trustStore.location
authorizer.zookeeper.ssl.trustStore.password
3) The three CLI tools mentioned above will support a new --zk-tls-config-file
<String: Zookeeper TLS configuration file path>" option. The following
properties will be recognized in that file, and unrecognized properties
will be ignored to allow the possibility of pointing zk-tls-config-file at
the broker's config file.
zookeeper.client.secure (default value = false)
zookeeper.clientCnxnSocket
zookeeper.ssl.keyStore.location
zookeeper.ssl.keyStore.password
zookeeper.ssl.trustStore.location
zookeeper.ssl.trustStore.password
It will be an error for any of the last 5 values to be left unspecified if
zookeeper.client.secure is explicitly set to true.
Ron
On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rndg...@gmail.com> wrote:
> Hi everyone. Let's get this discussion going again now that Kafka 2.4
> with Zookeeper 3.5.6 is out.
>
> First, regarding the KIP number, the other KIP that was using this number
> moved to KIP 534, so KIP 515 remains the correct number for this
> discussion. I've updated the Kafka Improvement Proposal page to list this
> KIP in the 515 slot, so we're all set there.
>
> Regarding the actual issues under discussion, I think there are some
> things we should clarify.
>
> 1) It is possible to use TLS connectivity to Zookeeper from Apache Kafka
> 2.4 -- the problem is that configuration information has to be passed via
> system properties as "-D" command line options on the java invocation of
> the broker, and those are not secure (anyone with access to the box can see
> the command line used to invoke the broker); the configuration includes
> sensitive information (e.g. a keystore password), so we need a secure
> mechanism for passing the configuration values. I believe the real
> motivation for this KIP is to harden the configuration mechanism for
> Zookeeper TLS connectivity.
>
> 2) I believe the list of CLI tools that continue to use direct Zookeeper
> connectivity in a non-deprecated fashion is:
> a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> b)
> kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
>
> 3) I believe Kafka.admin.ConfigCommand presents a conundrum because it
> explicitly states in a comment that a supported use case is bootstrapping a
> Kafka cluster with encrypted passwords in Zookeeper (see
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65).
> This means it will be especially difficult to deprecate this particular
> direct Zookeeper connectivity without a different storage mechanism for
> dynamic configuration values being available (i.e. the self-managed quorum
> referred to in KIP-500).
>
> I think it would be easier and simpler to harden the Zookeeper TLS
> configuration in both Kafka and in CLI tools compared to trying to
> deprecate the direct Zookeeper connectivity in the above 3 CLI tools --
> especially when ConfigCommand has no obvious short-term path to deprecation.
>
> Regarding how to harden the TLS configuration, adding the appropriate
> configs to Kafka is an obvious choice for the brokers. Not that these
> values cannot be dynamically reconfigurable because the values would be
> required to connect to Zookeeper via TLS in the first place.
>
> Regarding how to harden the TLS configuration for the 3 remaining CLI
> tools, it would be relatively straightforward to add support for a
> "--zk-tls-config-file <String: Zookeeper TLS configuration system
> properties file path>" option to each one.
>
> Thoughts?
>
> Ron
>
> On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <pere.ur...@gmail.com>
> wrote:
>
>> True, the number is duplicated.
>>
>> do you know how can we get the problem fix? I was not aware, I missed,
>> sorry the need to add the KIP to the table.
>>
>> -- Pere
>>
>> Missatge de Jorge Esteban Quilcate Otoya <quilcate.jo...@gmail.com> del
>> dia
>> dt., 3 de set. 2019 a les 13:43:
>>
>> > Hi Pere,
>> >
>> > Have you add your KIP to the list here
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>> ?
>> >
>> > I found the KIP number assigned to another.
>> >
>> >
>> >
>> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <pere.ur...@gmail.com>
>> > wrote:
>> >
>> >> Thanks for your time Harsha,
>> >> anyone else with comments? looking forward to hearing from you.
>> >>
>> >> Stupid question: when do you move from discussion to vote?
>> >>
>> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
>> d’ag.
>> >> 2019 a les 21:59:
>> >>
>> >> > Thanks Pere. KIP looks good to me.
>> >> > -Harsha
>> >> >
>> >> >
>> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
>> >> pere.ur...@gmail.com>
>> >> > wrote:
>> >> >
>> >> >> Not really,
>> >> >> my idea is to keep the JAAS parameter, so people don't see major
>> >> >> changes. But if you pass a properties file, then this takes
>> precedence
>> >> over
>> >> >> the other, with the idea that you can do sasl as well with the
>> >> properties
>> >> >> files.
>> >> >>
>> >> >> Makes sense?
>> >> >>
>> >> >> -- Pere
>> >> >>
>> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
>> >> d’ag.
>> >> >> 2019 a les 19:00:
>> >> >>
>> >> >>> Hi Pere,
>> >> >>> Thanks for the KIP. Enabling SSL for zookeeper for Kafka
>> >> makes
>> >> >>> sense.
>> >> >>> "The changes are planned to be introduced in a compatible way, by
>> >> >>> keeping the current JAAS variable precedence."
>> >> >>> Can you elaborate a bit here. If the user configures a JAAS file
>> with
>> >> >>> Client section it will take precedence over zookeeper SSL configs?
>> >> >>>
>> >> >>> Thanks,
>> >> >>> Harsha
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
>> >> pere.ur...@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> Hi,
>> >> >>>> quick question, I saw in another mail that 2.4 release is planned
>> for
>> >> >>>> September. I think it would be really awesome to have this for
>> this
>> >> >>>> release, do you think we can make it?
>> >> >>>>
>> >> >>>> -- Pere
>> >> >>>>
>> >> >>>> Missatge de Pere Urbón Bayes <pere.ur...@gmail.com> del dia dj.,
>> 29
>> >> >>>> d’ag. 2019 a les 20:10:
>> >> >>>>
>> >> >>>> Hi,
>> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm really
>> need
>> >> >>>> to the process. Looking forward to hearing from you and learn the
>> >> best
>> >> >>>> ropes here.
>> >> >>>>
>> >> >>>> I would like to propose this KIP-515 to enable the
>> ZookeeperClients
>> >> to
>> >> >>>> take full advantage of the TLS communication in the new Zookeeper
>> >> 3.5.5.
>> >> >>>> Specially interesting it the Zookeeper Security Migration, that
>> >> without
>> >> >>>> this change will not work with TLS, disabling users to use ACLs
>> when
>> >> the
>> >> >>>> Zookeeper cluster use TLS.
>> >> >>>>
>> >> >>>> link:
>> >> >>>>
>> >> >>>>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
>> >> >>>>
>> >> >>>> Looking forward to hearing from you on this,
>> >> >>>>
>> >> >>>> /cheers
>> >> >>>>
>> >> >>>> --
>> >> >>>> Pere Urbon-Bayes
>> >> >>>> Software Architect
>> >> >>>> http://www.purbon.com
>> >> >>>> https://twitter.com/purbon
>> >> >>>> https://www.linkedin.com/in/purbon/
>> >> >>>>
>> >> >>>> --
>> >> >>>> Pere Urbon-Bayes
>> >> >>>> Software Architect
>> >> >>>> http://www.purbon.com
>> >> >>>> https://twitter.com/purbon
>> >> >>>> https://www.linkedin.com/in/purbon/
>> >> >>>>
>> >> >>>
>> >> >>>
>> >> >>
>> >> >> --
>> >> >> Pere Urbon-Bayes
>> >> >> Software Architect
>> >> >> http://www.purbon.com
>> >> >> https://twitter.com/purbon
>> >> >> https://www.linkedin.com/in/purbon/
>> >> >>
>> >> >
>> >> >
>> >>
>> >> --
>> >> Pere Urbon-Bayes
>> >> Software Architect
>> >> http://www.purbon.com
>> >> https://twitter.com/purbon
>> >> https://www.linkedin.com/in/purbon/
>> >>
>> >
>>
>> --
>> Pere Urbon-Bayes
>> Software Architect
>> http://www.purbon.com
>> https://twitter.com/purbon
>> https://www.linkedin.com/in/purbon/
>>
>
