Hi everyone. Let's get this discussion going again now that Kafka 2.4 with
Zookeeper 3.5.6 is out.
First, regarding the KIP number, the other KIP that was using this number
moved to KIP 534, so KIP 515 remains the correct number for this
discussion. I've updated the Kafka Improvement Proposal page to list this
KIP in the 515 slot, so we're all set there.
Regarding the actual issues under discussion, I think there are some things
we should clarify.
1) It is possible to use TLS connectivity to Zookeeper from Apache Kafka
2.4 -- the problem is that configuration information has to be passed via
system properties as "-D" command line options on the java invocation of
the broker, and those are not secure (anyone with access to the box can see
the command line used to invoke the broker); the configuration includes
sensitive information (e.g. a keystore password), so we need a secure
mechanism for passing the configuration values. I believe the real
motivation for this KIP is to harden the configuration mechanism for
Zookeeper TLS connectivity.
2) I believe the list of CLI tools that continue to use direct Zookeeper
connectivity in a non-deprecated fashion is:
a) zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
b)
kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
3) I believe Kafka.admin.ConfigCommand presents a conundrum because it
explicitly states in a comment that a supported use case is bootstrapping a
Kafka cluster with encrypted passwords in Zookeeper (see
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65).
This means it will be especially difficult to deprecate this particular
direct Zookeeper connectivity without a different storage mechanism for
dynamic configuration values being available (i.e. the self-managed quorum
referred to in KIP-500).
I think it would be easier and simpler to harden the Zookeeper TLS
configuration in both Kafka and in CLI tools compared to trying to
deprecate the direct Zookeeper connectivity in the above 3 CLI tools --
especially when ConfigCommand has no obvious short-term path to deprecation.
Regarding how to harden the TLS configuration, adding the appropriate
configs to Kafka is an obvious choice for the brokers. Not that these
values cannot be dynamically reconfigurable because the values would be
required to connect to Zookeeper via TLS in the first place.
Regarding how to harden the TLS configuration for the 3 remaining CLI
tools, it would be relatively straightforward to add support for a
"--zk-tls-config-file <String: Zookeeper TLS configuration system
properties file path>" option to each one.
Thoughts?
Ron
On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes <pere.ur...@gmail.com>
wrote:
> True, the number is duplicated.
>
> do you know how can we get the problem fix? I was not aware, I missed,
> sorry the need to add the KIP to the table.
>
> -- Pere
>
> Missatge de Jorge Esteban Quilcate Otoya <quilcate.jo...@gmail.com> del
> dia
> dt., 3 de set. 2019 a les 13:43:
>
> > Hi Pere,
> >
> > Have you add your KIP to the list here
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> ?
> >
> > I found the KIP number assigned to another.
> >
> >
> >
> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes <pere.ur...@gmail.com>
> > wrote:
> >
> >> Thanks for your time Harsha,
> >> anyone else with comments? looking forward to hearing from you.
> >>
> >> Stupid question: when do you move from discussion to vote?
> >>
> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30 d’ag.
> >> 2019 a les 21:59:
> >>
> >> > Thanks Pere. KIP looks good to me.
> >> > -Harsha
> >> >
> >> >
> >> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
> >> pere.ur...@gmail.com>
> >> > wrote:
> >> >
> >> >> Not really,
> >> >> my idea is to keep the JAAS parameter, so people don't see major
> >> >> changes. But if you pass a properties file, then this takes
> precedence
> >> over
> >> >> the other, with the idea that you can do sasl as well with the
> >> properties
> >> >> files.
> >> >>
> >> >> Makes sense?
> >> >>
> >> >> -- Pere
> >> >>
> >> >> Missatge de Harsha Chintalapani <ka...@harsha.io> del dia dv., 30
> >> d’ag.
> >> >> 2019 a les 19:00:
> >> >>
> >> >>> Hi Pere,
> >> >>> Thanks for the KIP. Enabling SSL for zookeeper for Kafka
> >> makes
> >> >>> sense.
> >> >>> "The changes are planned to be introduced in a compatible way, by
> >> >>> keeping the current JAAS variable precedence."
> >> >>> Can you elaborate a bit here. If the user configures a JAAS file
> with
> >> >>> Client section it will take precedence over zookeeper SSL configs?
> >> >>>
> >> >>> Thanks,
> >> >>> Harsha
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
> >> pere.ur...@gmail.com>
> >> >>> wrote:
> >> >>>
> >> >>>> Hi,
> >> >>>> quick question, I saw in another mail that 2.4 release is planned
> for
> >> >>>> September. I think it would be really awesome to have this for this
> >> >>>> release, do you think we can make it?
> >> >>>>
> >> >>>> -- Pere
> >> >>>>
> >> >>>> Missatge de Pere Urbón Bayes <pere.ur...@gmail.com> del dia dj.,
> 29
> >> >>>> d’ag. 2019 a les 20:10:
> >> >>>>
> >> >>>> Hi,
> >> >>>> this is my first KIP for a change in Apache Kafka, so I'm really
> need
> >> >>>> to the process. Looking forward to hearing from you and learn the
> >> best
> >> >>>> ropes here.
> >> >>>>
> >> >>>> I would like to propose this KIP-515 to enable the ZookeeperClients
> >> to
> >> >>>> take full advantage of the TLS communication in the new Zookeeper
> >> 3.5.5.
> >> >>>> Specially interesting it the Zookeeper Security Migration, that
> >> without
> >> >>>> this change will not work with TLS, disabling users to use ACLs
> when
> >> the
> >> >>>> Zookeeper cluster use TLS.
> >> >>>>
> >> >>>> link:
> >> >>>>
> >> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> >> >>>>
> >> >>>> Looking forward to hearing from you on this,
> >> >>>>
> >> >>>> /cheers
> >> >>>>
> >> >>>> --
> >> >>>> Pere Urbon-Bayes
> >> >>>> Software Architect
> >> >>>> http://www.purbon.com
> >> >>>> https://twitter.com/purbon
> >> >>>> https://www.linkedin.com/in/purbon/
> >> >>>>
> >> >>>> --
> >> >>>> Pere Urbon-Bayes
> >> >>>> Software Architect
> >> >>>> http://www.purbon.com
> >> >>>> https://twitter.com/purbon
> >> >>>> https://www.linkedin.com/in/purbon/
> >> >>>>
> >> >>>
> >> >>>
> >> >>
> >> >> --
> >> >> Pere Urbon-Bayes
> >> >> Software Architect
> >> >> http://www.purbon.com
> >> >> https://twitter.com/purbon
> >> >> https://www.linkedin.com/in/purbon/
> >> >>
> >> >
> >> >
> >>
> >> --
> >> Pere Urbon-Bayes
> >> Software Architect
> >> http://www.purbon.com
> >> https://twitter.com/purbon
> >> https://www.linkedin.com/in/purbon/
> >>
> >
>
> --
> Pere Urbon-Bayes
> Software Architect
> http://www.purbon.com
> https://twitter.com/purbon
> https://www.linkedin.com/in/purbon/
>
