Few other concerns that were raised in the previous discussion were around
the challenges both to maintainers and users in making this API pluggable
and how does making the interface pluggable aligns with future goals for
the project. At the time this was difficult to discuss because there wasn't
a concrete proposal. I want to discuss these points in the context of this
specific proposal:
1. Problem: Pluggable APIs mean larger surface testing area and multiple
implementations to cover.
In this case: At the time, the Kafka project didn't have much
experience with pluggable APIs and components, so the concerns were very
valid. Right now Kafka has multiple pluggable components - Connectors,
converters, transformations, authentication protocols, authorization
database, coordination protocol, serializers, etc. I think that as a
community we gotten better at testing the interface, testing the very few
implementations that are included in Apache Kafka itself and allowing the
community to innovate and validate outside of the Kafka project. I don't
recall major issues either from lack of testing or from usability
perspective.
2. Problem: Users don't want to choose a consensus implementation, they
just don't want ZK.
In this case: I agree that users don't actually want to spend time
choosing consensus implementation and a simpler deployment model would
serve them better. IMO, if Apache Kafka ships with our well-tested ZK
implementation, 99% of the users will choose to use that (a vast majority
uses our less-than-amazing authorization plugin), and the few that really
need something else for whatever reason, will be able to get what they
need. As Jake said, we need to face the fact that development trajectory of
ZK isn't amazing at the moment, that it is lacking features our users need
(SSL) and it will be good to allow the user community to explore
alternatives.
3. Problem: Why got to the effort of refactoring if we know we want to get
rid of ZK.
In this case: This change isn't huge, it doesn't rewrite large portions
of Kafka and it does not make the future direction any more difficult. If
in 2 weeks or 2 month or 2 years we'll have a ZK-less solution, applying it
on Kafka with this KIP isn't any more challenging than applying it to Kafka
without this KIP. It is a step in an orthogonal direction, but not opposite
direction. I think that letting the perfect become the enemy of the good is
a repeated failure mode in this community. Can we discuss whether this
proposal is good even if there is a more complex proposal that may be
better? As long as they don't conflict?
Gwen
On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint <molnarcsi...@gmail.com>
wrote:
> Thanks, for the feedback.
>
> Developing an internal consensus service inside Kafka would require a team
> dedicated to this task.
> We second what Flavio said in
> https://lists.apache.org/thread.html/24ae56e073104c4531cf64f7a1f1c0
> a84f895d139d334c88e9fe6028@1449008733@%3Cdev.kafka.apache.org%3E
> that
> getting an implementation which really works and is maintainable is
> difficult.
> We think that Kafka being able to use another widely used consensus system
> beside Zookeeper its a safer and workable solution.
> It will be easier for users to use a consensus system with Kafka that they
> are already familiar with.
>
>
> The implementation found here:
> https://github.com/banzaicloud/apache-kafka-on-
> k8s/tree/kafka-on-etcd/core/src/main/scala/kafka/etcd
> is a first version of enabling Etcd in Kafka.
> This implementation hooked in Etcd with a slight change in the existing
> interfaces. While this implementation works its far from being complete.
> Ideally existing interfaces should be reworked to abstract out the used
> consensus system.
> We opened this KIP to start a discussion and the community to have a look
> at the initial implementation and receive feedback if this initiative is
> viable.
>
> Balint
>
> 2018-03-29 11:23 GMT+02:00 Jakub Scholz <ja...@scholz.cz>:
>
> > I can understand the concerns about the plugability of Zookeeper/Etcd. It
> > would not be good for Kafka community if it splits into several groups
> > using different implementations.
> >
> > On the other hand, Zookeeper development seems to be a bit stalled. So
> > maybe there should be at least a discussion whether it makes sense to
> > replace Zookeeper with something like Etcd or not.
> >
> > JAkub
> >
> > On Wed, Mar 28, 2018 at 6:18 PM, Molnár Bálint <molnarcsi...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
> > >
> > > Here is the link to the KIP:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> > >
> > > Looking forward to the discussion.
> > >
> > > Thanks,
> > > Balint
> > >
> >
>
--
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
<http://www.confluent.io/blog>
