On 2018/05/09 14:29:17, Molnár Bálint <molnarcsi...@gmail.com> wrote:
> Hi,
> I just rebased the Etcd implementation proposal on trunk. Pinging to see if
> anyone has feedback on my questions from my previous email.
>
> Molnár Bálint <molnarcsi...@gmail.com> ezt írta (időpont: 2018. ápr. 4.,
> Sze, 10:08):
>
> > Hi,
> > Thanks again for the feedback.
> >
> > Is there already ongoing work for having an own consensus implementation
> > within Kafka?
> > If that work haven't started yet, we think there is value in having an
> > interim solution, that allows the use of another consensus system besides
> > Zookeeper.
> >
> > We ask the community to take a look at the Etcd implementation proposal
> > we created and provide feedback on that.
> > This helps to asses rather this approach is viable at all.
> >
> > We are open to collaborate on integrating our proposed Etcd implementation
> > into any integration test system, to certify that all use cases works as
> > expected.
> >
> > Balint
> >
> > 2018-03-30 22:21 GMT+02:00 Gwen Shapira <g...@confluent.io>:
> >
> >> Hi,
> >>
> >> I had an offline discussion with Ismael and wanted to summarize the
> >> comments and questions he raised so we are all on the same page.
> >>
> >> The core issue is that this change adds a new public API. Since we already
> >> know that the goal for the next 1-2 years is to get rid of ZK completely.
> >> Do we want to go to the effort of adding (and discussing and reviewing) a
> >> new public API, knowing that it will be completely removed in a year? And
> >> since building and testing a plugin also involves effort, will anyone do
> >> it
> >> for something that is going to be temporary by design?
> >>
> >> Ismael, correct me if this isn't a fair representation of your concerns.
> >>
> >> Gwen
> >>
> >>
> >>
> >> On Thu, Mar 29, 2018 at 9:33 AM, Gwen Shapira <g...@confluent.io> wrote:
> >>
> >> > Few other concerns that were raised in the previous discussion were
> >> around
> >> > the challenges both to maintainers and users in making this API
> >> pluggable
> >> > and how does making the interface pluggable aligns with future goals for
> >> > the project. At the time this was difficult to discuss because there
> >> wasn't
> >> > a concrete proposal. I want to discuss these points in the context of
> >> this
> >> > specific proposal:
> >> >
> >> > 1. Problem: Pluggable APIs mean larger surface testing area and multiple
> >> > implementations to cover.
> >> > In this case: At the time, the Kafka project didn't have much
> >> > experience with pluggable APIs and components, so the concerns were very
> >> > valid. Right now Kafka has multiple pluggable components - Connectors,
> >> > converters, transformations, authentication protocols, authorization
> >> > database, coordination protocol, serializers, etc. I think that as a
> >> > community we gotten better at testing the interface, testing the very
> >> few
> >> > implementations that are included in Apache Kafka itself and allowing
> >> the
> >> > community to innovate and validate outside of the Kafka project. I don't
> >> > recall major issues either from lack of testing or from usability
> >> > perspective.
> >> >
> >> > 2. Problem: Users don't want to choose a consensus implementation, they
> >> > just don't want ZK.
> >> > In this case: I agree that users don't actually want to spend time
> >> > choosing consensus implementation and a simpler deployment model would
> >> > serve them better. IMO, if Apache Kafka ships with our well-tested ZK
> >> > implementation, 99% of the users will choose to use that (a vast
> >> majority
> >> > uses our less-than-amazing authorization plugin), and the few that
> >> really
> >> > need something else for whatever reason, will be able to get what they
> >> > need. As Jake said, we need to face the fact that development
> >> trajectory of
> >> > ZK isn't amazing at the moment, that it is lacking features our users
> >> need
> >> > (SSL) and it will be good to allow the user community to explore
> >> > alternatives.
> >> >
> >> > 3. Problem: Why got to the effort of refactoring if we know we want to
> >> get
> >> > rid of ZK.
> >> > In this case: This change isn't huge, it doesn't rewrite large
> >> > portions of Kafka and it does not make the future direction any more
> >> > difficult. If in 2 weeks or 2 month or 2 years we'll have a ZK-less
> >> > solution, applying it on Kafka with this KIP isn't any more challenging
> >> > than applying it to Kafka without this KIP. It is a step in an
> >> orthogonal
> >> > direction, but not opposite direction. I think that letting the perfect
> >> > become the enemy of the good is a repeated failure mode in this
> >> community.
> >> > Can we discuss whether this proposal is good even if there is a more
> >> > complex proposal that may be better? As long as they don't conflict?
> >> >
> >> > Gwen
> >> >
> >> > On Thu, Mar 29, 2018 at 8:31 AM, Molnár Bálint <molnarcsi...@gmail.com>
> >> > wrote:
> >> >
> >> >> Thanks, for the feedback.
> >> >>
> >> >> Developing an internal consensus service inside Kafka would require a
> >> team
> >> >> dedicated to this task.
> >> >> We second what Flavio said in
> >> >> https://lists.apache.org/thread.html/24ae56e073104c4531cf64f
> >> >> 7a1f1c0a84f895d139d334c88e9fe6028@1449008733@%3Cdev.kafka.apache.org
> >> %3E
> >> >> that
> >> >> getting an implementation which really works and is maintainable is
> >> >> difficult.
> >> >> We think that Kafka being able to use another widely used consensus
> >> system
> >> >> beside Zookeeper its a safer and workable solution.
> >> >> It will be easier for users to use a consensus system with Kafka that
> >> they
> >> >> are already familiar with.
> >> >>
> >> >>
> >> >> The implementation found here:
> >> >> https://github.com/banzaicloud/apache-kafka-on-k8s/tree/
> >> >> kafka-on-etcd/core/src/main/scala/kafka/etcd
> >> >> is a first version of enabling Etcd in Kafka.
> >> >> This implementation hooked in Etcd with a slight change in the existing
> >> >> interfaces. While this implementation works its far from being
> >> complete.
> >> >> Ideally existing interfaces should be reworked to abstract out the used
> >> >> consensus system.
> >> >> We opened this KIP to start a discussion and the community to have a
> >> look
> >> >> at the initial implementation and receive feedback if this initiative
> >> is
> >> >> viable.
> >> >>
> >> >> Balint
> >> >>
> >> >> 2018-03-29 11:23 GMT+02:00 Jakub Scholz <ja...@scholz.cz>:
> >> >>
> >> >> > I can understand the concerns about the plugability of
> >> Zookeeper/Etcd.
> >> >> It
> >> >> > would not be good for Kafka community if it splits into several
> >> groups
> >> >> > using different implementations.
> >> >> >
> >> >> > On the other hand, Zookeeper development seems to be a bit stalled.
> >> So
> >> >> > maybe there should be at least a discussion whether it makes sense to
> >> >> > replace Zookeeper with something like Etcd or not.
> >> >> >
> >> >> > JAkub
> >> >> >
> >> >> > On Wed, Mar 28, 2018 at 6:18 PM, Molnár Bálint <
> >> molnarcsi...@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> > > Hi all,
> >> >> > >
> >> >> > > I have created KIP-273: Kafka to support using ETCD beside
> >> Zookeeper
> >> >> > >
> >> >> > > Here is the link to the KIP:
> >> >> > >
> >> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> >> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> >> >> > >
> >> >> > > Looking forward to the discussion.
> >> >> > >
> >> >> > > Thanks,
> >> >> > > Balint
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > *Gwen Shapira*
> >> > Product Manager | Confluent
> >> > 650.450.2760 <(650)%20450-2760> | @gwenshap
> >> > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> >> > <http://www.confluent.io/blog>
> >> >
> >> >
> >>
> >>
> >> --
> >> *Gwen Shapira*
> >> Product Manager | Confluent
> >> 650.450.2760 | @gwenshap
> >> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> >> <http://www.confluent.io/blog>
> >>
> >
> >
>
Hello, Colin!
I have read the discussion on KIP-273 and want to raise this question.
Deploying Kafka to Kubernetes is a crucial problem in my current work. And
usage of ZooKeeper makes troubles, because it works unstable in container.
In this discussion you have mentioned that the goal is to get rid of it in
future releases. But since then there were no posts about it. Also, i can not
see such plan in a roadmap. Cuold you please tell us more about work made in
this stream.
Thanks,
Leonid
