Neha & Jay,
We did look at the open source alternatives. Our concern
is what's the patch acceptance and adding features/ bug-fixes to the
existing project under a Github (although it's licensed under Apache 2.0).
It would be great if that project made available under Apache and driven by
the community. Adding to the above, not all Kafka users are interested in
using the Java client API, they would like to have simple REST API where
they can code against using any language. I do believe this adds value to
Apache Kafka in itself.
"For 1, I don't think there is value in giving in to the NIH syndrome and
reinventing the wheel. What I'm looking for is a detailed comparison of the
gaps and why those can't be improved in the REST proxy that already exists
and is actively maintained."
We are not looking at this as NIH. What we are worried about is a project
that's not maintained in a community. So the process of accepting patches
and priorities is not clear, and it's not developed in Apache community.
Not only that, existing REST API project doesn't support new client API and
hence there is no security support either.
We don't know the timeline when that's made available. We would like to add
admin functionality into the REST API. So the Roadmap of that project is
not driven by Apache.
"This doesn't materially have an impact on expanding the usability of
Kafka. In my experience, REST proxy + Java clients only cover ~50% of all
Kafka users, and maybe 10% of those are the ones who will use the REST
proxy. The remaining 50% are non-java client users (C, python, go, node
etc)."
REST API is most often asked feature in my interactions with Kafka users.
In an organization, There will be independent teams who will write their
Kafka clients using different language libraries available today, and
there is no way to standardize this. Instead of supporting several
different client libraries users will be interested in using a REST API
server. The need for a REST API will only increase as more and more users
start using Kafka.
"More surface area means more work to keep things consistent. Failure
to do that has, in fact, hurt the user experience."
Having myriad Kafka client GitHub projects that support different languages
hurts the user experience and pushes burden to maintain these libraries.
REST API is a simple code base that uses existing java client libraries to
make life easier for the users.
Thanks,
Harsha
On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <n...@confluent.io> wrote:
> Manikumar,
>
> Thanks for sharing the proposal. I think there are 2 parts to this
> discussion -
>
> 1. Should we rewrite a REST proxy given that there is a feature-complete,
> open-source and actively maintained REST proxy in the community?
> 2. Does adding a REST proxy to Apache Kafka make us more agile and maintain
> the high-quality experience that Kafka users have today?
>
> For 1, I don't think there is value in giving in to the NIH syndrome and
> reinventing the wheel. What I'm looking for is a detailed comparison of the
> gaps and why those can't be improved in the REST proxy that already exists
> and is actively maintained. For example, we depend on zkClient and have
> found as well as fixed several bugs by working closely with the people who
> maintain zkClient. This should be possible for REST proxy as well, right?
>
> For 2, I'd like us to review our history of expanding the surface area to
> add more clients in the past. Here is a summary -
>
> - This doesn't materially have an impact on expanding the usability of
> Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> all
> Kafka users, and maybe 10% of those are the ones who will use the REST
> proxy. The remaining 50% are non-java client users (C, python, go, node
> etc).
> - People are a lot more excited about promising to contribute while
> adding the surface area but not on an ongoing basis down the line.
> - More surface area means more work to keep things consistent. Failure
> to do that has, in fact, hurt the user experience.
> - More surface area hurts agility. We want to do a few things really
> well as well as be agile to be able to build on our core competency.
>
>
> On Sat, Oct 1, 2016 at 5:38 AM Manikumar <manikumar.re...@gmail.com>
> wrote:
>
> > Hi Jay,
> >
> > Thanks for your reply.
> >
> > I agree that we can not add all the clients/tools available in ecosystem
> > page to Kafka repo itself. But we feel REST Interface is different from
> > other clients/tools. Since any language that can work with HTTP can
> > easily integrate with this interface, Having an "official" REST
> > interface helps user community. This also helps us to integrate well
> > with external management and provisioning tools. Apache Kafka release
> > with Java clients + REST interface is sufficient for most of the user
> > deployments/requirements. This helps users to deal with less number
> > of distributions/builds.
> >
> > Thanks,
> > Manikumar
> >
> >
> > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <j...@confluent.io> wrote:
> >
> > > Hey guys,
> > >
> > > There's already a REST interface maintained as a separate project--it's
> > > open source and apache licensed and actively maintained (
> > > https://github.com/confluentinc/kafka-rest). What is wrong with that?
> > You
> > > mentioned that there was some compatibility concern, but compatibility
> > has
> > > to do with the consumer protocol guarantees not the repo the code is
> in,
> > > right? Not sure that concern makes sense.
> > >
> > > We could argue for adding pretty much anything and everything in the
> > > ecosystem page in Kafka itself but I'm not sure that would make the
> > project
> > > more agile.
> > >
> > > -Jay
> > >
> > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <manikumar.re...@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Kafka Devs,
> > > >
> > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > > >
> > > > There are already open-source alternatives are available. But we
> would
> > > > like to add REST server that
> > > > many users ask for under Apache Kafka repo. Many data Infra tools
> comes
> > > up
> > > > with Rest Interface.
> > > > It is useful to have inbuilt Rest API support for Produce, Consume
> > > messages
> > > > and admin interface for
> > > > integrating with external management and provisioning tools.This will
> > > also
> > > > allow the maintenance of
> > > > REST server and adding new features makes it easy because apache
> > > community.
> > > >
> > > > The KIP wiki is the following:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 80%3A+Kafka+Rest+Server
> > > >
> > > > Your comments and feedback are welcome.
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > >
> >
> --
> Thanks,
> Neha
>
