Manikumar, Thanks for sharing the proposal. I think there are 2 parts to this discussion -
1. Should we rewrite a REST proxy given that there is a feature-complete, open-source and actively maintained REST proxy in the community? 2. Does adding a REST proxy to Apache Kafka make us more agile and maintain the high-quality experience that Kafka users have today? For 1, I don't think there is value in giving in to the NIH syndrome and reinventing the wheel. What I'm looking for is a detailed comparison of the gaps and why those can't be improved in the REST proxy that already exists and is actively maintained. For example, we depend on zkClient and have found as well as fixed several bugs by working closely with the people who maintain zkClient. This should be possible for REST proxy as well, right? For 2, I'd like us to review our history of expanding the surface area to add more clients in the past. Here is a summary - - This doesn't materially have an impact on expanding the usability of Kafka. In my experience, REST proxy + Java clients only cover ~50% of all Kafka users, and maybe 10% of those are the ones who will use the REST proxy. The remaining 50% are non-java client users (C, python, go, node etc). - People are a lot more excited about promising to contribute while adding the surface area but not on an ongoing basis down the line. - More surface area means more work to keep things consistent. Failure to do that has, in fact, hurt the user experience. - More surface area hurts agility. We want to do a few things really well as well as be agile to be able to build on our core competency. On Sat, Oct 1, 2016 at 5:38 AM Manikumar <manikumar.re...@gmail.com> wrote: > Hi Jay, > > Thanks for your reply. > > I agree that we can not add all the clients/tools available in ecosystem > page to Kafka repo itself. But we feel REST Interface is different from > other clients/tools. Since any language that can work with HTTP can > easily integrate with this interface, Having an "official" REST > interface helps user community. This also helps us to integrate well > with external management and provisioning tools. Apache Kafka release > with Java clients + REST interface is sufficient for most of the user > deployments/requirements. This helps users to deal with less number > of distributions/builds. > > Thanks, > Manikumar > > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <j...@confluent.io> wrote: > > > Hey guys, > > > > There's already a REST interface maintained as a separate project--it's > > open source and apache licensed and actively maintained ( > > https://github.com/confluentinc/kafka-rest). What is wrong with that? > You > > mentioned that there was some compatibility concern, but compatibility > has > > to do with the consumer protocol guarantees not the repo the code is in, > > right? Not sure that concern makes sense. > > > > We could argue for adding pretty much anything and everything in the > > ecosystem page in Kafka itself but I'm not sure that would make the > project > > more agile. > > > > -Jay > > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <manikumar.re...@gmail.com> > > wrote: > > > > > Hi Kafka Devs, > > > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository. > > > > > > There are already open-source alternatives are available. But we would > > > like to add REST server that > > > many users ask for under Apache Kafka repo. Many data Infra tools comes > > up > > > with Rest Interface. > > > It is useful to have inbuilt Rest API support for Produce, Consume > > messages > > > and admin interface for > > > integrating with external management and provisioning tools.This will > > also > > > allow the maintenance of > > > REST server and adding new features makes it easy because apache > > community. > > > > > > The KIP wiki is the following: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 80%3A+Kafka+Rest+Server > > > > > > Your comments and feedback are welcome. > > > > > > Thanks, > > > Manikumar > > > > > > -- Thanks, Neha
