Harsha, You seem to be saying that your only two options are to fork or duplicate the existing REST project or add it to Kafka to be able to contribute. I don't think those are the only two options. The other option is to contribute to the existing successful project--which is Apache licensed and getting active contribution today. You always have the power to fork/duplicate later if you don't like the governance/community/direction. Saying that you have to do this proactively doesn't really make sense.
-Jay On Thu, Oct 20, 2016 at 2:27 PM, Harsha Chintalapani <ka...@harsha.io> wrote: > Jay, > REST API is something every user is in need of. If the argument is to > clone and write your API, this will do a disservice to the users as they > now have to choose one vs. others instead of keeping one API that is > supported in Kafka community. > > "Pre-emptively re-creating another > REST layer when it seems like we all quite agree on what needs to be done > and we have an existing code base for HTTP/Kafka access that is heavily > used in production seems quite silly." > Exactly our point. Why can't we develop this in Apache Kafka > community? Instead of us open sourcing another GitHub project and creating > a divide in users and another version of API. Let's build this in Kafka > Community and use the governance model that is proven to provide vendor > free user driven consensus features. The argument that is adding this REST > server to Kafka will affect the agility of the project doesn't mak sense. > > It looks like your argument is either we develop all these small tools or > none at all. We as a community need to look at supporting critical > tools/API. Instead of dividing this project into individual external > communities. We should build this as part of Kafka which best serves the > needs of users. > The Streams and Connect projects that were pushed into Kafka could > have been left in their own Github projects based on your arguments. What > about the REST API is so different that such that it should stay out of the > Kafka project? From my experience, more users are asking for the REST API. > > Thanks, > Harsha > > On Sun, Oct 16, 2016 at 5:19 PM Jungtaek Lim <kabh...@gmail.com> wrote: > > > I guess no one doubts its power on REST server or even UI. I understand > the > > difficulty to add a module to project, but it's maximized when there is > > less support expected hence maintenance issue is likely to rise, and IMHO > > this seems to be not the case. > > > > There're also pain points when project doesn't maintain features and > > delegates to ecosystem. Based on some points (last commit date, pull > > request open and closed, and contributor graph), kafka-manager seems to > > have similar activity to kafka-rest, but it doesn't show any responses > for > > pull request supporting Kafka 0.10.0 even though numerous users leave > > comments wish to support. What Kafka community can do for that project to > > follow up? Nothing but just persuading by leaving comments hoping that > will > > be merged. (or finally come up another implementation) Kafka project > keeps > > agile but in point of whole ecosystem it can be less agile. > > > > Yes decisions and roadmap of the project are driven by PMCs and I think > > it's valid right. But we also imagine ASF projects as driven by community > > aspect, though it's alike to ideal world. KIP makes innovation on > adopting > > new feature transparently, which makes many developers inspiring and > > adopting it to their projects. Hopes that Kafka community continuously > > drives the transparency model among the ASF projects, and beyond. > > > > - Jungtaek Lim (HeartSaVioR) > > > > 2016년 10월 17일 (월) 오전 7:56, Jay Kreps <j...@confluent.io>님이 작성: > > > > Hey Nacho, > > > > Yeah, I think it is definitely a call we have to make case by case. We > have > > some experience with this: originally we attempted to maintain things > like > > non-java clients, a hadoop connector, etc all in the main project. The > > difficulty of that lead us to the current federated approach. In terms of > > what is included now, yes, I agree you could potentially have even less > > included. > > > > -Jay > > > > On Wed, Oct 12, 2016 at 11:37 AM, Nacho Solis > <nso...@linkedin.com.invalid > > > > > wrote: > > > > > What is the criteria for keeping things in and out of Kafka, what code > > goes > > > in or out and what is part of the architecture or not? > > > > > > The discussion of what goes into a project and what stays out is an > > always > > > evolving question. Different projects treat this in different ways. > > > > > > Let me paint 2 extremes. On one side, you have a single monolithic > > project > > > that brings everything in one tent. On the other side you have the > many > > > modules approach. From what I've learned, Kafka falls in the middle. > > > Because of this, the question is bound to come up with respect to the > > > criteria used to bring something into the fold. > > > > > > I'll be the first to point out that the distinction between modules, > > > architecture, software, repositories, governance and community are > > blurry. > > > Not to mention that many things are how they are for historical > reasons. > > > > > > I, personally, can't understand why we would not have REST as part of > the > > > main Kafka project given that a lot of people use it and we include > many > > > things with the current distribution. What many things you may ask? > > Well, > > > if we took the modular approach Kafka is a mixture of components, > here's > > > the first 4 that come to mind: > > > 1. The Kafka protocol > > > 2. The Kafka java libraries > > > 3. The Kafka broker > > > 4. The Kafka stream framework > > > 5. Kafka Connect > > > 6. MirrorMaker > > > > > > All of these could be separate products. You should be able to evolve > > each > > > one independently. Even if they have dependencies on each other, you > > could > > > potentially replace one part. > > > > > > The choice of keeping them all in a single repository, with a single > > > distribution, under the same governance and community, brings a number > of > > > trade offs. It's easy to keep things coherent for example. There is > > less > > > of a need to rely on inherent versioning and compatibility (which we > end > > up > > > providing anyway because of the way people usually deploy kafka). We > all > > > focus our efforts on a single code base. > > > > > > The downside is that it's harder to remove modules that are old or > > unused. > > > Modules that are only used by a small subset of the community will have > > an > > > impact on the rest of the community. It mixes incentives of what > people > > > want to work on and what holds them back. We also need to decide what > > > belongs in the blessed bundle and what doesnt. > > > > > > So, my question boils down to, what criteria is used for bringing stuff > > in. > > > > > > If we have Streams and MirrorMaker and Connect in there, why not have > > REST? > > > Specially if there is more than one person/group willing to work on it? > > > Alternatively, if REST is not included because it's not used by all, > then > > > why not remove Streams, Connect and MirrorMaker since they're > definitely > > > not used by all? I realize I say this even though at LinkedIn we have a > > > REST setup of our own, just speaking from a community perspective. > > > > > > Nacho > > > > > > > > > (I'm relatively new and I haven't read all of the mail archive, so I'm > > sure > > > this has been brought up before, but I decided to chime in anyway) > > > > > > On Wed, Oct 12, 2016 at 8:03 AM, Jay Kreps <j...@confluent.io> wrote: > > > > > > > I think the questions around governance make sense, I think we should > > > > really clarify that to make the process more clear so it can be fully > > > > inclusive. > > > > > > > > The idea that we should not collaborate on what is there now, though, > > > > because in the future we might disagree about direction does not > really > > > > make sense to me. If in the future we disagree, that is the beauty of > > > open > > > > source, you can always fork off a copy of the code and start an > > > independent > > > > project either in Apache or elsewhere. Pre-emptively re-creating > > another > > > > REST layer when it seems like we all quite agree on what needs to be > > done > > > > and we have an existing code base for HTTP/kafka access that is > heavily > > > > used in production seems quite silly. > > > > > > > > Let me give some background on how I at least think about these > things. > > > > I've participated in open source projects out of LinkedIn via github > as > > > > well as via the ASF. I don't think there is a "right" answer to how > to > > do > > > > these but rather some tradeoffs. We thought about this quite a lot in > > the > > > > context of Kafka based on the experience with the Hadoop ecosystem as > > > well > > > > as from other open source communities. > > > > > > > > There is a rich ecosystem around Kafka. Many of the projects are > quite > > > > small--single clients or tools that do a single thing well--and > almost > > > none > > > > of them are top level apache projects. I don't think trying to force > > each > > > > of these to turn into independent Apache projects is necessarily the > > best > > > > thing for the ecosystem. > > > > > > > > My observation of how this can go wrong is really what I think has > > > happened > > > > to the Hadoop ecosystem. There you see quite a zoo of projects which > > all > > > > drift apart and don't quite work together well. Coordinating even > > simple > > > > changes and standardization across these is exceptionally difficult. > > The > > > > result is a bit of a mess for users--the pieces just don't really > come > > > > together very well. This makes sense for independent infrastructure > > > systems > > > > (Kudu vs HDFS) but I'm not at all convinced that doing this for every > > > > little tool or helper library has lead to a desirable state. I think > > the > > > > mode of operating where the Hadoop vendors spawn off a few new Apache > > > > projects for each new product initiative, especially since often that > > > > project is only valued by that vendor (and the other vendor has a > > > different > > > > competing Apache project) doesn't necessarily do a better job at > > > producing > > > > high quality communities or high quality software. > > > > > > > > These tools/connects/clients/proxies and other integration pieces can > > > take > > > > many forms, but my take of what makes one of these things good is > that > > it > > > > remains simple, does its one thing well, and cleaves as closely as > > > possible > > > > to the conventions for Kafka itself--i.e. doesn't invent new ways of > > > > monitoring, configuring, etc. For the tools we've contributed we've > > tried > > > > really hard to make them consistent with Kafka as well as with each > > other > > > > in how testing, configuration, monitoring, etc works. > > > > > > > > I think what Apache does superbly well is create a community for > > > managing a > > > > large infrastructure layer like Kafka in a vendor independent way. > What > > I > > > > think is less successful is attempting to form full and independent > > > apache > > > > communities around very simple single purpose tools, especially if > you > > > hope > > > > for these to come together into a cohesive toolset across multiple > such > > > > tools. Much of what Apache does--create a collective decision making > > > > process for resolving disagreement, help to trademark and protect the > > > marks > > > > of the project, etc just isn't that relevant for simple > single-purpose > > > > tools. > > > > > > > > So my take is there are a couple of options: > > > > > > > > 1. We can try to put all the small tools into the Apache Project. > I > > > > think this is not the right approach as there is simply too many > of > > > > them, > > > > many in different languages, serving different protocols, > > integrating > > > > with > > > > particular systems, and a single community can't effectively > > maintain > > > > them > > > > all. Doing this would significantly slow the progress of the Kafka > > > > project. > > > > As a protocol for messaging, I don't really see a case for > including > > > > REST > > > > but not MQTT or AMQP which are technically much better suited to > > > > messaging > > > > and both are widely used for that. > > > > 2. We can treat ecosystem projects that aren't top level Apache > > > projects > > > > as invalid and try to recreate them all as Apache projects. > > Honestly, > > > > though, if you go to the Kafka ecosystem page virtually none of > the > > > most > > > > popular add-ons to Kafka are Apache projects. The most successful > > > > things in > > > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a number of > > > other > > > > clients, as well as the existing REST layer have succeeded at > > > developing > > > > communities that actively contribute and use these pieces and I > > don't > > > > know > > > > that that is a bad thing unless that community proves to be > > > uninclusive, > > > > unresponsive, or goes in a bad technical direction--and those are > > > > failure > > > > modes that all open source efforts face. > > > > 3. We can do what I think makes the most sense and try to work > with > > > the > > > > projects that exist in the ecosystem and if the project doesn't > have > > a > > > > responsive community or wants to go in a different direction fork > or > > > > recreate that work. > > > > > > > > Of course any person can choose whatever of these options they want. > > But > > > > from my point of view, option (3) has been the path of the community > so > > > far > > > > and I think it has been quite successful. > > > > > > > > -Jay > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani < > ka...@harsha.io > > > > > > > wrote: > > > > > > > > > Neha, > > > > > "But I haven't seen any community emails or patches being submitted > > by > > > > you > > > > > guys, so I'm wondering why you are concerned about whether the > > > community > > > > is > > > > > open to accepting patches or not." > > > > > > > > > > I think you are talking about contributing patches to this > repository > > > > > right? https://github.com/confluentinc/kafka-rest . All I am > saying > > > > > the guidelines/governance model is not clear on the project and I > > guess > > > > its > > > > > driven by opening a github issue request. Its the repository owned > > by > > > > > confluent and as much I appreciate that the features we mentioned > are > > > in > > > > > the roadmap and welcoming us to contribute to the project. It > doesn't > > > > > gurantee what we want to add in the furture will be in your > roadmap. > > > > > > > > > > Hence the reason having it part of Kafka community will help a lot > as > > > > other > > > > > users can participate in the discussions. We are happy to drive > any > > > > > feature additions through KIPs which gives everyone a chance to > > > > participate > > > > > and add to the discussions. > > > > > > > > > > Thanks, > > > > > Harsha > > > > > > > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce < > > michael.pea...@ig.com> > > > > > wrote: > > > > > > > > > > > +1 > > > > > > > > > > > > I agree on the governance comments whole heartedly. > > > > > > > > > > > > Also i agree about the contribution comments made earlier in the > > > > thread, > > > > > i > > > > > > personally am less likely to spend any of mine, or give project > > time > > > > > within > > > > > > my internal projects to developers contributing to another > > commercial > > > > > > companies project even so technically open source, as then there > is > > > > that > > > > > > commercial companies interest will always prevail and essentially > > can > > > > > > always have a final vote where disagreement. Im sure they never > > > intend > > > > > to, > > > > > > but there is that true reality. This is why we have community > open > > > > source > > > > > > projects. > > > > > > > > > > > > I can find many different implementations now of a rest endpoint > on > > > > > > GitHub, BitBucket etc. Each one has their benefits and > > disadvantages > > > in > > > > > > their implementation. By making / providing one this would bring > > > > together > > > > > > these solutions, unifying those developers and also bringing the > > best > > > > of > > > > > > all. > > > > > > > > > > > > I understand the concern on the community burden > adding/supporting > > > more > > > > > > surface area for every client. But something like REST is > universal > > > and > > > > > > worthy to be owned by the community. > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > ________________________________________ > > > > > > From: Andrew Schofield <andrew_schofield_j...@outlook.com> > > > > > > Sent: Saturday, October 8, 2016 1:19 AM > > > > > > To: dev@kafka.apache.org > > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > > > > > > > There's a massive difference between the governance of Kafka and > > the > > > > > > governance of the REST proxy. > > > > > > > > > > > > In Kafka, there is a broad community of people contributing their > > > > > opinions > > > > > > about future enhancements in the form of KIPs. There's some > really > > > deep > > > > > > consideration that goes into some of the trickier KIPs. There are > > > > people > > > > > > outside Confluent deeply knowledgeable in Kafka and building the > > > > > > reputations to become committers. I get the impression that the > > > roadmap > > > > > of > > > > > > Kafka is not really community-owned (what's the big feature for > > Kafka > > > > > 0.11, > > > > > > for example), but the conveyor belt of smaller features in the > form > > > of > > > > > KIPs > > > > > > works nicely. It's a good example of open-source working well. > > > > > > > > > > > > The equivalent for the REST proxy is basically issues on GitHub. > > The > > > > > > roadmap is less clear. There's not really a community properly > > > engaged > > > > in > > > > > > the way that there is with Kafka. So, you could say that it's > clear > > > > that > > > > > > fewer people are interested, but I think the whole governance > > thing > > > > is a > > > > > > big barrier to engagement. And it's looking like it's getting out > > of > > > > > date. > > > > > > > > > > > > In technical terms, I can think of two big improvements to the > REST > > > > > proxy. > > > > > > First, it needs to use the new consumer API so that it's possible > > to > > > > > secure > > > > > > connections between the REST proxy and Kafka. I don't care too > much > > > > which > > > > > > method calls it uses actually uses to consume messages, but I do > > > care > > > > > that > > > > > > I cannot secure connections because of network security rules. > > > Second, > > > > > > there's an affinity between a consumer and the instance of the > REST > > > > proxy > > > > > > to which it first connected. Kafka itself avoids this kind of > > > affinity > > > > > for > > > > > > good reason, and in the name of availability the REST proxy > should > > > too. > > > > > > These are natural KIPs. > > > > > > > > > > > > I think it would be good to have the code for the REST proxy > > > > contributed > > > > > > to Apache Kafka so that it would be able to be developed in the > > same > > > > way. > > > > > > > > > > > > Andrew Schofield > > > > > > > > > > > > From: Suresh Srinivas <sur...@hortonworks.com> > > > > > > Sent: 07 October 2016 22:41:52 > > > > > > To: dev@kafka.apache.org > > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server > > > > > > > > > > > > ASF already gives us a clear framework and governance model for > > > > community > > > > > > development. This is already understood by the people > contributing > > to > > > > > > Apache Kafka project, and they are the same people who want to > > > > contribute > > > > > > to the REST server capability as well. Everyone is in agreement > on > > > the > > > > > > need for collaborating on this effort. So why not contribute the > > code > > > > to > > > > > > Apache Kafka. This will help avoid duplication of effort and > forks > > > that > > > > > > may crop up, hugely benefitting the user community. This will > also > > > > avoid > > > > > > having to define a process similar to ASF on a GitHub project and > > > > instead > > > > > > there is a single community with clear understanding community > > > process > > > > as > > > > > > defined in ASF. > > > > > > > > > > > > As others have said, this is an important capability for Apache > > > Kafka. > > > > It > > > > > > is worth maintaining this as a part of the project. > > > > > > > > > > > > Regards, > > > > > > Suresh > > > > > > > > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <ofir.ma...@equalum.io> wrote: > > > > > > > > > > > > >I personally think it would be quite wasteful to re-implement > the > > > REST > > > > > > >gateway just because that an actively-maintained piece of > > > > > Apache-licensed > > > > > > >software is not governed directly by the Apache Kafka > community... > > > > While > > > > > > >kafka-rest repo is owned by Confluent, the contributors > including > > > the > > > > > main > > > > > > >one are also part of the Apache Kafka community, so there is a > > > chance > > > > > to > > > > > > >work this out. > > > > > > > > > > > > > >However, there are two valid concerns here that could be > > addressed, > > > > > around > > > > > > >community and accessibility: > > > > > > >>> What we are worried about is a project > > > > > > >>> that's not maintained in a community. So the process of > > accepting > > > > > > >>>patches > > > > > > >>> and priorities is not clear, and it's not developed in Apache > > > > > > >>>community. > > > > > > >>> Not only that, existing REST API project doesn't support new > > > client > > > > > API > > > > > > >and > > > > > > >>> hence there is no security support either. > > > > > > > > > > > > > >This might be easy to fix. Maybe Confluent / kafka-rest > community > > > can > > > > > > >clarify that - what is their contribution policy, dev style, > > roadmap > > > > > etc. > > > > > > >If they want, they can make an effort to encourage participation > > > from > > > > > > >people outside Confluent (easily accept contributions, invite > > > external > > > > > > >commiters or have open dev process similar to Apache Kafka etc), > > as > > > > > there > > > > > > >is definitely seems to be some interest on the list. That might > > > clear > > > > > the > > > > > > >community concern and help kafka-rest project (but that is a > > > > calculation > > > > > > >Confluent will have to make). > > > > > > > > > > > > > >The other, independent, concern is that REST is something that > is > > > > > expected > > > > > > >to be available out of the box with Kafka. I personally don't > feel > > > > > > >strongly > > > > > > >about it (better use proper, efficient APIs from day one), > though > > it > > > > is > > > > > > >definitely way smaller than adding a stream processing engine to > > the > > > > > > >project :) > > > > > > >Again,the kafka-rest "community" could take steps to make it > even > > > > easier > > > > > > >to > > > > > > >install, configure and run kafka-rest for new users on vanilla > > > Apache > > > > > > >Kafka > > > > > > >(outside the Confluent platform), if they wish that (or welcome > > > > > > >contributions to that end), but that is up to them. > > > > > > >Finally, if after the above steps were taken there would still a > > > > strong > > > > > > >desire to include a great rest gateway with Apache Kafka, I > assume > > > the > > > > > > >community could hypothetically fork the existing kafka-rest into > > an > > > > > Apache > > > > > > >Kafka subproject and maintain it "within Apache" instead of > > > > implementing > > > > > > >it > > > > > > >from scratch (though I'm not a lawyer etc) - but I cannot > imagine > > it > > > > > > >happen > > > > > > >without Confluent blessing, and I think that is likely much less > > > > optimal > > > > > > >(pulling in other Confluent / Apache licensed dependencies) than > > > > having > > > > > a > > > > > > >separate external community around kafka-rest. > > > > > > > > > > > > > > > > > > > > >Just my two cents, > > > > > > > > > > > > > > > > > > > > >Ofir Manor > > > > > > > > > > > > > >Co-Founder & CTO | Equalum > > > > > > > > > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286> > > <+972%2054-780-1286> <+972%2054-780-1286> > > | Email: > > > > > > ofir.ma...@equalum.io > > > > > > > > > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani < > > > ka...@harsha.io > > > > > > > > > > > >wrote: > > > > > > > > > > > > > >> Neha & Jay, > > > > > > >> We did look at the open source alternatives. > > Our > > > > > > >>concern > > > > > > >> is what's the patch acceptance and adding features/ bug-fixes > to > > > the > > > > > > >> existing project under a Github (although it's licensed under > > > Apache > > > > > > >>2.0). > > > > > > >> It would be great if that project made available under Apache > > and > > > > > > >>driven by > > > > > > >> the community. Adding to the above, not all Kafka users are > > > > > interested > > > > > > >>in > > > > > > >> using the Java client API, they would like to have simple REST > > API > > > > > where > > > > > > >> they can code against using any language. I do believe this > adds > > > > value > > > > > > >>to > > > > > > >> Apache Kafka in itself. > > > > > > >> > > > > > > >> "For 1, I don't think there is value in giving in to the NIH > > > > syndrome > > > > > > >>and > > > > > > >> reinventing the wheel. What I'm looking for is a detailed > > > comparison > > > > > of > > > > > > >>the > > > > > > >> gaps and why those can't be improved in the REST proxy that > > > already > > > > > > >>exists > > > > > > >> and is actively maintained." > > > > > > >> > > > > > > >> We are not looking at this as NIH. What we are worried about > is > > a > > > > > > >>project > > > > > > >> that's not maintained in a community. So the process of > > accepting > > > > > > >>patches > > > > > > >> and priorities is not clear, and it's not developed in Apache > > > > > community. > > > > > > >> Not only that, existing REST API project doesn't support new > > > client > > > > > API > > > > > > >>and > > > > > > >> hence there is no security support either. > > > > > > >> We don't know the timeline when that's made available. We > would > > > like > > > > > to > > > > > > >>add > > > > > > >> admin functionality into the REST API. So the Roadmap of that > > > > project > > > > > is > > > > > > >> not driven by Apache. > > > > > > >> > > > > > > >> > > > > > > >> "This doesn't materially have an impact on expanding the > > usability > > > > of > > > > > > >> Kafka. In my experience, REST proxy + Java clients only > cover > > > > ~50% > > > > > of > > > > > > >> all > > > > > > >> Kafka users, and maybe 10% of those are the ones who will > use > > > the > > > > > > >>REST > > > > > > >> proxy. The remaining 50% are non-java client users (C, > > python, > > > > go, > > > > > > >>node > > > > > > >> etc)." > > > > > > >> > > > > > > >> REST API is most often asked feature in my interactions with > > Kafka > > > > > > >>users. > > > > > > >> In an organization, There will be independent teams who will > > write > > > > > their > > > > > > >> Kafka clients using different language libraries available > > today, > > > > and > > > > > > >> there is no way to standardize this. Instead of supporting > > several > > > > > > >> different client libraries users will be interested in using a > > > REST > > > > > API > > > > > > >> server. The need for a REST API will only increase as more and > > > more > > > > > > >>users > > > > > > >> start using Kafka. > > > > > > >> > > > > > > >> "More surface area means more work to keep things consistent. > > > > Failure > > > > > > >> to do that has, in fact, hurt the user experience." > > > > > > >> Having myriad Kafka client GitHub projects that support > > different > > > > > > >>languages > > > > > > >> hurts the user experience and pushes burden to maintain these > > > > > libraries. > > > > > > >> REST API is a simple code base that uses existing java client > > > > > libraries > > > > > > >>to > > > > > > >> make life easier for the users. > > > > > > >> > > > > > > >> Thanks, > > > > > > >> Harsha > > > > > > >> > > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede < > > n...@confluent.io> > > > > > > wrote: > > > > > > >> > > > > > > >> > Manikumar, > > > > > > >> > > > > > > > >> > Thanks for sharing the proposal. I think there are 2 parts > to > > > this > > > > > > >> > discussion - > > > > > > >> > > > > > > > >> > 1. Should we rewrite a REST proxy given that there is a > > > > > > >>feature-complete, > > > > > > >> > open-source and actively maintained REST proxy in the > > community? > > > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us more > agile > > > and > > > > > > >> maintain > > > > > > >> > the high-quality experience that Kafka users have today? > > > > > > >> > > > > > > > >> > For 1, I don't think there is value in giving in to the NIH > > > > syndrome > > > > > > >>and > > > > > > >> > reinventing the wheel. What I'm looking for is a detailed > > > > comparison > > > > > > >>of > > > > > > >> the > > > > > > >> > gaps and why those can't be improved in the REST proxy that > > > > already > > > > > > >> exists > > > > > > >> > and is actively maintained. For example, we depend on > zkClient > > > and > > > > > > >>have > > > > > > >> > found as well as fixed several bugs by working closely with > > the > > > > > people > > > > > > >> who > > > > > > >> > maintain zkClient. This should be possible for REST proxy as > > > well, > > > > > > >>right? > > > > > > >> > > > > > > > >> > For 2, I'd like us to review our history of expanding the > > > surface > > > > > > >>area to > > > > > > >> > add more clients in the past. Here is a summary - > > > > > > >> > > > > > > > >> > - This doesn't materially have an impact on expanding the > > > > > > >>usability of > > > > > > >> > Kafka. In my experience, REST proxy + Java clients only > > cover > > > > > ~50% > > > > > > >>of > > > > > > >> > all > > > > > > >> > Kafka users, and maybe 10% of those are the ones who will > > use > > > > the > > > > > > >>REST > > > > > > >> > proxy. The remaining 50% are non-java client users (C, > > > python, > > > > > go, > > > > > > >> node > > > > > > >> > etc). > > > > > > >> > - People are a lot more excited about promising to > > contribute > > > > > while > > > > > > >> > adding the surface area but not on an ongoing basis down > > the > > > > > line. > > > > > > >> > - More surface area means more work to keep things > > > consistent. > > > > > > >>Failure > > > > > > >> > to do that has, in fact, hurt the user experience. > > > > > > >> > - More surface area hurts agility. We want to do a few > > things > > > > > > >>really > > > > > > >> > well as well as be agile to be able to build on our core > > > > > > >>competency. > > > > > > >> > > > > > > > >> > > > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar < > > > > manikumar.re...@gmail.com > > > > > > > > > > > > >> > wrote: > > > > > > >> > > > > > > > >> > > Hi Jay, > > > > > > >> > > > > > > > > >> > > Thanks for your reply. > > > > > > >> > > > > > > > > >> > > I agree that we can not add all the clients/tools > available > > in > > > > > > >> ecosystem > > > > > > >> > > page to Kafka repo itself. But we feel REST Interface is > > > > different > > > > > > >>from > > > > > > >> > > other clients/tools. Since any language that can work with > > > HTTP > > > > > can > > > > > > >> > > easily integrate with this interface, Having an "official" > > > REST > > > > > > >> > > interface helps user community. This also helps us to > > > integrate > > > > > well > > > > > > >> > > with external management and provisioning tools. Apache > > Kafka > > > > > > >>release > > > > > > >> > > with Java clients + REST interface is sufficient for most > of > > > the > > > > > > >>user > > > > > > >> > > deployments/requirements. This helps users to deal with > less > > > > > number > > > > > > >> > > of distributions/builds. > > > > > > >> > > > > > > > > >> > > Thanks, > > > > > > >> > > Manikumar > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps < > j...@confluent.io > > > > > > > > wrote: > > > > > > >> > > > > > > > > >> > > > Hey guys, > > > > > > >> > > > > > > > > > >> > > > There's already a REST interface maintained as a > separate > > > > > > >> project--it's > > > > > > >> > > > open source and apache licensed and actively maintained > ( > > > > > > >> > > > https://github.com/confluentinc/kafka-rest). What is > > wrong > > > > with > > > > > > > > > > > > > > > > > > > > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka > > > > > > github.com > > > > > > The Kafka REST Proxy provides a RESTful interface to a Kafka > > cluster. > > > > It > > > > > > makes it easy to produce and consume messages, view the state of > > the > > > > > > cluster, and ... > > > > > > > > > > > > >> that? > > > > > > >> > > You > > > > > > >> > > > mentioned that there was some compatibility concern, but > > > > > > >> compatibility > > > > > > >> > > has > > > > > > >> > > > to do with the consumer protocol guarantees not the repo > > the > > > > > code > > > > > > >>is > > > > > > >> > in, > > > > > > >> > > > right? Not sure that concern makes sense. > > > > > > >> > > > > > > > > > >> > > > We could argue for adding pretty much anything and > > > everything > > > > in > > > > > > >>the > > > > > > >> > > > ecosystem page in Kafka itself but I'm not sure that > would > > > > make > > > > > > >>the > > > > > > >> > > project > > > > > > >> > > > more agile. > > > > > > >> > > > > > > > > > >> > > > -Jay > > > > > > >> > > > > > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar < > > > > > > >> manikumar.re...@gmail.com > > > > > > >> > > > > > > > > >> > > > wrote: > > > > > > >> > > > > > > > > > >> > > > > Hi Kafka Devs, > > > > > > >> > > > > > > > > > > >> > > > > I created KIP-80 to add Kafka REST Server to Kafka > > > > Repository. > > > > > > >> > > > > > > > > > > >> > > > > There are already open-source alternatives are > > available. > > > > But > > > > > > >>we > > > > > > >> > would > > > > > > >> > > > > like to add REST server that > > > > > > >> > > > > many users ask for under Apache Kafka repo. Many data > > > Infra > > > > > > >>tools > > > > > > >> > comes > > > > > > >> > > > up > > > > > > >> > > > > with Rest Interface. > > > > > > >> > > > > It is useful to have inbuilt Rest API support for > > Produce, > > > > > > >>Consume > > > > > > >> > > > messages > > > > > > >> > > > > and admin interface for > > > > > > >> > > > > integrating with external management and provisioning > > > > > tools.This > > > > > > >> will > > > > > > >> > > > also > > > > > > >> > > > > allow the maintenance of > > > > > > >> > > > > REST server and adding new features makes it easy > > because > > > > > apache > > > > > > >> > > > community. > > > > > > >> > > > > > > > > > > >> > > > > The KIP wiki is the following: > > > > > > >> > > > > https://cwiki.apache.org/ > confluence/display/KAFKA/KIP- > > > > > > >> > > > > 80%3A+Kafka+Rest+Server > > > > > > >> > > > > > > > > > > >> > > > > Your comments and feedback are welcome. > > > > > > >> > > > > > > > > > > >> > > > > Thanks, > > > > > > >> > > > > Manikumar > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > -- > > > > > > >> > Thanks, > > > > > > >> > Neha > > > > > > >> > > > > > > > >> > > > > > > The information contained in this email is strictly confidential > > and > > > > for > > > > > > the use of the addressee only, unless otherwise indicated. If you > > are > > > > not > > > > > > the intended recipient, please do not read, copy, use or disclose > > to > > > > > others > > > > > > this message or any attachment. Please also notify the sender by > > > > replying > > > > > > to this email or by telephone (+44(020 7896 0011) and then delete > > the > > > > > email > > > > > > and any copies of it. Opinions, conclusion (etc) that do not > relate > > > to > > > > > the > > > > > > official business of this company shall be understood as neither > > > given > > > > > nor > > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a > > company > > > > > > registered in England and Wales, company number 04008957) and IG > > > Index > > > > > > Limited (a company registered in England and Wales, company > number > > > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate > > > Hill, > > > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) > > and > > > > IG > > > > > > Index Limited (register number 114059) are authorised and > regulated > > > by > > > > > the > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Nacho (Ignacio) Solis > > > Kafka > > > nso...@linkedin.com > > > > > >
