I think there would be many advantages to having a REST API in Kafka, as
a core component. Not just for data services (producing/consuming
messages), but also for admin. I could imagine a Kafka web console that
showed various stats, allowed for maintenance operations, showed pretty
viz of the cluster, etc.
The REST prototype I put together using Jetty was really a simplistic
POC. In reality, more modern web techniques like chunked transfer
encoding and streaming should be used for high throughput (in, and out).
Netty would probably be a good choice for a networking library.
I'd be willing to bet that a sufficiently high performance and easy to
use REST API could put a lot of the 3rd party client libraries out of
business, so-to-speak.
To more directly answer your question, Jay, I think if something like
this is done it should be done as a top-level component in Kafka. Is
contrib even maintained/updated anymore? Hasn't Camus supplanted the
hadoop stuff?
On 7/19/13 4:24 PM, Jay Kreps wrote:
I think this is really cool and would be useful for a lot of people. David,
do you think it would really benefit from being a contrib package in kafka
versus a separately hosted github project?
Here was our experience: contrib packages tend to get added but the authors
and don't necessarily want involvement in kafka internals or necessarily
follow the mailing list to answer questions. When we take then they end up
kind of stagnating because there is overhead for the code review process,
and the main kafka committers aren't generally qualified to fully maintain
the code.
What we have seen is that external projects that host add ons do really
well. The author tends to get more credit for their work, directly responds
to questions from users, etc.
On the other hand maybe we feel this is central enough that it should just
be a top-level package in kafka. I think this makes sense for things which
are closely tied to internals, are general enough that there is only one
good way to do it, and will be very common for kafka users to want. If this
sounds better for the REST proxy the important thing is just that someone
has bandwidth to be able to support it and ensure we we document it, have
sufficient tests to keep it in sync, etc.
Let me know what you think, if the preference is to add it to kafka then I
can help with code reviews and stuff.
-Jay
On Thu, Jul 18, 2013 at 8:15 PM, David Arthur <mum...@gmail.com> wrote:
I did some work on this a while ago: https://github.com/mumrah/**
kafka/commits/rest <https://github.com/mumrah/kafka/commits/rest>
I think people would still be interested. I know I would :)
-David
On 7/18/13 6:19 PM, Yogita Bijani wrote:
Hi,
I am a newbie, I was just checking if there has been any updates on the
REST proxy.
I have tried to come up with a draft of REST API and can post it for
review and discussion if it is still an open topic.
Thanks
Yogita
Sent from my iPad
