Hi Jay.
Agreed, we're planning to try to do the compression before the producer
with 0.8 as we get back to this part, hopefully this week. I saw your other
JIRA issue about the copying in the code path so that looks promising as
well.
Thanks for all the responses, really appreciate the effort and
Cool. With respect to compression performance, we definitely see the same
thing, no debate.
Of course if you want to just compress the message payloads you can do that
now without needing much help from kafka--just pass in the compressed data.
Whether it not it will do much depends on the size of
Thanks for the responses. Additional follow-up inline.
On Fri, Aug 2, 2013 at 2:21 PM, Jay Kreps wrote:
> Great comments, answers inline!
>
> On Fri, Aug 2, 2013 at 12:28 PM, Chris Hogue wrote:
>
> > These sounds like great steps. A couple of votes and questions:
> >
> > 1. Moving serializati
Great comments, answers inline!
On Fri, Aug 2, 2013 at 12:28 PM, Chris Hogue wrote:
> These sounds like great steps. A couple of votes and questions:
>
> 1. Moving serialization out and basing it all off of byte[] for key and
> payload makes sense. Echoing a response below, we've ended up doing
I believe there are some open source C++ producer implementations. At
linkedin we have a C++ implementation. We would like to open source this if
there is interest. We would like to eventually include a C++ consumer as
well.
-Jay
On Mon, Jul 29, 2013 at 6:03 AM, Sybrandy, Casey <
casey.sybra...@
These sounds like great steps. A couple of votes and questions:
1. Moving serialization out and basing it all off of byte[] for key and
payload makes sense. Echoing a response below, we've ended up doing that in
some cases anyway, and the others do a trivial transform to bytes with an
Encoder.
2
+1 to this.
Our company use Java, Python, Node.js for most of our interactions. Our
data science team uses Python and a little Java. Most of our producers /
consumers are node.js (for RabbitMQ). Our services use mostly Java with
some node.js. Our application layers use node.js.
So if our comp
In the past there was some discussion about having a C client for non-JVM
languages. Is this still planned as well? Being able to work with Kafka from
other languages would be a great thing. Where I work, we interact with Kafka
via Java and Ruby (producer), so having an official C library tha
Jay,
This seems like a great direction. Simplifying the consumer client would
be a big win, and +1 for more native java client integration.
On the last point, regarding memory usage for buffering per partition. I
would think it could be possible to devise a dynamic queuing system, to
allow high
