Hi Jay.
Agreed, we're planning to try to do the compression before the producer
with 0.8 as we get back to this part, hopefully this week. I saw your other
JIRA issue about the copying in the code path so that looks promising as
well.
Thanks for all the responses, really appreciate the effort and
Cool. With respect to compression performance, we definitely see the same
thing, no debate.
Of course if you want to just compress the message payloads you can do that
now without needing much help from kafka--just pass in the compressed data.
Whether it not it will do much depends on the size of
Thanks for the responses. Additional follow-up inline.
On Fri, Aug 2, 2013 at 2:21 PM, Jay Kreps wrote:
> Great comments, answers inline!
>
> On Fri, Aug 2, 2013 at 12:28 PM, Chris Hogue wrote:
>
> > These sounds like great steps. A couple of votes and questions:
> >
> > 1. Moving serializati
Great comments, answers inline!
On Fri, Aug 2, 2013 at 12:28 PM, Chris Hogue wrote:
> These sounds like great steps. A couple of votes and questions:
>
> 1. Moving serialization out and basing it all off of byte[] for key and
> payload makes sense. Echoing a response below, we've ended up doing
I believe there are some open source C++ producer implementations. At
linkedin we have a C++ implementation. We would like to open source this if
there is interest. We would like to eventually include a C++ consumer as
well.
-Jay
On Mon, Jul 29, 2013 at 6:03 AM, Sybrandy, Casey <
casey.sybra...@
These sounds like great steps. A couple of votes and questions:
1. Moving serialization out and basing it all off of byte[] for key and
payload makes sense. Echoing a response below, we've ended up doing that in
some cases anyway, and the others do a trivial transform to bytes with an
Encoder.
2
In the past there was some discussion about having a C client for non-JVM
languages. Is this still planned as well? Being able to work with Kafka from
other languages would be a great thing. Where I work, we interact with Kafka
via Java and Ruby (producer), so having an official C library tha
Hey Chris,
Great questions.
1. Sync vs async. Actually I am saying the client will (1) always be async
but (2) always return a response. So
val resp = client.send(k, v) // returns immediately
resp.await() // waits for request to complete
resp.offset() // waits for request to complete (i
Hey Jay,
Reading over the wiki (and email thread). Here are some questions/comments:
"Make the producer fully async to to allow issuing sends to all brokers
simultaneously and having multiple in-flight requests simultaneously. This
will dramatically reduce the impact of latency on throughput (whi
+1 to making the API use bytes and push serialization into the client. This
is effectively what I am doing currently anyway. I implemented a generic
Encoder which just passes the bytes through.
I also like the idea of the client being written in pure Java. Interacting
with Scala code from Java isn
Jay,
This seems like a great direction. Simplifying the consumer client would
be a big win, and +1 for more native java client integration.
On the last point, regarding memory usage for buffering per partition. I
would think it could be possible to devise a dynamic queuing system, to
allow high
11 matches
Mail list logo
