Hi Jay.
Agreed, we're planning to try to do the compression before the producer
with 0.8 as we get back to this part, hopefully this week. I saw your other
JIRA issue about the copying in the code path so that looks promising as
well.
Thanks for all the responses, really appreciate the effort and
Cool. With respect to compression performance, we definitely see the same
thing, no debate.
Of course if you want to just compress the message payloads you can do that
now without needing much help from kafka--just pass in the compressed data.
Whether it not it will do much depends on the size of
Thanks for the responses. Additional follow-up inline.
On Fri, Aug 2, 2013 at 2:21 PM, Jay Kreps wrote:
> Great comments, answers inline!
>
> On Fri, Aug 2, 2013 at 12:28 PM, Chris Hogue wrote:
>
> > These sounds like great steps. A couple of votes and questions:
> >
> > 1. Moving serializati
Great comments, answers inline!
On Fri, Aug 2, 2013 at 12:28 PM, Chris Hogue wrote:
> These sounds like great steps. A couple of votes and questions:
>
> 1. Moving serialization out and basing it all off of byte[] for key and
> payload makes sense. Echoing a response below, we've ended up doing
M
> To: d...@kafka.apache.org; users@kafka.apache.org
> Subject: Client improvement discussion
>
> I sent around a wiki a few weeks back proposing a set of client
> improvements that essentially amount to a rewrite of the producer and
> consumer java clients.
>
> https://cwiki.apac
These sounds like great steps. A couple of votes and questions:
1. Moving serialization out and basing it all off of byte[] for key and
payload makes sense. Echoing a response below, we've ended up doing that in
some cases anyway, and the others do a trivial transform to bytes with an
Encoder.
2
that can be used
from within Ruby would make it easier to have the same version of the client in
Java and Ruby.
-Original Message-
From: Jay Kreps [mailto:jay.kr...@gmail.com]
Sent: Friday, July 26, 2013 3:00 PM
To: d...@kafka.apache.org; users@kafka.apache.org
Subject: Client improvement
Hey Chris,
Great questions.
1. Sync vs async. Actually I am saying the client will (1) always be async
but (2) always return a response. So
val resp = client.send(k, v) // returns immediately
resp.await() // waits for request to complete
resp.offset() // waits for request to complete (i
Hey Jay,
Reading over the wiki (and email thread). Here are some questions/comments:
"Make the producer fully async to to allow issuing sends to all brokers
simultaneously and having multiple in-flight requests simultaneously. This
will dramatically reduce the impact of latency on throughput (whi
+1 to making the API use bytes and push serialization into the client. This
is effectively what I am doing currently anyway. I implemented a generic
Encoder which just passes the bytes through.
I also like the idea of the client being written in pure Java. Interacting
with Scala code from Java isn
Jay,
This seems like a great direction. Simplifying the consumer client would
be a big win, and +1 for more native java client integration.
On the last point, regarding memory usage for buffering per partition. I
would think it could be possible to devise a dynamic queuing system, to
allow high
I sent around a wiki a few weeks back proposing a set of client
improvements that essentially amount to a rewrite of the producer and
consumer java clients.
https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite
The below discussion assumes you have read this wiki.
I started to do a li
12 matches
Mail list logo
