Thanks Rajiv, looking forward to your prototype.
Guozhang
On Mon, Jan 26, 2015 at 2:30 PM, Rajiv Kurian wrote:
> Hi Guozhang,
>
> I am a bit busy at work. When I get the change I'll definitely try to get a
> proof of concept going. Not the kafka protocol, but just the buffering and
> threading
Hi Guozhang,
I am a bit busy at work. When I get the change I'll definitely try to get a
proof of concept going. Not the kafka protocol, but just the buffering and
threading structures, maybe just write to another socket. I think it would
be useful just to get the queueing and buffer management go
Hi Rajiv,
Thanks for this proposal, it would be great if you can upload some
implementation patch for the CAS idea and show some memory usage / perf
differences.
Guozhang
On Sun, Dec 14, 2014 at 9:27 PM, Rajiv Kurian wrote:
> Resuscitating this thread. I've done some more experiments and profi
Resuscitating this thread. I've done some more experiments and profiling.
My messages are very tiny (currently 25 bytes) per message and creating
multiple objects per message leads to a lot of churn. The memory churn
through creation of convenience objects is more than the memory being used
by my o
Yeah this is the org.apache.kafka.producer client in the clients/ directory.
-Jay
On Fri, Oct 24, 2014 at 9:54 AM, Rajiv Kurian wrote:
> Thanks!
>
> On Fri, Oct 24, 2014 at 9:03 AM, Guozhang Wang wrote:
>
> > I think 0.8.2 already used the new producer as the standard client.
> >
> > Guozhang
Thanks!
On Fri, Oct 24, 2014 at 9:03 AM, Guozhang Wang wrote:
> I think 0.8.2 already used the new producer as the standard client.
>
> Guozhang
>
> On Fri, Oct 24, 2014 at 8:51 AM, Rajiv Kurian
> wrote:
>
> > Thanks I'll take a look at both. Just to be sure we are talking about
> > client vers
I think 0.8.2 already used the new producer as the standard client.
Guozhang
On Fri, Oct 24, 2014 at 8:51 AM, Rajiv Kurian wrote:
> Thanks I'll take a look at both. Just to be sure we are talking about
> client version 0.82 right?
>
>
>
> On Fri, Oct 24, 2014 at 8:39 AM, Guozhang Wang wrote:
>
Thanks I'll take a look at both. Just to be sure we are talking about
client version 0.82 right?
On Fri, Oct 24, 2014 at 8:39 AM, Guozhang Wang wrote:
> Rajiv,
>
> The new producer does maintain a buffer per partition, but you need to
> consider synchronizing the access to the buffer since it
Rajiv,
The new producer does maintain a buffer per partition, but you need to
consider synchronizing the access to the buffer since it can take data from
multiple caller threads. I think Jay's suggestion 1) does the same thing
for your purpose if you already have the data buffer storing your data:
I want to avoid allocations since I am using Java in a C mode. Even though
creating objects is a mere thread local pointer bump in Java, freeing them
is not so cheap and causes uncontrollable jitter. The second motivation is
to avoid copying of data. Since I have objects which really look like C
s
My use case is that though I am using Java, I want to work in as close to
zero garbage environment. All my internal data structures are based on
ByteBuffers or buffers allocated using Unsafe. Thus my objects are already
in a state where they can be transmitted without any serialization step
i.e. my
It sounds like you are primarily interested in optimizing the producer?
There is no way to produce data without any allocation being done and I
think getting to that would be pretty hard and lead to bad apis, but
avoiding memory allocation entirely shouldn't be necessary. Small transient
objects i
Rajiv,
Could you let me know your use case? Are you sending a very large file and
hence would prefer streaming manner instead of messages?
Guozhang
On Thu, Oct 23, 2014 at 4:03 PM, Rajiv Kurian wrote:
> I have a flyweight style protocol that I use for my messages. Thus they
> require no serial
I have a flyweight style protocol that I use for my messages. Thus they
require no serialization/deserialization to be processed. The messages are
just offset, length pairs within a ByteBuffer.
Is there a producer and consumer API that forgoes allocation? I just want
to give the kakfa producer off
14 matches
Mail list logo
