It sounds like you are primarily interested in optimizing the producer?

There is no way to produce data without any allocation being done and I
think getting to that would be pretty hard and lead to bad apis, but
avoiding memory allocation entirely shouldn't be necessary. Small transient
objects in java are pretty cheap to allocate and deallocate. The new Kafka
producer API that is on trunk and will be in 0.8.2 is much more disciplined
in it's usage of memory though there is still some allocation. The goal is
to avoid copying the *data* multiple times, even if we do end up creating
some small helper objects along the way (the idea is that the data may be
rather large).

If you wanted to further optimize the new producer there are two things
that could be done that would help:
1. Avoid the copy when creating the ProducerRecord instance. This could be
done by accepting a length/offset along with the key and value and making
use of this when writing to the records instance. As it is your key and
value need to be complete byte arrays.
2. Avoid the copy during request serialization. This is a little trickier.
During request serialization we need to take the records for each partition
and create a request that contains all of them. It is possible to do this
with no further recopying of data but somewhat tricky.

My recommendation would be to try the new producer api and see how that
goes. If you need to optimize further we would definitely take patches for
(1) and (2).

-Jay

On Thu, Oct 23, 2014 at 4:03 PM, Rajiv Kurian <ra...@signalfuse.com> wrote:

> I have a flyweight style protocol that I use for my messages. Thus they
> require no serialization/deserialization to be processed. The messages are
> just offset, length pairs within a ByteBuffer.
>
> Is there a producer and consumer API that forgoes allocation? I just want
> to give the kakfa producer offsets from a ByteBuffer. Similarly it would be
> ideal if I could get a ByteBuffer and offsets into it from the consumer.
> Even if I could get byte arrays (implying a copy but no decoding phase) on
> the consumer that would be great. Right now it seems to me that the only
> way to get messages from Kafka is through a message object, which implies
> Kafka allocates these messages all the time. I am willing to use the
> upcoming 0.9 API too.
>
> Thanks.
>

Reply via email to