HI Joel,
Thanks for info we have set the fetch.message.max.bytes and
socket.receive.buffer.bytes to 2Mib that is max size of message in our
case. I have increase queued.max.
message.chunks to drain and reduce blockage. We just read and hand the
message to different threads for processing...So fe
I actually meant to say that you typically don't need to bump up the
queued chunk setting - you can profile your consumer to see if
significant time is being spent waiting on dequeuing from the chunk
queues.
If you happen to have a consumer consuming from a remote data center,
then you should cons
Thanks for info. I will have to tune the memory. What else do you
recommend for High level Consumer for optimal performance and drain as
quickly as possible with auto commit on ?
Thanks,
Bhavesh
On Tue, Nov 4, 2014 at 9:59 AM, Joel Koshy wrote:
> We used to default to 10, but two should be su
We used to default to 10, but two should be sufficient. There is
little reason to buffer more than that. If you increase it to 2000 you
will most likely run into memory issues. E.g., if your fetch size is
1MB you would enqueue 1MB*2000 chunks in each queue.
On Tue, Nov 04, 2014 at 09:05:44AM -0800
