On Friday 17 July 2015 10:14 AM, Jiangjie Qin wrote:
I think the rough calculation of max memory footprint for each high level
consumer would be:
(Number Of Partitions For All Topics) * fetch.message.max.bytes *
queued.max.message.chunks + (some decompression memory cost for a message)
Is this "Number of Partitions for All Topics in the system" or "Number
of partitions for the topics that the consumer is requesting to consume
from" (for example via regex on a topic name filter)?
-Jaikiran
In your case, it would be 10 times above.
Thanks,
Jiangjie (Becket) Qin
On 7/16/15, 1:40 PM, "Kris K" <squareksc...@gmail.com> wrote:
Hi All,
Is there a way to calculate the amount of memory used per thread in case
of
a high level consumer?
I am particularly interested in calculating the memory required by a
process running 10 high level consumer threads for 15 topics with max.
file
size set to 100 MB.
Thanks,
Kris
