I think the rough calculation of max memory footprint for each high level
consumer would be:
(Number Of Partitions For All Topics) * fetch.message.max.bytes *
queued.max.message.chunks + (some decompression memory cost for a message)
In your case, it would be 10 times above.
Thanks,
Jiangjie (B
Hi All,
Is there a way to calculate the amount of memory used per thread in case of
a high level consumer?
I am particularly interested in calculating the memory required by a
process running 10 high level consumer threads for 15 topics with max. file
size set to 100 MB.
Thanks,
Kris
Hi,
I found that the consumer config param fetch.message.max.bytes is set to
100 MB on the consumer and I think this is what caused the problem.
It would really be helpful if anyone can explain how much memory the
consumer (running 100 threads) is going to need for consuming 100K 1 MB
messages fr
Hi,
I was just wondering if there is any difference in the memory footprint of
a high level consumer when:
1. the consumer is live and continuously consuming messages with no backlogs
2. when the consumer is down for quite some time and needs to be brought up
to clear the backlog.
My test case w
