Hi,
I was going through this document.
https://docs.confluent.io/current/kafka/deployment.html
“ does not require setting heap sizes more than 6 GB. This will result in a
file system cache of up to 28-30 GB on a 32 GB machine.”
Can someone please put focus on above statement? Its bit unclear to me
Hi Guys,
Hope you all are doing well.
All of sudden I see very high throughput for _consumer_offsets topic. it
is around 30-40K per second. What could be the reason for such a high
rate ? Do I need to be concerned around this ?
[image: image.png]
This happens for an hour and after that it
Nag,
Technically the `DefaultPartitioner` uses Mumur2 as you can see in the
implementation code from Kafka's trunk:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java
The `RoundRobinPartitioner` should be used i
Sunil,
This has to do with Kafka's behavior of being persistent and using the
broker's filesystem as the storage mechanism for the commit log. In
modern operating systems a watermark of *85%* of the available RAM is
dedicated to page cache and therefore, with Kafka running in a machine
with *
As I understand it, the consumer can only read "committed" messages - which
I believe, if we look at internals of it, committed messages are nothing
but messages which are upto the high watermark.
*The high watermark is the offset of the last message that was successfully
copied to all of the log’s
The short answer is : yes, a consumer can only consume messages up to the
High Watermark.
The long answer is not exactly, for the following reasons:
At the partition level you have 3 major offsets that are important to the
health of the partition and accessibility from the consumer pov:
LeO (log
Thanks D C. Thanks a lot . That is quite a detailed explanation.
If I understand correctly, ( ignoring the case where producers
create transactions) - since the replica is down and never comes , the high
watermark CANNOT advance and the consumer CAN NOT read the messages which
were sent after the r
Hey Nag Y,
I’m not exactly sure if reducing the replication factor while a broker is
down would release the messages to be consumed (or at least not on all
partitions) for the simple fact that it might just remove the last replica
in the list which might not mach your unreachable broker.
Personall
Hi Nag,
In my experience running Kafka in production for 6 years, so long as the
number of replicas (and the leader is one of those replicas) in the insync
replica set (ISR) is greater than the min.insync.replica setting, the
partition leader will accept writes, and customers can read those writes
