Hi All, Not sure if this is the right channel and thread to ask, but would like to discuss a potential improvement to Java Kafka Producer.
``` Currently the Kafka Producer is able to identify unavailable partitions and avoid routing messages to them, but the definition of an unavailable partitions is - the leader of the partition is not available. >From Producer point of view, acks for sending messages can be [all, -1, 0, 1] 1. When acks is set to either 0 or 1, leader availability is good enough to determine whether we should route messages to that partition. 2. When acks is set to -1 or all, leader available doesn't mean we are able to persist messages to that partition successfully, instead, we need to make sure a. leader is available. b. at least min.insync.replicas number of replicas are available ``` To achieve 2, what we need is to carry min.insync.replicas information of a topic to the metadata response, so that Producer is able to determine if it should route messages to that partition when there's no enough replicas available and it's acks is set to -1 or all. Advantages that I can think of 1. Avoid exhausting the entire Producer cache when a partition is not available for a long time and a. retries is set to a large value b. acks is set to all 2. Avoid unnecessary network tries Not sure if this is a valid case but would like to hear any opinions. Br, Arvin
