Arvin, thanks for your email. This is definitely the right channel. I am personally not familiar enough with the producer code, but what you say makes sense to me from a high level.
Maybe it would be best if you would file a Jira to improve the producer accordingly? I guess, this change would require a KIP. Of course, if you are interested, feel free to pick it up yourself. -Matthias On 6/28/20 8:53 AM, Arvin Zheng wrote: > Hi All, > > Not sure if this is the right channel and thread to ask, but would like to > discuss a potential improvement to Java Kafka Producer. > > ``` > Currently the Kafka Producer is able to identify unavailable partitions and > avoid routing messages to them, but the definition of an unavailable > partitions is - the leader of the partition is not available. > From Producer point of view, acks for sending messages can be [all, -1, 0, > 1] > 1. When acks is set to either 0 or 1, leader availability is good enough to > determine whether we should route messages to that partition. > 2. When acks is set to -1 or all, leader available doesn't mean we are able > to persist messages to that partition successfully, instead, we need to > make sure > a. leader is available. > b. at least min.insync.replicas number of replicas are available > ``` > > To achieve 2, what we need is to carry min.insync.replicas information of a > topic to the metadata response, so that Producer is able to determine if it > should route messages to that partition when there's no enough replicas > available and it's acks is set to -1 or all. > > Advantages that I can think of > 1. Avoid exhausting the entire Producer cache when a partition is not > available for a long time and > a. retries is set to a large value > b. acks is set to all > 2. Avoid unnecessary network tries > > Not sure if this is a valid case but would like to hear any opinions. > > Br, > Arvin >
signature.asc
Description: OpenPGP digital signature
