Hi Christo and all, > Currently a long-running client refreshes their metadata from a set of brokers obtained when first contacting the cluster. If they have been “away” for too long those brokers might have all changed and upon trying to refresh the metadata the client will fail because it cannot find an available broker. What you propose is that whenever such a situation is encountered the client should try to get the new set of brokers by communicating with the bootstrap-servers again.
Yes, your understanding is correct. > To answer your question, in my opinion this behaviour should not be guarded by a configuration and should be the default once implemented. As a customer of Kafka, I cannot think of a situation where I would prefer my clients to give up if they have stale data without even trying to get the latest information. As far as I understand, the new behaviour will be entirely constrained to the client code which makes this change easier. This sounds reasonable. If we consider the current behavior as a bug, then the config is probably not needed for the "fix". It's a slight change of the client behavior and may be considered as incompatibility. However, I agree with you, it's hard to come up with a case where the current behavior is preferred over the proposed. I don't know what's the current approach to this in the community, would be great to hear some more opinions. > As a starting point can we confirm that this is indeed the current behaviour either by a reproducible manual test or by a branch with a failing unit/integration test? Yes, the reproduction is very easy and consistent. I reproduced this manually and I also have tests for both consumers and producers that fail without the fix [1] Best, Ivan. [1] https://github.com/ivanyu/kafka/blob/bd769d3eb76bf72f8aaf37553d6a1507dd7f8c55/core/src/test/scala/integration/kafka/api/RebootstrapTest.scala On Mon, 23 Jan 2023 at 12:26, Christo Lolov <christolo...@gmail.com> wrote: > Hello! > > Thank you for the KIP. I would like to summarise my understanding of the > problem in case I am wrong. > > Currently a long-running client refreshes their metadata from a set of > brokers obtained when first contacting the cluster. If they have been > “away” for too long those brokers might have all changed and upon trying to > refresh the metadata the client will fail because it cannot find an > available broker. What you propose is that whenever such a situation is > encountered the client should try to get the new set of brokers by > communicating with the bootstrap-servers again. > > If I have understood this correctly, then I agree with what is proposed as > a solution in this KIP. To answer your question, in my opinion this > behaviour should not be guarded by a configuration and should be the > default once implemented. As a customer of Kafka, I cannot think of a > situation where I would prefer my clients to give up if they have stale > data without even trying to get the latest information. As far as I > understand, the new behaviour will be entirely constrained to the client > code which makes this change easier. > > As a starting point can we confirm that this is indeed the current > behaviour either by a reproducible manual test or by a branch with a > failing unit/integration test? > > Best, > Christo > > > On 18 Jan 2023, at 12:07, Ivan Yurchenko <ivan0yurche...@gmail.com> > wrote: > > > > Hello! > > I would like to start the discussion thread on KIP-899: Allow clients to > > rebootstrap. > > This KIP proposes to allow Kafka clients to repeat the bootstrap process > > when fetching metadata if none of the known nodes are available. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+clients+to+rebootstrap > > > > A question right away: should we eventually change the default behavior > or > > it can remain configurable "forever"? The latter is proposed in the KIP. > > > > Thank you! > > > > Ivan >
