Ziyun Fu created KAFKA-21408:
--------------------------------
Summary: Admin client metadata refresh uses a flat
retry.backoff.ms when the response lacks the node a pending call needs
Key: KAFKA-21408
URL: https://issues.apache.org/jira/browse/KAFKA-21408
Project: Kafka
Issue Type: Improvement
Components: admin
Reporter: Ziyun Fu
When a metadata request fails, the admin client retries it with exponential
backoff and jitter, bounded by \{{retry.backoff.max.ms}}
([KafkaAdminClient.java#L989|https://github.com/apache/kafka/blob/8e65883c1255f93361bbf29efa8425e30b40c5ec/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L989]).
There is no such backoff when the metadata request succeeds but the response
doesn't contain the node that a pending call needs. Examples are a call that
targets a specific broker id (such as \{{describeConfigs}} for a broker, or
\{{describeLogDirs}}) while that broker is not in the metadata, or a call that
needs the controller when none is present. The node provider requests another
metadata update and leaves the call pending
([KafkaAdminClient.java#L822-L833|https://github.com/apache/kafka/blob/8e65883c1255f93361bbf29efa8425e30b40c5ec/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L822-L833]).
\{{AdminMetadataManager}} then waits a fixed \{{retry.backoff.ms}} before the
next attempt
([AdminMetadataManager.java#L298-L301|https://github.com/apache/kafka/blob/8e65883c1255f93361bbf29efa8425e30b40c5ec/clients/src/main/java/org/apache/kafka/clients/admin/internals/AdminMetadataManager.java#L298-L301]).
It doesn't take \{{retry.backoff.max.ms}} into account
([KafkaAdminClient.java#L583-L586|https://github.com/apache/kafka/blob/8e65883c1255f93361bbf29efa8425e30b40c5ec/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L583-L586]),
and the delay doesn't grow across attempts.
As a result, the client sends a Metadata request every \{{retry.backoff.ms}}
(100 ms by default) until the pending call times out
(\{{default.api.timeout.ms}}, 60 s by default).
The producer and consumer already back off exponentially in the equivalent
situation, where metadata responses succeed but make no progress
(\{{equivalentResponseCount}} in
[Metadata.java#L154-L168|https://github.com/apache/kafka/blob/8e65883c1255f93361bbf29efa8425e30b40c5ec/clients/src/main/java/org/apache/kafka/clients/Metadata.java#L154-L168]).
h3. Suggested change
When successive metadata responses don't change the cluster view,
\{{AdminMetadataManager}} should apply exponential backoff with jitter, bounded
by \{{retry.backoff.max.ms}}, to the next refresh. The backoff should reset
once a response makes progress. This matches the existing documentation of
\{{retry.backoff.max.ms}} and the behaviour of \{{Metadata}}, and needs no new
config.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
