[
https://issues.apache.org/jira/browse/KAFKA-9677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17053839#comment-17053839
]
Jun Rao commented on KAFKA-9677:
--------------------------------
[~apovzner] : Thanks for finding this issue! I agree with your suggested fix by
capping fetchMaxBytes.
> Low consume bandwidth quota may cause consumer not being able to fetch data
> ---------------------------------------------------------------------------
>
> Key: KAFKA-9677
> URL: https://issues.apache.org/jira/browse/KAFKA-9677
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 2.0.1, 2.1.1, 2.2.2, 2.4.0, 2.3.1
> Reporter: Anna Povzner
> Assignee: Anna Povzner
> Priority: Major
>
> When we changed quota communication with KIP-219, fetch requests get
> throttled by returning empty response with the delay in `throttle_time_ms`
> and Kafka consumer retrying again after the delay.
> With default configs, the maximum fetch size could be as big as 50MB (or 10MB
> per partition). The default broker config (1-second window, 10 full windows
> of tracked bandwidth/thread utilization usage) means that < 5MB/s consumer
> quota (per broker) may stop fetch request from ever being successful.
> Or the other way around: 1 MB/s consumer quota (per broker) means that any
> fetch request that gets >= 10MB of data (10 seconds * 1MB/second) in the
> response will never get through. From consumer point of view, the behavior
> will be: Consumer will get an empty response with throttle_time_ms > 0, Kafka
> consumer will wait for throttle time delay and then send fetch request again,
> the fetch response is still too big so broker sends another empty response
> with throttle time, and so on in never ending loop
> h3. Proposed fix
> Return less data in fetch response in this case: Cap `fetchMaxBytes` passed
> to replicaManager.fetchMessages() from KafkaApis.handleFetchRequest() to
> <tracking window> * <consume bandwidth quota>. In the example of default
> configs and 1MB/s consumer bandwidth quota, fetchMaxBytes will be 10MB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
