[
https://issues.apache.org/jira/browse/KAFKA-15777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamal Chandraprakash updated KAFKA-15777:
-----------------------------------------
Description:
A consumer can configure the amount of local bytes to read from each partition
in the FETCH request.
{{max.fetch.bytes}} = 50 MB
{{max.partition.fetch.bytes}} = 1 MB
Similar to this, the consumer should be able to configure
{{remote.max.partition.fetch.bytes}}. The default value will be 1 MB.
While handling the {{FETCH}} request, if we encounter a partition to read data
from remote storage, then rest of the partitions in the request are ignored.
Essentially, we are serving only 1 MB of remote data per FETCH request when all
the partitions in the request are to be served from the remote storage.
Providing one more configuration to the client help the user to tune the values
depending on their storage plugin. The user might want to optimize the number
of calls to remote storage vs amount of bytes returned back to the client in
the FETCH response.
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L1454]
was:
A consumer can configure the amount of local bytes to read from each partition
in the FETCH request.
{{max.fetch.bytes}} = 50 MB
{{max.partition.fetch.bytes}} = 1 MB
Similar to this, the consumer should be able to configure
{{max.remote.partition.fetch.bytes}} = 4 MB.
While handling the {{FETCH}} request, if we encounter a partition to read data
from remote storage, then rest of the partitions in the request are ignored.
Essentially, we are serving only 1 MB of remote data per FETCH request when all
the partitions in the request are to be served from the remote storage.
Providing one more configuration to the client help the user to tune the values
depending on their storage plugin. The user might want to optimise the number
of calls to remote storage vs amount of bytes returned back to the client in
the FETCH response.
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L1454]
> Configurable remote fetch bytes per partition from Consumer
> -----------------------------------------------------------
>
> Key: KAFKA-15777
> URL: https://issues.apache.org/jira/browse/KAFKA-15777
> Project: Kafka
> Issue Type: Task
> Reporter: Kamal Chandraprakash
> Assignee: Kamal Chandraprakash
> Priority: Major
> Labels: kip
>
> A consumer can configure the amount of local bytes to read from each
> partition in the FETCH request.
> {{max.fetch.bytes}} = 50 MB
> {{max.partition.fetch.bytes}} = 1 MB
> Similar to this, the consumer should be able to configure
> {{remote.max.partition.fetch.bytes}}. The default value will be 1 MB.
> While handling the {{FETCH}} request, if we encounter a partition to read
> data from remote storage, then rest of the partitions in the request are
> ignored. Essentially, we are serving only 1 MB of remote data per FETCH
> request when all the partitions in the request are to be served from the
> remote storage.
> Providing one more configuration to the client help the user to tune the
> values depending on their storage plugin. The user might want to optimize the
> number of calls to remote storage vs amount of bytes returned back to the
> client in the FETCH response.
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L1454]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
