Nandini Singhal created KAFKA-20019:
---------------------------------------
Summary: Fetch responses can exceed maxBytes when mixing local and
tiered storage partitions
Key: KAFKA-20019
URL: https://issues.apache.org/jira/browse/KAFKA-20019
Project: Kafka
Issue Type: Bug
Reporter: Nandini Singhal
When a fetch request spans multiple partitions with a mix of local and remote
(tiered storage) data, the broker can return more than `fetch.max.bytes` in a
single response. For example, with `fetch.max.bytes=1MB`:
- If partition 1 returns 1 MB from local storage
- And partition 2 returns 1 MB from tiered storage
- The total response can be 2 MB, exceeding the configured limit
referring to the code in
[https://github.com/apache/kafka/blob/bb300ef757d8501b58f6923759a48f15753ff69b/core/src/main/scala/kafka/server/ReplicaManager.scala#L1887-L1903]
A tiered storage fetch has a placeholder {{readResult}} which has
{{{}readResult.info.sizeInBytes=0{}}}.
This can cause consumer-side issues:
If consumers aren't configured to accept payloads this large, Unexpected large
allocations on consumer side.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
