[
https://issues.apache.org/jira/browse/KAFKA-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167488#comment-17167488
]
Ismael Juma commented on KAFKA-10324:
-------------------------------------
Yeah, it's usually ok to have one smaller batch per segment. And in the common
case where you are reading from the end of the log, there may not be another
segment. Scanning ahead seems ok for down conversion, but it's not clear if
it's worth it for the common case (there is a cost to scanning ahead).
> Pre-0.11 consumers can get stuck when messages are downconverted from V2
> format
> -------------------------------------------------------------------------------
>
> Key: KAFKA-10324
> URL: https://issues.apache.org/jira/browse/KAFKA-10324
> Project: Kafka
> Issue Type: Bug
> Reporter: Tommy Becker
> Priority: Major
>
> As noted in KAFKA-5443, The V2 message format preserves a batch's lastOffset
> even if that offset gets removed due to log compaction. If a pre-0.11
> consumer seeks to such an offset and issues a fetch, it will get an empty
> batch, since offsets prior to the requested one are filtered out during
> down-conversion. KAFKA-5443 added consumer-side logic to advance the fetch
> offset in this case, but this leaves old consumers unable to consume these
> topics.
> The exact behavior varies depending on consumer version. The 0.10.0.0
> consumer throws RecordTooLargeException and dies, believing that the record
> must not have been returned because it was too large. The 0.10.1.0 consumer
> simply spins fetching the same empty batch over and over.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
