[
https://issues.apache.org/jira/browse/KAFKA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786456#comment-17786456
]
Divij Vaidya edited comment on KAFKA-15388 at 11/15/23 6:04 PM:
----------------------------------------------------------------
Nice work on having a reproducer Arpit! (y)
>From a criticality perspective, we are treating all compaction related bugs
>for 3.7.0 and aren't backporting to 3.6.1. Currently it's marked as blocker
>but IMO it's a blocker for TS production launch instead of a blocker for
>3.7.0. Let's fix it as part of this Jira itself. We can create new ones if we
>find more bugs on fetch path related to compaction. One request I would have
>for you is to add exhaustive unit tests for such historically compacted cases.
>I am on vacation for next one month so my responses may not be fast, but other
>folks in the community who are familiar with TS will provide code review here.
>cc: [~showuon] [~satish.duggana]
was (Author: divijvaidya):
Nice work on having a reproducer Arpit! (y)
>From a criticality perspective, we are treating all compaction related bugs
>for 3.7.0 and aren't backporting to 3.6.1. Let's fix it as part of this Jira
>itself. We can create new ones if we find more bugs on fetch path related to
>compaction. One request I would have for you is to add exhaustive unit tests
>for such historically compacted cases. I am on vacation for next one month so
>my responses may not be fast, but other folks in the community who are
>familiar with TS will provide code review here. cc: [~showuon]
>[~satish.duggana]
> Handle topics that were having compaction as retention earlier are changed to
> delete only retention policy and onboarded to tiered storage.
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-15388
> URL: https://issues.apache.org/jira/browse/KAFKA-15388
> Project: Kafka
> Issue Type: Bug
> Reporter: Satish Duggana
> Assignee: Arpit Goyal
> Priority: Blocker
> Fix For: 3.7.0
>
> Attachments: Screenshot 2023-11-15 at 3.47.54 PM.png,
> tieredtopicloglist.png
>
>
> Context: [https://github.com/apache/kafka/pull/13561#discussion_r1300055517]
>
> There are 3 paths I looked at:
> * When data is moved to remote storage (1)
> * When data is read from remote storage (2)
> * When data is deleted from remote storage (3)
> (1) Does not have a problem with compacted topics. Compacted segments are
> uploaded and their metadata claims they contain offset from the baseOffset of
> the segment until the next segment's baseOffset. There are no gaps in offsets.
> (2) Does not have a problem if a customer is querying offsets which do not
> exist within a segment, but there are offset after the queried offset within
> the same segment. *However, it does have a problem when the next available
> offset is in a subsequent segment.*
> (3) For data deleted via DeleteRecords there is no problem. For data deleted
> via retention there is no problem.
>
> *I believe the proper solution to (2) is to make tiered storage continue
> looking for the next greater offset in subsequent segments.*
> Steps to reproduce the issue:
> {code:java}
> // TODO (christo)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
