[
https://issues.apache.org/jira/browse/KAFKA-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679014#comment-17679014
]
hzh0425 commented on KAFKA-14642:
---------------------------------
Hi, [~satishd] [~junrao] [~Hangleton] , Could you pls help to review this issue?
> TopicBased RemoteLogMetadataManager can't support reassign user-partitions
> --------------------------------------------------------------------------
>
> Key: KAFKA-14642
> URL: https://issues.apache.org/jira/browse/KAFKA-14642
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 3.3.1
> Reporter: hzh0425
> Assignee: hzh0425
> Priority: Major
>
> {*}Background{*}:
> In [KIP-405: Kafka Tiered Storage - Apache Kafka - Apache Software
> Foundation|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage],
> kafka introduced the feature of hierarchical storage.
> Also, [KAFKA-9555] Topic-based implementation for the
> RemoteLogMetadataManager - ASF JIRA (apache.org) implements the default RLMM
> - 'TopicBased-RLMM'.
> {*}Problem{*}:
> TopicBased-RLMM will only subscribe to the Partitions where the current
> Broker is Leader or Follower. If the current Broker is not the Leader or
> Follower, then RLMM will directly skip the metadata records related to these
> Partitions.
> When reassign user-partitions occurs, rlmm will subscribe to new
> user-partitions, assuming that the metadata-partition to which the new
> user-partition belongs is 'metadata-partition0', and RLMM has consumed
> 'metadata-partition0' *to offset = 100* before the reassign partition occurs,
> then {*}after reassign{*}, RMLM will *not* consume 'metadata-partition0'
> \{*}from the beginning{*}, and finally cause the metadata records related to
> the new user-partition to *be lost with offset < 100.*
> *Solution*
> Let RLMM subscribe to all user-patitions, instead of only subscribing to
> partitions where the current broker is leader or follower.
> In this way, when reassign partition occurs, RLMM will have new partition's
> metadata records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
