[
https://issues.apache.org/jira/browse/KAFKA-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679124#comment-17679124
]
hzh0425 commented on KAFKA-14642:
---------------------------------
Thanks for your reply! [~satish.duggana] [~Hangleton]
So what is your solution? Let RLMM subscribe to all topics? Have you considered
using rocksdb to store the full amount of metadata?
I'm looking forward to having the opportunity to implement Tiered-Stoarge with
you, because I am now promoting the implementation of kafka tiered-storage in
my team.
> TopicBased RemoteLogMetadataManager can't support reassign user-partitions
> --------------------------------------------------------------------------
>
> Key: KAFKA-14642
> URL: https://issues.apache.org/jira/browse/KAFKA-14642
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 3.3.1
> Reporter: hzh0425
> Assignee: hzh0425
> Priority: Blocker
>
> {*}Background{*}:
> In [KIP-405: Kafka Tiered Storage - Apache Kafka - Apache Software
> Foundation|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage],
> kafka introduced the feature of hierarchical storage.
> Also, [KAFKA-9555] Topic-based implementation for the
> RemoteLogMetadataManager - ASF JIRA (apache.org) implements the default RLMM
> - 'TopicBased-RLMM'.
> {*}Problem{*}:
> TopicBased-RLMM will only subscribe to the Partitions where the current
> Broker is Leader or Follower. If the current Broker is not the Leader or
> Follower, then RLMM will directly skip the metadata records related to these
> Partitions.
> When reassign user-partitions occurs, rlmm will subscribe to new
> user-partitions, assuming that the metadata-partition to which the new
> user-partition belongs is 'metadata-partition0', and RLMM has consumed
> 'metadata-partition0' *to offset = 100* before the reassign partition occurs,
> then {*}after reassign{*}, RMLM will *not* consume 'metadata-partition0'
> \{*}from the beginning{*}, and finally cause the metadata records related to
> the new user-partition to *be lost with offset < 100.*
> *Solution*
> Let RLMM subscribe to all user-patitions, instead of only subscribing to
> partitions where the current broker is leader or follower.
> In this way, when reassign partition occurs, RLMM will have new partition's
> metadata records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
