Hi Alexandre, Thank you for your feedback!
> One question I would have is, what is the benefit of adding these > custom metadata in the rlmMetadata rather than letting the plugin > manage access and persistence to them? Could you please elaborate? Do I understand correctly that the idea is that the plugin will have its own storage for those custom metadata, for example a special topic? > It would be possible for a user > to use custom metadata large enough to adversely impact access to and > caching of the rlmMetadata by Kafka. Since the custom metadata is 100% under control of the RSM plugin, the risk is as big as the risk of running a third-party code (i.e. the RSM plugin). The cluster admin must make the decision if they trust it. To mitigate this risk and put it under control, the RSM plugin implementations could document what custom metadata they use and estimate their size. Best, Ivan On Mon, 17 Apr 2023 at 18:14, Alexandre Dupriez <alexandre.dupr...@gmail.com> wrote: > Hi Ivan, > > Thank you for the KIP. > > I think the KIP clearly explains the need for out-of-band metadata > authored and used by an implementation of the remote storage manager. > One question I would have is, what is the benefit of adding these > custom metadata in the rlmMetadata rather than letting the plugin > manage access and persistence to them? > > Maybe one disadvantage and potential risk with the approach proposed > in the KIP is that the rlmMetadata is not of a predefined, relatively > constant size (although corner cases with thousands of leader epochs > in the leader epoch map are possible). It would be possible for a user > to use custom metadata large enough to adversely impact access to and > caching of the rlmMetadata by Kafka. > > Thanks, > Alexandre > > Le jeu. 6 avr. 2023 à 16:03, hzh0425 <hzhka...@163.com> a écrit : > > > > I think it's a good idea as we may want to store remote segments in > different buckets > > > > > > > > | | > > hzhka...@163.com > > | > > | > > 邮箱：hzhka...@163.com > > | > > > > > > > > > > ---- 回复的原邮件 ---- > > | 发件人 | Ivan Yurchenko<ivan0yurche...@gmail.com> | > > | 日期 | 2023年04月06日 22:37 | > > | 收件人 | dev@kafka.apache.org<dev@kafka.apache.org> | > > | 抄送至 | | > > | 主题 | [DISCUSS] KIP-917: Additional custom metadata for remote log > segment | > > Hello! > > > > I would like to start the discussion thread on KIP-917: Additional custom > > metadata for remote log segment [1] > > This KIP is fairly small and proposes to add a new field to the remote > > segment metadata. > > > > Thank you! > > > > Best, > > Ivan > > > > [1] > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment >
