Thanks Federico for the KIP. This feature is helpful for developers while debugging tiered storage related issues.
Even though RLMM is a pluggable interface, it is still useful to have a utility that is meant for the default/inbuilt implementation based on the internal topic. We can clarify that in the help notes and user docs. Users can still use alternatives like others suggested if they need to dump in a different format - Running the dump-logs tool with custom decoder - Running kafka-consumer.sh on the topic. ~Satish. ~Satish. On Mon, 17 Jun 2024 at 15:55, Federico Valeri <fedeval...@gmail.com> wrote: > > Hi Kamal, > > On Mon, Jun 17, 2024 at 11:44 AM Kamal Chandraprakash > <kamal.chandraprak...@gmail.com> wrote: > > > > We can use the console-consumer to read the contents of the > > `__remote_log_metadata` topic. Why are we proposing a new tool? > > > > sh kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic > > __remote_log_metadata --consumer-property exclude.internal.topics=false > > --formatter > > org.apache.kafka.server.log.remote.metadata.storage.serialization.RemoteLogMetadataSerde\$RemoteLogMetadataFormatter > > --from-beginning > > > > Thanks from bringing this up. It works fine but a running broker is > required, so it would make it inconvenient for a remote support > engineer. Also you may have to deal with client security > configuration, and it would be complicated to only dump specific > segments. I'm adding to the rejected alternative for now, but I'm open > to changes. > > > > > > > On Mon, Jun 17, 2024 at 12:53 PM Federico Valeri <fedeval...@gmail.com> > > wrote: > > > > > Hi Divij, > > > > > > On Sun, Jun 16, 2024 at 7:38 PM Divij Vaidya <divijvaidy...@gmail.com> > > > wrote: > > > > > > > > Hello Federico > > > > > > > > Please note that the topic-based RLMM is one of the possible > > > > implementations of RLMM. Hence, whatever solution we design here should: > > > 1\ > > > > be explicit that this tooling only works for topic based RLMM 2\ specify > > > > the handling of the failure mode when topic based RLMM is not being > > > > used. > > > > > > > > > > That's true, thanks for pointing out. > > > > > > > I would argue that Topic based RLMM cannot be treated the same as other > > > > internal topics. Topic based RLMM topic is an optional topic which can > > > have > > > > any possible schema (depending on plugin implementation) whereas > > > > other internal topics are always guaranteed to be present with a fixed > > > > schema. > > > > > > > > > > Right, I updated the KIP with an improved option description. > > > > > > > In light of the above statements, the rejected alternative sounds better > > > to > > > > me because: > > > > 1\ it provides the ability to dump logs for "any" RLMM implementation > > > > and > > > > not just topic based RLMM. > > > > 2\ we don't have to deal with schema evolution of topic based RLMM in > > > this > > > > tool. That responsibility will be delegated to the decoder class which > > > the > > > > operator can define using the flag "--value-decoder-class". > > > > > > > > Is there a reason that you are unable to use the rejected solution > > > > (which > > > > requires no changes) for debugging purposes? > > > > > > > > > > The rejected alternative will still be available, but I thought that > > > having a dedicated flag would make debugging easier, as I guess most > > > people will use the default RLMM implementation. I would be happy to > > > hear other opinions on this. > > > > > > > -- > > > > Divij Vaidya > > > > > > > > > > > > > > > > On Sat, Jun 15, 2024 at 4:43 PM Federico Valeri <fedeval...@gmail.com> > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I'd like to kick off a discussion for KIP-1057, that proposes to add > > > > > remote log metadata flag to the dump log tool, which is useful when > > > > > debugging. > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1057%3A+Add+remote+log+metadata+flag+to+the+dump+log+tool > > > > > > > > > > Thanks, > > > > > Fede > > > > > > > >
