Hello, I am currently using kafka 3.7 in kraft mode, have cluster of 3
controllers and 5 brokers. I issued a `/opt/kafka/bin/kafka-topics.sh ...
--topic T --delete` on a topic whose sole partition had only one replica on
a broker that was at the time offline (in process of recovering). The
operation succeeded and by the time the broker got online it's possible
that the topic had gotten automatically recreated by some consumer or
producer. At that moment the broker moved the logs into a dir named
something like `topic-partition.[0-9a-f]*-stray`. Now the logs dir has
hundreds of GB in these stray directories and I am wondering what is the
safest way to clean this mess up. In this particular case I do not care for
the contents of the original topics. But I am very reluctant to simply
remove the directories manually from the underlying disk. I couldn't find a
mention in the documentation. The comment in the source code [1] does not
allude to what should be done with such stray logs. Any suggestions? Thanks
in advance.

[1]
https://github.com/apache/kafka/blob/3.7.0/core/src/main/scala/kafka/log/LogManager.scala#L1261

A side question: is it normal that kafka brokers traverse essentially all
the data stored in all partition logs upon ungraceful restart? Because it
seems that is what happened when this broker with roughly 800GB of data
did. The first 8 hours of it starting up was filled with messages such as:

    Recovering unflushed segment NNN. N/M recovered for topic-partition.
(kafka.log.LogLoader)

Furthermore since then the broker crashed again and went ro recovery and to
my surprise subjected the stray logs to the same recovery resulting in
another 8 hours long outage.

Reply via email to