Hello, I am currently using kafka 3.7 in kraft mode, have cluster of 3 controllers and 5 brokers. I issued a `/opt/kafka/bin/kafka-topics.sh ... --topic T --delete` on a topic whose sole partition had only one replica on a broker that was at the time offline (in process of recovering). The operation succeeded and by the time the broker got online it's possible that the topic had gotten automatically recreated by some consumer or producer. At that moment the broker moved the logs into a dir named something like `topic-partition.[0-9a-f]*-stray`. Now the logs dir has hundreds of GB in these stray directories and I am wondering what is the safest way to clean this mess up. In this particular case I do not care for the contents of the original topics. But I am very reluctant to simply remove the directories manually from the underlying disk. I couldn't find a mention in the documentation. The comment in the source code [1] does not allude to what should be done with such stray logs. Any suggestions? Thanks in advance.
[1] https://github.com/apache/kafka/blob/3.7.0/core/src/main/scala/kafka/log/LogManager.scala#L1261 A side question: is it normal that kafka brokers traverse essentially all the data stored in all partition logs upon ungraceful restart? Because it seems that is what happened when this broker with roughly 800GB of data did. The first 8 hours of it starting up was filled with messages such as: Recovering unflushed segment NNN. N/M recovered for topic-partition. (kafka.log.LogLoader) Furthermore since then the broker crashed again and went ro recovery and to my surprise subjected the stray logs to the same recovery resulting in another 8 hours long outage.
