The resource (https://mike.seid.io/blog/decommissiong-a-kafka-node.html) may help you. I have created (https://issues.apache.org/jira/browse/KAFKA-13860) to replay the case .
On Thu, Apr 28, 2022 at 10:33 PM Fares Oueslati <oueslati.fa...@gmail.com> wrote: > Hello, > > I'm not sure how to report this properly but I didn't get any answer in the > user mailing list. > > In order to remove a disk in a JBOD setup, I moved all data from one disk > to another on every Kafka broker using kafka-reassign-partitions, then I > went through some weird behaviour. > Basically, the disk storage kept increasing even though there is no change > on bytes in metric per broker. > After investigation, I’ve seen that all segment log files in the new > log.dir had a modification date set to the moment when the move had been > done. > So I guess the process applying the retention policy (log cleaner?) uses > that timestamp to check whether the segment file should be deleted or not. > So I ended up with a lot more data than we were supposed to store, since we > are basically doubling the retention time of all the freshly moved data. > > This seems to me to be a buggy behavior of the command, is it possible to > create a JIRA to track and eventually fix this? > The only option I see to fix it is to keep the modification date before > moving the data and applying it manually afterwards for every segment > file, touching those files manually doesn't seem very safe imho. > > Thanks > Fares Oueslati >
