Thanks for your help! I'm not sure how that would help me though. I'm not actually trying to decommission a Kafka broker. I would like to move all the data from one disk (log.dir) to another within the same broker while keeping the original modification time of the moved segment files. After that I would like to delete the disk, not the broker.
Kind Regards, Fares On Thu, Apr 28, 2022 at 7:05 PM lqjacklee <lqjack...@gmail.com> wrote: > The resource (https://mike.seid.io/blog/decommissiong-a-kafka-node.html) > may help you. > I have created (https://issues.apache.org/jira/browse/KAFKA-13860) to > replay the case . > > On Thu, Apr 28, 2022 at 10:33 PM Fares Oueslati <oueslati.fa...@gmail.com> > wrote: > >> Hello, >> >> I'm not sure how to report this properly but I didn't get any answer in >> the >> user mailing list. >> >> In order to remove a disk in a JBOD setup, I moved all data from one disk >> to another on every Kafka broker using kafka-reassign-partitions, then I >> went through some weird behaviour. >> Basically, the disk storage kept increasing even though there is no change >> on bytes in metric per broker. >> After investigation, I’ve seen that all segment log files in the new >> log.dir had a modification date set to the moment when the move had been >> done. >> So I guess the process applying the retention policy (log cleaner?) uses >> that timestamp to check whether the segment file should be deleted or not. >> So I ended up with a lot more data than we were supposed to store, since >> we >> are basically doubling the retention time of all the freshly moved data. >> >> This seems to me to be a buggy behavior of the command, is it possible to >> create a JIRA to track and eventually fix this? >> The only option I see to fix it is to keep the modification date before >> moving the data and applying it manually afterwards for every segment >> file, touching those files manually doesn't seem very safe imho. >> >> Thanks >> Fares Oueslati >> >
