Hello Lqjacklee, Is there any news on this please, especially regarding my last message?
Do you think it is possible to modify the segment files manually with touch -a -m -t 203801181205.09 my_sement_file? I could keep the original modification date of the files before the move and update them afterwards. Thanks a lot! Fares On Fri, Apr 29, 2022 at 3:24 AM Fares Oueslati <oueslati.fa...@gmail.com> wrote: > Thanks for your help! > > I'm not sure how that would help me though. I'm not actually trying to > decommission a Kafka broker. > I would like to move all the data from one disk (log.dir) to another > within the same broker while keeping the original modification time of the > moved segment files. > After that I would like to delete the disk, not the broker. > > Kind Regards, > Fares > > On Thu, Apr 28, 2022 at 7:05 PM lqjacklee <lqjack...@gmail.com> wrote: > >> The resource (https://mike.seid.io/blog/decommissiong-a-kafka-node.html) >> may help you. >> I have created (https://issues.apache.org/jira/browse/KAFKA-13860) to >> replay the case . >> >> On Thu, Apr 28, 2022 at 10:33 PM Fares Oueslati <oueslati.fa...@gmail.com> >> wrote: >> >>> Hello, >>> >>> I'm not sure how to report this properly but I didn't get any answer in >>> the >>> user mailing list. >>> >>> In order to remove a disk in a JBOD setup, I moved all data from one disk >>> to another on every Kafka broker using kafka-reassign-partitions, then I >>> went through some weird behaviour. >>> Basically, the disk storage kept increasing even though there is no >>> change >>> on bytes in metric per broker. >>> After investigation, I’ve seen that all segment log files in the new >>> log.dir had a modification date set to the moment when the move had been >>> done. >>> So I guess the process applying the retention policy (log cleaner?) uses >>> that timestamp to check whether the segment file should be deleted or >>> not. >>> So I ended up with a lot more data than we were supposed to store, since >>> we >>> are basically doubling the retention time of all the freshly moved data. >>> >>> This seems to me to be a buggy behavior of the command, is it possible to >>> create a JIRA to track and eventually fix this? >>> The only option I see to fix it is to keep the modification date before >>> moving the data and applying it manually afterwards for every segment >>> file, touching those files manually doesn't seem very safe imho. >>> >>> Thanks >>> Fares Oueslati >>> >>
