Re: Possible Bug: kafka-reassign-partitions causing the data retention time to be reset

2022-05-02 Thread Luke Chen
Hi Fared, > So I guess the process applying the retention policy (log cleaner?) uses that timestamp to check whether the segment file should be deleted or not. So I ended up with a lot more data than we were supposed to store, since we are basically doubling the retention time of all the freshly m

Re: Possible Bug: kafka-reassign-partitions causing the data retention time to be reset

2022-05-02 Thread Fares Oueslati
Hello Lqjacklee, Is there any news on this please, especially regarding my last message? Do you think it is possible to modify the segment files manually with touch -a -m -t 203801181205.09 my_sement_file? I could keep the original modification date of the files before the move and update them af

Re: Possible Bug: kafka-reassign-partitions causing the data retention time to be reset

2022-04-28 Thread Fares Oueslati
Thanks for your help! I'm not sure how that would help me though. I'm not actually trying to decommission a Kafka broker. I would like to move all the data from one disk (log.dir) to another within the same broker while keeping the original modification time of the moved segment files. After that

Re: Possible Bug: kafka-reassign-partitions causing the data retention time to be reset

2022-04-28 Thread lqjacklee
The resource (https://mike.seid.io/blog/decommissiong-a-kafka-node.html) may help you. I have created (https://issues.apache.org/jira/browse/KAFKA-13860) to replay the case . On Thu, Apr 28, 2022 at 10:33 PM Fares Oueslati wrote: > Hello, > > I'm not sure how to report this properly but I didn't

Possible Bug: kafka-reassign-partitions causing the data retention time to be reset

2022-04-28 Thread Fares Oueslati
Hello, I'm not sure how to report this properly but I didn't get any answer in the user mailing list. In order to remove a disk in a JBOD setup, I moved all data from one disk to another on every Kafka broker using kafka-reassign-partitions, then I went through some weird behaviour. Basically, th