Re: Ignoring state's offset when restoring checkpoints

2022-07-11 Thread Robin Cassan via user
Thanks a lot Alexander and Tzu-Li for your answers, this helps a lot!! Cheers, Robin Le ven. 8 juil. 2022 à 17:40, Tzu-Li (Gordon) Tai a écrit : > Hi Robin, > > Apart from what Alexander suggested, I think you could also try the > following first: > Let the job use a "new" Kafka source, which y

Re: Ignoring state's offset when restoring checkpoints

2022-07-08 Thread Tzu-Li (Gordon) Tai
Hi Robin, Apart from what Alexander suggested, I think you could also try the following first: Let the job use a "new" Kafka source, which you can achieve by simply assigning a different operator ID than before. If you previously did not set an ID, then previously by default it would have been a h

Re: Ignoring state's offset when restoring checkpoints

2022-07-08 Thread Alexander Fedulov
Hi Robin, you should be able to use the State Processor API to modify the operator state (sources) and override the offsets manually there. I never tried that, but I believe conceptually it should work. Best, Alexander Fedulov

Ignoring state's offset when restoring checkpoints

2022-07-08 Thread Robin Cassan via user
Hello all! We have a need where, for specific recovery cases, we would need to manually reset our Flink kafka consumer offset to a given date but have the Flink job restore its state. As I understand, restoring from a checkpoint necessarily sets the group's offset to the one that was in the checkp