Hi mates, I’m trying to configure my job to retain checkpoints on it’s cancellation and got some troubles.
I got the following args why not to use savepoints: 1. we already have all the job state on DFS in checkpoints directory 2. I can multiply size of the state on 2, when stopping the job, because the state is already stored in checkpoints dir, and I’ll save it one more time into savepoint dir 3. creation of checkpoints is incremental and savepoints - is not, so in my case (10 Tb state) the process of savepoint creation will took too long time 4. As I know, we can rescale or job from retained checkpoints I've configured my job to retain checkpoints on job cancelation and found an interesting issue - _metadata file is removed, when job is cancelled. So, I couldn’t restore my job from the retained checkpoint, is it an expected behaviour ? If so, what is wrong ? Sincerely yours, Rinat Sharipov Software Engineer at 1DMP CORE Team email: r.shari...@cleverdata.ru <mailto:a.totma...@cleverdata.ru> mobile: +7 (925) 416-37-26 CleverDATA make your data clever
