Hi Alexey, thanks for reaching out to the community. I have a question: What do you mean by "the shared subfolder still grows"? As far as I understand, the shared folder contains the state of incremental checkpoints. If you cancel the corresponding job and start a new job from one of the retained incremental checkpoints, it is required for the shared folder of the previous job to be still around since it contains the state. The new job would then create its own shared subfolder. Any new incremental checkpoints will write their state into the new job's shared subfolder while still relying on shared state of the previous job for older data. The RocksDB Backend is in charge of consolidating the incremental state.
Hence, you should be careful with removing the shared folder in case you're planning to restart the job later on. I'm adding Seth to this thread. He might have more insights and/or correct my limited knowledge of the incremental checkpoint process. Best, Matthias On Wed, Aug 25, 2021 at 1:39 AM Alexey Trenikhun <yen...@msn.com> wrote: > Hello, > I use incremental checkpoints, not externalized, should content of > checkpoint/.../shared be removed when I cancel job (or cancel with > savepoint). Looks like in our case shared continutes to grow... > > Thanks, > Alexey >
