That's an interesting idea.
I guess we can decouple the actual state cleanup delegation from the
correctness issues. I don't see any reason why it can't be implemented
without changing notifications (for FLIP-158, however, we'll probably have
to ask "random" TMs because FLIP-158 adds state sharing
Thanks for clarifying.
Concerning the JM aborted checkpoints and state handles: I was thinking
about it the other day as well and was considering an approach like that:
The core idea is to move the cleanup from JM to TM. That solves two issues:
(1) The StateBackends / DSTL delete the artifacts t
Thanks for your reply Stephan.
Yes, there is overlap between FLIP-151 and FLIP-158 as both
address incremental state updates. However, I think that FLIP-151 on top
of FLIP-158 increases efficiency by:
1. "Squashing" the changes made to the same key. For example, if some
counter was changed 10 tim
Thanks, Roman for publishing this design.
There seems to be quite a bit of overlap with FLIP-158 (generalized
incremental checkpoints).
I would go with +1 to the effort if it is a pretty self-contained and
closed effort. Meaning we don't expect that this needs a ton of follow-ups,
other than comm
Hi Stefan,
Thanks for your reply. Very interesting ideas!
If I understand correctly, SharedStateRegistry will still be responsible
for pruning the old state; for that, it will maintain some (ordered)
mapping between StateMaps and their versions, per key group.
I think one modification to this appr
Hi,
Very happy to see that the incremental checkpoint idea is finally becoming a
reality for the heap backend! Overall the proposal looks pretty good to me.
Just wanted to point out one possible improvement from what I can still
remember from my ideas back then: I think you can avoid doing peri
Hi devs,
I'd like to start a discussion of FLIP-151: Incremental snapshots for
heap-based state backend [1]
Heap backend, while being limited state sizes fitting into memory, also has
some advantages compared to RocksDB backend:
1. Serialization once per checkpoint, not per state modification. Th
