Thank you for bringing this up! +1
On Mon, Jun 13, 2022 at 1:48 PM Rui Fan <1996fan...@gmail.com> wrote: > Strongly +1 > > Best, > Rui Fan > > On Mon, Jun 13, 2022 at 7:35 PM Martijn Visser <martijnvis...@apache.org> > wrote: > > > > BTW, from my knowledge, nothing would happen for HashMapStateBackend, > > which does not support incremental checkpoint yet, when enabling > > incremental checkpoints. > > > > Thanks Yun, if no errors would occur then definitely +1 to enable it by > > default > > > > Op ma 13 jun. 2022 om 12:42 schreef Alexander Fedulov < > > alexan...@ververica.com>: > > > > > +1 > > > > > > From my experience, it is actually hard to come up with use cases where > > > incremental checkpoints should explicitly not be enabled with the > RocksDB > > > state backend. If the state is so small that the full snapshots do not > > > have any negative impact, one should consider using HashMapStateBackend > > > anyway. > > > > > > Best, > > > Alexander Fedulov > > > > > > > > > On Mon, Jun 13, 2022 at 12:26 PM Jing Ge <j...@ververica.com> wrote: > > > > > > > +1 > > > > > > > > Glad to see the kickoff of this discussion. Thanks Lihe for driving > > this! > > > > > > > > We have actually already discussed it internally a few months ago. > > After > > > > considering some corner cases, all agreed on enabling the incremental > > > > checkpoint as default. > > > > > > > > Best regards, > > > > Jing > > > > > > > > On Mon, Jun 13, 2022 at 12:17 PM Yun Tang <myas...@live.com> wrote: > > > > > > > > > Strongly +1 for making incremental checkpoints as default. Many > users > > > > have > > > > > ever been asking why this configuration is not enabled by default. > > > > > > > > > > BTW, from my knowledge, nothing would happen for > HashMapStateBackend, > > > > > which does not support incremental checkpoint yet, when enabling > > > > > incremental checkpoints. > > > > > > > > > > > > > > > Best > > > > > Yun Tang > > > > > ________________________________ > > > > > From: Martijn Visser <martijnvis...@apache.org> > > > > > Sent: Monday, June 13, 2022 18:05 > > > > > To: dev@flink.apache.org <dev@flink.apache.org> > > > > > Subject: Re: [DISCUSS ] Make state.backend.incremental as true by > > > default > > > > > > > > > > Hi Lihe, > > > > > > > > > > What happens if we enable incremental checkpoints by default while > > the > > > > used > > > > > memory backend is HashMapStateBackend, which doesn't support > > > incremental > > > > > checkpoints? > > > > > > > > > > Best regards, > > > > > > > > > > Martijn > > > > > > > > > > Op ma 13 jun. 2022 om 11:59 schreef Lihe Ma <ma_l...@163.com>: > > > > > > > > > > > Hi, Everyone, > > > > > > > > > > > > I would like to open a discussion on setting incremental > checkpoint > > > as > > > > > > default behavior. > > > > > > > > > > > > Currently, the configuration of state.backend.incremental is set > as > > > > false > > > > > > by default. Incremental checkpoint has been adopted widely in > > > industry > > > > > > community for many years , and it is also well-tested from the > > > feedback > > > > > in > > > > > > the community discussion. Incremental checkpointing is more > > > > > light-weighted: > > > > > > shorter checkpoint duration, less uploaded data and less resource > > > > > > consumption. > > > > > > > > > > > > In terms of backward compatibility, enable incremental > > checkpointing > > > > > would > > > > > > not make any data loss no matter restoring from a full > > > > > checkpoint/savepoint > > > > > > or an incremental checkpoint. > > > > > > > > > > > > FLIP-193 (Snapshot ownership)[1] has been released in 1.15, > > > incremental > > > > > > checkpoint no longer depends on a previous restored checkpoint in > > > > default > > > > > > NO_CLAIM mode, which makes the checkpoint lineage much cleaner, > it > > > is a > > > > > > good chance to change the configuration state.backend.incremental > > to > > > > true > > > > > > as default. > > > > > > > > > > > > Thus, based on the above discussion, I suggest to make > > > > > > state.backend.incremental as true by default. What do you think > of > > > this > > > > > > proposal? > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership > > > > > > > > > > > > Best regards, > > > > > > Lihe Ma > > > > > > > > > > > > > > > > > > > > >
