+1 for the suggestion. We have use the incremental checkpoint in our production for a long time.
Hangxiang Yu <master...@gmail.com> 于2022年6月14日周二 15:41写道： > +1 > It's basically enabled in most scenarios in production environments. > For HashMapStateBackend, it will adopt a full checkpoint even if we enable > incremental checkpoint. It will also support incremental checkpoint after > [1]. It's compatible. > BTW, I think we may also need to improve the documentation of incremental > checkpoints which users usually ask. There are some tickets like [2][3]. > > Best, > Hangxiang. > > [1] https://issues.apache.org/jira/browse/FLINK-21648 > [2] https://issues.apache.org/jira/browse/FLINK-22797 > [3] https://issues.apache.org/jira/browse/FLINK-7449 > > On Mon, Jun 13, 2022 at 7:48 PM Rui Fan <1996fan...@gmail.com> wrote: > > > Strongly +1 > > > > Best, > > Rui Fan > > > > On Mon, Jun 13, 2022 at 7:35 PM Martijn Visser <martijnvis...@apache.org > > > > wrote: > > > > > > BTW, from my knowledge, nothing would happen for HashMapStateBackend, > > > which does not support incremental checkpoint yet, when enabling > > > incremental checkpoints. > > > > > > Thanks Yun, if no errors would occur then definitely +1 to enable it by > > > default > > > > > > Op ma 13 jun. 2022 om 12:42 schreef Alexander Fedulov < > > > alexan...@ververica.com>: > > > > > > > +1 > > > > > > > > From my experience, it is actually hard to come up with use cases > where > > > > incremental checkpoints should explicitly not be enabled with the > > RocksDB > > > > state backend. If the state is so small that the full snapshots do > not > > > > have any negative impact, one should consider using > HashMapStateBackend > > > > anyway. > > > > > > > > Best, > > > > Alexander Fedulov > > > > > > > > > > > > On Mon, Jun 13, 2022 at 12:26 PM Jing Ge <j...@ververica.com> wrote: > > > > > > > > > +1 > > > > > > > > > > Glad to see the kickoff of this discussion. Thanks Lihe for driving > > > this! > > > > > > > > > > We have actually already discussed it internally a few months ago. > > > After > > > > > considering some corner cases, all agreed on enabling the > incremental > > > > > checkpoint as default. > > > > > > > > > > Best regards, > > > > > Jing > > > > > > > > > > On Mon, Jun 13, 2022 at 12:17 PM Yun Tang <myas...@live.com> > wrote: > > > > > > > > > > > Strongly +1 for making incremental checkpoints as default. Many > > users > > > > > have > > > > > > ever been asking why this configuration is not enabled by > default. > > > > > > > > > > > > BTW, from my knowledge, nothing would happen for > > HashMapStateBackend, > > > > > > which does not support incremental checkpoint yet, when enabling > > > > > > incremental checkpoints. > > > > > > > > > > > > > > > > > > Best > > > > > > Yun Tang > > > > > > ________________________________ > > > > > > From: Martijn Visser <martijnvis...@apache.org> > > > > > > Sent: Monday, June 13, 2022 18:05 > > > > > > To: dev@flink.apache.org <dev@flink.apache.org> > > > > > > Subject: Re: [DISCUSS ] Make state.backend.incremental as true by > > > > default > > > > > > > > > > > > Hi Lihe, > > > > > > > > > > > > What happens if we enable incremental checkpoints by default > while > > > the > > > > > used > > > > > > memory backend is HashMapStateBackend, which doesn't support > > > > incremental > > > > > > checkpoints? > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Martijn > > > > > > > > > > > > Op ma 13 jun. 2022 om 11:59 schreef Lihe Ma <ma_l...@163.com>: > > > > > > > > > > > > > Hi, Everyone, > > > > > > > > > > > > > > I would like to open a discussion on setting incremental > > checkpoint > > > > as > > > > > > > default behavior. > > > > > > > > > > > > > > Currently, the configuration of state.backend.incremental is > set > > as > > > > > false > > > > > > > by default. Incremental checkpoint has been adopted widely in > > > > industry > > > > > > > community for many years , and it is also well-tested from the > > > > feedback > > > > > > in > > > > > > > the community discussion. Incremental checkpointing is more > > > > > > light-weighted: > > > > > > > shorter checkpoint duration, less uploaded data and less > resource > > > > > > > consumption. > > > > > > > > > > > > > > In terms of backward compatibility, enable incremental > > > checkpointing > > > > > > would > > > > > > > not make any data loss no matter restoring from a full > > > > > > checkpoint/savepoint > > > > > > > or an incremental checkpoint. > > > > > > > > > > > > > > FLIP-193 (Snapshot ownership)[1] has been released in 1.15, > > > > incremental > > > > > > > checkpoint no longer depends on a previous restored checkpoint > in > > > > > default > > > > > > > NO_CLAIM mode, which makes the checkpoint lineage much cleaner, > > it > > > > is a > > > > > > > good chance to change the configuration > state.backend.incremental > > > to > > > > > true > > > > > > > as default. > > > > > > > > > > > > > > Thus, based on the above discussion, I suggest to make > > > > > > > state.backend.incremental as true by default. What do you think > > of > > > > this > > > > > > > proposal? > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership > > > > > > > > > > > > > > Best regards, > > > > > > > Lihe Ma > > > > > > > > > > > > > > > > > > > > > > > > > > > >
