Hi everyone, It has been 6 days since the last call for discussion. I'd like to start a vote after another 2 days.
Please let me know if you have any concerns. Thanks! Best, Zakelly On Tue, Jan 16, 2024 at 2:54 PM Zakelly Lan <zakelly....@gmail.com> wrote: > Thanks for the suggestion Rui! The type is added. > > > Best, > Zakelly > > On Tue, Jan 16, 2024 at 2:33 PM Rui Fan <1996fan...@gmail.com> wrote: > >> Hi Zakelly, >> >> Would you mind adding the option type in the FLIP doc? >> For example, String, Boolean or Enum, etc. Thank you. >> >> Best, >> Rui >> >> On Tue, Jan 16, 2024 at 2:29 PM Zakelly Lan <zakelly....@gmail.com> >> wrote: >> >> > Hi everyone, >> > >> > Thanks all for joining the discussion! I'd like to speed this up since >> it >> > lasts for nearly a month. I made changes on this FLIP based on >> suggestions >> > and compromises acceptable to most people. Please feel free to give your >> > opinion. Thanks! >> > If there are no more suggestions, I will consider starting a vote >> within a >> > week. >> > >> > >> > Best, >> > Zakelly >> > >> > On Thu, Jan 11, 2024 at 10:31 AM Xuannan Su <suxuanna...@gmail.com> >> wrote: >> > >> > > Hi Zakelly, >> > > >> > > I am fine with either Option 2 or Option 3. I think the naming in >> > > Option 2 makes it clear that it is a boolean configuration. However, >> > > most of the currently available boolean configurations do not use >> > > "enable" as a suffix. Therefore, Option 3 looks good to me as it >> > > follows the current practice. >> > > >> > > Best regards, >> > > Xuannan >> > > >> > > On Thu, Jan 11, 2024 at 9:50 AM Hangxiang Yu <master...@gmail.com> >> > wrote: >> > > > >> > > > > >> > > > > That's a very good point. I realize that the word 'recovery' means >> > way >> > > too >> > > > > many things. So I suggest picking a more specific word here, how >> > about >> > > > > 'execution.state-recovery.*' ? Checkpointing and state recovery >> are >> > > > > corresponding terms and won't make ambiguity. >> > > > > >> > > > >> > > > This makes the configuration clearer to me. We could focus on the >> > > > `state-recovery` at first. >> > > > >> > > > I think we could create another FLIP for the deprecation of LEGACY >> > mode. >> > > > > >> > > > >> > > > LGTM, Let's create a new FLIP to do this. >> > > > >> > > > IIUC, there is no clear ownership of the local copy files from the >> > > previous >> > > > > job and it's better to define one. This needs more discussion so >> we >> > > could >> > > > > create another thread for this. WDYT? >> > > > > >> > > > >> > > > Yeah, I have created a new ticket FLINK-34032 to track and discuss >> > this. >> > > > >> > > > On Wed, Jan 10, 2024 at 6:31 PM Zakelly Lan <zakelly....@gmail.com> >> > > wrote: >> > > > >> > > > > Hi everyone, >> > > > > >> > > > > It seems we still don't have a consensus on the rules for boolean >> > type >> > > > > options. Let me recap the alternatives we have: >> > > > > >> > > > > Option 1: Use enumeration options instead if possible. But this >> may >> > > cause >> > > > > some name collisions or confusion as we discussed and we should >> unify >> > > the >> > > > > statement everywhere. >> > > > > Option 2: Use boolean options and add 'enabled' as the suffix. >> > > > > Option 3: Use boolean options and ONLY add 'enabled' when there >> are >> > > more >> > > > > detailed configurations under the same prefix, to prevent one name >> > from >> > > > > serving as a prefix to another. >> > > > > >> > > > > I am inclined to Option 3, since it is more in line with current >> > > practice >> > > > > and friendly for existing users. Also It reduces the length of >> > > > > configuration names as much as possible. >> > > > > >> > > > > Looking forward to your opinions! Thanks! >> > > > > >> > > > > >> > > > > Best, >> > > > > Zakelly >> > > > > >> > > > > On Wed, Jan 10, 2024 at 3:30 PM Zakelly Lan < >> zakelly....@gmail.com> >> > > wrote: >> > > > > >> > > > > > Hi Hangxiang, >> > > > > > >> > > > > > Thanks for your suggestions! >> > > > > > >> > > > > > 1. Could execution.recovery also contain some other behaviors >> about >> > > > > >> recovery ? e.g. restart-strategy. >> > > > > > >> > > > > > >> > > > > > That's a very good point. I realize that the word 'recovery' >> means >> > > way >> > > > > too >> > > > > > many things. So I suggest picking a more specific word here, how >> > > about >> > > > > > 'execution.state-recovery.*' ? Checkpointing and state recovery >> are >> > > > > > corresponding terms and won't make ambiguity. >> > > > > > >> > > > > > 2. Could we also remove some legacy configuration value ? e.g. >> > LEGACY >> > > > > Mode >> > > > > >> for >> > execution.savepoint-restore-mode/execution.recovery.claim-mode. >> > > > > > >> > > > > > >> > > > > > I think we could create another FLIP for the deprecation of >> LEGACY >> > > mode. >> > > > > > >> > > > > > >> > > > > >> 3. Could the local checkpoint be cleaned >> > > > > >> if execution.checkpointing.local-copy.enabled is true and >> > > > > >> execution.recovery.from-local is false ? I found it's also an >> > issue >> > > if >> > > > > >> current local-recovery from enabled to disabled. Maybe another >> > > ticket is >> > > > > >> needed. >> > > > > > >> > > > > > >> > > > > > IIUC, there is no clear ownership of the local copy files from >> the >> > > > > > previous job and it's better to define one. This needs more >> > > discussion so >> > > > > > we could create another thread for this. WDYT? >> > > > > > >> > > > > > >> > > > > > Best, >> > > > > > Zakelly >> > > > > > >> > > > > > On Tue, Jan 9, 2024 at 11:23 AM Hangxiang Yu < >> master...@gmail.com> >> > > > > wrote: >> > > > > > >> > > > > >> Hi, Zakelly. >> > > > > >> Thanks for driving this. Overall LGTM as we discussed offline. >> > > > > >> >> > > > > >> Some comments/suggestions just came to mind: >> > > > > >> 1. Could execution.recovery also contain some other behaviors >> > about >> > > > > >> recovery ? e.g. restart-strategy. >> > > > > >> 2. Could we also remove some legacy configuration value ? e.g. >> > > LEGACY >> > > > > Mode >> > > > > >> for >> > execution.savepoint-restore-mode/execution.recovery.claim-mode. >> > > > > >> 3. Could the local checkpoint be cleaned >> > > > > >> if execution.checkpointing.local-copy.enabled is true and >> > > > > >> execution.recovery.from-local is false ? I found it's also an >> > issue >> > > if >> > > > > >> current local-recovery from enabled to disabled. Maybe another >> > > ticket is >> > > > > >> needed. >> > > > > >> 4. +1 for enabling execution.checkpointing.incremental by >> default >> > > which >> > > > > is >> > > > > >> basically default configuration in our production environment. >> > > > > >> >> > > > > >> >> > > > > >> On Mon, Jan 8, 2024 at 6:06 PM Zakelly Lan < >> zakelly....@gmail.com >> > > >> > > > > wrote: >> > > > > >> >> > > > > >> > Hi Yun, >> > > > > >> > >> > > > > >> > Thanks for your comments! >> > > > > >> > >> > > > > >> > 1. We shall not describe the configuration with its >> > > implementation >> > > > > for >> > > > > >> > > 'execution.checkpointing.local-copy.*' options, for hashmap >> > > > > >> > state-backend, >> > > > > >> > > it would write two streams and for Rocksdb state-backend, >> it >> > > would >> > > > > use >> > > > > >> > > hard-link for backup. Thus, I think >> > > > > >> > > 'execution.checkpointing.local-backup.*' looks better. >> > > > > >> > >> > > > > >> > I agreed that we'd better name the option in user's >> perspective >> > > > > instead >> > > > > >> of >> > > > > >> > the implementation, thus I name it as a copy of the >> checkpoint >> > in >> > > the >> > > > > >> > local disk, regardless of the way of generating it. The word >> > > 'backup' >> > > > > is >> > > > > >> > also suitable for this case, so I agree to change to >> > > > > >> > 'execution.checkpointing.local-backup.*' if no one objects. >> > > > > >> > >> > > > > >> > 2. What does the >> > 'execution.checkpointing.data-inline-threshold' >> > > > > >> mean? It >> > > > > >> > > seems not so easy to understand. >> > > > > >> > >> > > > > >> > The 'execution.checkpointing.data-inline-threshold' (original >> > one >> > > as >> > > > > >> > 'state.storage.fs.memory-threshold') stands for the size >> > threshold >> > > > > below >> > > > > >> > which state chunks will store inline with the metadata, thus >> I >> > > call it >> > > > > >> > 'data-inline-threshold'. >> > > > > >> > >> > > > > >> > >> > > > > >> > Best, >> > > > > >> > Zakelly >> > > > > >> > >> > > > > >> > On Mon, Jan 8, 2024 at 10:09 AM Yun Tang <myas...@live.com> >> > > wrote: >> > > > > >> > >> > > > > >> > > Hi Zakelly, >> > > > > >> > > >> > > > > >> > > Thanks for driving this topic. I have two concerns here: >> > > > > >> > > >> > > > > >> > > 1. We shall not describe the configuration with its >> > > > > implementation >> > > > > >> for >> > > > > >> > > 'execution.checkpointing.local-copy.*' options, for hashmap >> > > > > >> > state-backend, >> > > > > >> > > it would write two streams and for Rocksdb state-backend, >> it >> > > would >> > > > > use >> > > > > >> > > hard-link for backup. Thus, I think >> > > > > >> > > 'execution.checkpointing.local-backup.*' looks better. >> > > > > >> > > 2. What does the >> > > 'execution.checkpointing.data-inline-threshold' >> > > > > >> mean? >> > > > > >> > > It seems not so easy to understand. >> > > > > >> > > >> > > > > >> > > Best >> > > > > >> > > Yun Tang >> > > > > >> > > ________________________________ >> > > > > >> > > From: Piotr Nowojski <pnowoj...@apache.org> >> > > > > >> > > Sent: Thursday, January 4, 2024 22:37 >> > > > > >> > > To: dev@flink.apache.org <dev@flink.apache.org> >> > > > > >> > > Subject: Re: [DISCUSS] FLIP-406: Reorganize State & >> > > Checkpointing & >> > > > > >> > > Recovery Configuration >> > > > > >> > > >> > > > > >> > > Hi, >> > > > > >> > > >> > > > > >> > > Thanks for trying to clean this up! I don't have strong >> > > opinions on >> > > > > >> the >> > > > > >> > > topics discussed here, so generally speaking +1 from my >> side! >> > > > > >> > > >> > > > > >> > > Best, >> > > > > >> > > Piotrek >> > > > > >> > > >> > > > > >> > > śr., 3 sty 2024 o 04:16 Rui Fan <1996fan...@gmail.com> >> > > napisał(a): >> > > > > >> > > >> > > > > >> > > > Thanks for the feedback! >> > > > > >> > > > >> > > > > >> > > > Using the `execution.checkpointing.incremental.enabled`, >> > > > > >> > > > and enabling it by default sounds good to me. >> > > > > >> > > > >> > > > > >> > > > Best, >> > > > > >> > > > Rui >> > > > > >> > > > >> > > > > >> > > > On Wed, Jan 3, 2024 at 11:10 AM Zakelly Lan < >> > > > > zakelly....@gmail.com> >> > > > > >> > > wrote: >> > > > > >> > > > >> > > > > >> > > > > Hi Rui, >> > > > > >> > > > > >> > > > > >> > > > > Thanks for your comments! >> > > > > >> > > > > >> > > > > >> > > > > IMO, given that the state backend can be plugably >> loaded >> > > (as you >> > > > > >> can >> > > > > >> > > > > specify a state backend factory), I prefer not >> providing >> > > state >> > > > > >> > backend >> > > > > >> > > > > specified options in the framework. >> > > > > >> > > > > >> > > > > >> > > > > Secondly, the incremental checkpoint is actually a >> sharing >> > > file >> > > > > >> > > strategy >> > > > > >> > > > > across checkpoints, which means the state backend >> *could* >> > > reuse >> > > > > >> files >> > > > > >> > > > from >> > > > > >> > > > > previous cp but not *must* do so. When the state >> backend >> > > could >> > > > > not >> > > > > >> > > reuse >> > > > > >> > > > > the files, it is reasonable to fallback to a full >> > > checkpoint. >> > > > > >> > > > > >> > > > > >> > > > > Thus, I suggest we make it >> > > `execution.checkpointing.incremental` >> > > > > >> and >> > > > > >> > > > enable >> > > > > >> > > > > it by default. For those state backends not supporting >> > this, >> > > > > they >> > > > > >> > > perform >> > > > > >> > > > > full checkpoints and print a warning to inform users. >> > Users >> > > do >> > > > > not >> > > > > >> > need >> > > > > >> > > > to >> > > > > >> > > > > pay special attention to different options to control >> this >> > > > > across >> > > > > >> > > > different >> > > > > >> > > > > state backends. This is more user-friendly in my >> opinion. >> > > WDYT? >> > > > > >> > > > > >> > > > > >> > > > > On Tue, Jan 2, 2024 at 10:49 AM Rui Fan < >> > > 1996fan...@gmail.com> >> > > > > >> > wrote: >> > > > > >> > > > > >> > > > > >> > > > > > Hi Zakelly, >> > > > > >> > > > > > >> > > > > >> > > > > > I'm not sure whether we could add the state backend >> type >> > > in >> > > > > the >> > > > > >> > > > > > new key name of state.backend.incremental. It means >> we >> > use >> > > > > >> > > > > > `execution.checkpointing.rocksdb-incremental` or >> > > > > >> > > > > > >> `execution.checkpointing.rocksdb-incremental.enabled`. >> > > > > >> > > > > > >> > > > > >> > > > > > So far, state.backend.incremental only works for >> rocksdb >> > > state >> > > > > >> > > backend. >> > > > > >> > > > > > And this feature or optimization is very valuable and >> > > huge for >> > > > > >> > large >> > > > > >> > > > > > state flink jobs. I believe it's enabled for most >> > > production >> > > > > >> flink >> > > > > >> > > jobs >> > > > > >> > > > > > with large rocksdb state. >> > > > > >> > > > > > >> > > > > >> > > > > > If this option isn't generic for all state backend >> > types, >> > > I >> > > > > >> guess >> > > > > >> > we >> > > > > >> > > > > > can enable >> > > > > `execution.checkpointing.rocksdb-incremental.enabled` >> > > > > >> > > > > > by default in Flink 2.0. >> > > > > >> > > > > > >> > > > > >> > > > > > But if it works for all state backends, it's hard to >> > > enable it >> > > > > >> by >> > > > > >> > > > > default. >> > > > > >> > > > > > Enabling great and valuable features or improvements >> are >> > > > > useful >> > > > > >> > > > > > for users, especially a lot of new flink users. >> > > Out-of-the-box >> > > > > >> > > options >> > > > > >> > > > > > are good for users. >> > > > > >> > > > > > >> > > > > >> > > > > > WDYT? >> > > > > >> > > > > > >> > > > > >> > > > > > Best, >> > > > > >> > > > > > Rui >> > > > > >> > > > > > >> > > > > >> > > > > > On Fri, Dec 29, 2023 at 1:45 PM Zakelly Lan < >> > > > > >> zakelly....@gmail.com >> > > > > >> > > >> > > > > >> > > > > wrote: >> > > > > >> > > > > > >> > > > > >> > > > > > > Hi everyone, >> > > > > >> > > > > > > >> > > > > >> > > > > > > Thanks all for your comments! >> > > > > >> > > > > > > >> > > > > >> > > > > > > As many of you have questions about the names for >> > > boolean >> > > > > >> > options, >> > > > > >> > > I >> > > > > >> > > > > > > suggest we make a naming rule for them. For now I >> > could >> > > > > think >> > > > > >> of >> > > > > >> > > > three >> > > > > >> > > > > > > options: >> > > > > >> > > > > > > >> > > > > >> > > > > > > Option 1: Use enumeration options if possible. But >> > this >> > > may >> > > > > >> cause >> > > > > >> > > > some >> > > > > >> > > > > > name >> > > > > >> > > > > > > collisions or confusion as we discussed and we >> should >> > > unify >> > > > > >> the >> > > > > >> > > > > statement >> > > > > >> > > > > > > everywhere. >> > > > > >> > > > > > > Option 2: Use boolean options and add 'enabled' as >> the >> > > > > suffix. >> > > > > >> > > > > > > Option 3: Use boolean options and ONLY add >> 'enabled' >> > > when >> > > > > >> there >> > > > > >> > are >> > > > > >> > > > > more >> > > > > >> > > > > > > detailed configurations under the same prefix, to >> > > prevent >> > > > > one >> > > > > >> > name >> > > > > >> > > > from >> > > > > >> > > > > > > serving as a prefix to another. >> > > > > >> > > > > > > >> > > > > >> > > > > > > I am slightly inclined to Option 3, since it is >> more >> > in >> > > line >> > > > > >> with >> > > > > >> > > > > current >> > > > > >> > > > > > > practice and friendly for existing users. Also It >> > > reduces >> > > > > the >> > > > > >> > > length >> > > > > >> > > > of >> > > > > >> > > > > > > configuration names as much as possible. I really >> want >> > > to >> > > > > hear >> > > > > >> > your >> > > > > >> > > > > > > opinions. >> > > > > >> > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > > @Xuannan >> > > > > >> > > > > > > >> > > > > >> > > > > > > I agree with your comments 1 and 3. >> > > > > >> > > > > > > >> > > > > >> > > > > > > For 2, If we decide to change the name, maybe >> > > > > >> > > > > > > `execution.checkpointing.parallel-cleaner` is >> better? >> > > And as >> > > > > >> for >> > > > > >> > > > > whether >> > > > > >> > > > > > to >> > > > > >> > > > > > > add 'enabled' I suggest we discuss the rule above. >> > WDYT? >> > > > > >> > > > > > > Thanks! >> > > > > >> > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > > Best, >> > > > > >> > > > > > > Zakelly >> > > > > >> > > > > > > >> > > > > >> > > > > > > On Fri, Dec 29, 2023 at 12:02 PM Xuannan Su < >> > > > > >> > suxuanna...@gmail.com >> > > > > >> > > > >> > > > > >> > > > > > wrote: >> > > > > >> > > > > > > >> > > > > >> > > > > > > > Hi Zakelly, >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > Thanks for driving this! The organization of the >> > > > > >> configuration >> > > > > >> > > > option >> > > > > >> > > > > > > > in the FLIP looks much cleaner and easier to >> > > understand. >> > > > > +1 >> > > > > >> to >> > > > > >> > > the >> > > > > >> > > > > > > > FLIP. >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > Just some questions from me. >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > 1. I think the change to the ConfigOptions >> should be >> > > put >> > > > > in >> > > > > >> the >> > > > > >> > > > > > > > `Public Interface` section, instead of `Proposed >> > > Changed`, >> > > > > >> as >> > > > > >> > > those >> > > > > >> > > > > > > > configuration options are public interface. >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > 2. The key >> `state.checkpoint.cleaner.parallel-mode` >> > > seems >> > > > > >> > > > confusing. >> > > > > >> > > > > > > > It feels like it is used to choose different >> modes. >> > In >> > > > > >> fact, it >> > > > > >> > > is >> > > > > >> > > > a >> > > > > >> > > > > > > > boolean flag to indicate whether to enable >> parallel >> > > clean. >> > > > > >> How >> > > > > >> > > > about >> > > > > >> > > > > > > > making it >> > > > > `state.checkpoint.cleaner.parallel-mode.enabled`? >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > 3. The `execution.checkpointing.write-buffer` may >> > > better >> > > > > be >> > > > > >> > > > > > > > `execution.checkpointing.write-buffer-size` so >> that >> > we >> > > > > know >> > > > > >> it >> > > > > >> > is >> > > > > >> > > > > > > > configuring the size of the buffer. >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > Best, >> > > > > >> > > > > > > > Xuannan >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > On Wed, Dec 27, 2023 at 7:17 PM Yanfei Lei < >> > > > > >> > fredia...@gmail.com> >> > > > > >> > > > > > wrote: >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > Hi Zakelly, >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > > Considering the name occupation, how about >> > naming >> > > it >> > > > > as >> > > > > >> > > > > > > > `execution.checkpointing.type`? >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > `Checkpoint Type`[1,2] is used to describe >> > > > > >> aligned/unaligned >> > > > > >> > > > > > > > > checkpoint, I am inclined to make a choice >> between >> > > > > >> > > > > > > > > `execution.checkpointing.incremental` and >> > > > > >> > > > > > > > > `execution.checkpointing.incremental.enabled`. >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > [1] >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > >> > >> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/monitoring/checkpoint_monitoring/ >> > > > > >> > > > > > > > > [2] >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > >> > >> https://github.com/apache/flink/blob/master/flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.html#L27 >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > -- >> > > > > >> > > > > > > > > Best, >> > > > > >> > > > > > > > > Yanfei >> > > > > >> > > > > > > > > >> > > > > >> > > > > > > > > Zakelly Lan <zakelly....@gmail.com> >> > 于2023年12月27日周三 >> > > > > >> 14:41写道： >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > Hi Lijie, >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > Thanks for the reminder! I missed this. >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > Considering the name occupation, how about >> > naming >> > > it >> > > > > as >> > > > > >> > > > > > > > > > `execution.checkpointing.type`? >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > Actually I think the current >> > > > > >> `execution.checkpointing.mode` >> > > > > >> > > is >> > > > > >> > > > > > > > confusing in >> > > > > >> > > > > > > > > > some ways, maybe >> > > > > >> `execution.checkpointing.data-consistency` >> > > > > >> > > is >> > > > > >> > > > > > > better. >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > Best, >> > > > > >> > > > > > > > > > Zakelly >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > On Wed, Dec 27, 2023 at 12:59 PM Lijie Wang < >> > > > > >> > > > > > > wangdachui9...@gmail.com> >> > > > > >> > > > > > > > > > wrote: >> > > > > >> > > > > > > > > > >> > > > > >> > > > > > > > > > > Hi Zakelly, >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > > >> I'm wondering if >> > > > > >> > `execution.checkpointing.savepoint-dir` >> > > > > >> > > > > would >> > > > > >> > > > > > > be >> > > > > >> > > > > > > > > > > better. >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > > `execution.checkpointing.dir` and >> > > > > >> > > > > > > > `execution.checkpointing.savepoint-dir` >> > > > > >> > > > > > > > > > > are also fine for me. >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > > >> So I think an enumeration option >> > > > > >> > > > > > `execution.checkpointing.mode` >> > > > > >> > > > > > > > which >> > > > > >> > > > > > > > > > > can be 'full' (default) or 'incremental' >> would >> > > be >> > > > > >> better >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > > I agree with using an enumeration option. >> But >> > > > > >> currently >> > > > > >> > > there >> > > > > >> > > > > is >> > > > > >> > > > > > > > already a >> > > > > >> > > > > > > > > > > configuration option called >> > > > > >> > `execution.checkpointing.mode`, >> > > > > >> > > > > which >> > > > > >> > > > > > > is >> > > > > >> > > > > > > > used >> > > > > >> > > > > > > > > > > to choose EXACTLY_ONCE or AT_LEAST_ONCE. >> Maybe >> > > we >> > > > > >> need to >> > > > > >> > > use >> > > > > >> > > > > > > > another name >> > > > > >> > > > > > > > > > > or merge these two options. >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > > Best, >> > > > > >> > > > > > > > > > > Lijie >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > > Zakelly Lan <zakelly....@gmail.com> >> > > 于2023年12月27日周三 >> > > > > >> > > 11:43写道： >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > > > > > Hi everyone, >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > Thanks all for your comments! >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > @Yanfei >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > 1. For some state backends that do not >> > > support >> > > > > >> > > > incremental >> > > > > >> > > > > > > > checkpoint, >> > > > > >> > > > > > > > > > > > > how does the >> > > > > >> > execution.checkpointing.incrementaloption >> > > > > >> > > > take >> > > > > >> > > > > > > > effect? Or >> > > > > >> > > > > > > > > > > > > is it better to put incremental under >> > > > > >> > > > > > > > state.backend.xxx.incremental? >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > I'd rather not put the option for >> > incremental >> > > > > >> > checkpoint >> > > > > >> > > > > under >> > > > > >> > > > > > > the >> > > > > >> > > > > > > > > > > > 'state.backend', since it is more about >> the >> > > > > >> > checkpointing >> > > > > >> > > > > > instead >> > > > > >> > > > > > > > of >> > > > > >> > > > > > > > > > > state >> > > > > >> > > > > > > > > > > > accessing. Of course, the state backend >> may >> > > not >> > > > > >> > > necessarily >> > > > > >> > > > > do >> > > > > >> > > > > > > > > > > incremental >> > > > > >> > > > > > > > > > > > checkpoint as requested. If the state >> > backend >> > > is >> > > > > not >> > > > > >> > > > capable >> > > > > >> > > > > of >> > > > > >> > > > > > > > taking >> > > > > >> > > > > > > > > > > > incremental cp, it is better to fallback >> to >> > > the >> > > > > full >> > > > > >> > cp. >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > 2. I'm a little worried that putting all >> > > > > >> configurations >> > > > > >> > > > into >> > > > > >> > > > > > > > > > > > > `ExecutionCheckpointingOptions` will >> > > introduce >> > > > > >> some >> > > > > >> > > > > > dependency >> > > > > >> > > > > > > > > > > > > problems. Some options would be used by >> > > > > >> flink-runtime >> > > > > >> > > > > module, >> > > > > >> > > > > > > but >> > > > > >> > > > > > > > > > > > > flink-runtime should not depend on >> > > > > >> > > flink-streaming-java. >> > > > > >> > > > > e.g. >> > > > > >> > > > > > > > > > > > > FLINK-28286[1]. >> > > > > >> > > > > > > > > > > > > So, I prefer to move configurations to >> > > > > >> > > > > > `CheckpointingOptions`, >> > > > > >> > > > > > > > WDYT? >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > Yes, that's a very good point. Moving to >> > > > > >> > > > > > > > > > > > `CheckpointingOptions`(flink-core) makes >> > > sense. >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > @Lijie >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > How about >> > > > > >> > > > > > > > > > > > > state.savepoints.dir -> >> > > > > >> > > > > execution.checkpointing.savepoint.dir >> > > > > >> > > > > > > > > > > > > state.checkpoints.dir -> >> > > > > >> > > > > > execution.checkpointing.checkpoint.dir >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > Actually, I think the >> > > `checkpointing.checkpoint` >> > > > > may >> > > > > >> > > cause >> > > > > >> > > > > some >> > > > > >> > > > > > > > > > > confusion. >> > > > > >> > > > > > > > > > > > But I'm ok if others agree. >> > > > > >> > > > > > > > > > > > I'm wondering if >> > > > > >> > `execution.checkpointing.savepoint-dir` >> > > > > >> > > > > would >> > > > > >> > > > > > be >> > > > > >> > > > > > > > better. >> > > > > >> > > > > > > > > > > > WDYT? >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > 2. We changed the >> > > > > >> execution.checkpointing.local-copy' >> > > > > >> > to >> > > > > >> > > > > > > > > > > > > >> > > 'execution.checkpointing.local-copy.enabled'. >> > > > > >> Should >> > > > > >> > we >> > > > > >> > > > > also >> > > > > >> > > > > > > add >> > > > > >> > > > > > > > > > > > "enabled" >> > > > > >> > > > > > > > > > > > > suffix for other boolean type >> > configuration >> > > > > >> options ? >> > > > > >> > > For >> > > > > >> > > > > > > > example, >> > > > > >> > > > > > > > > > > > > execution.checkpointing.incremental -> >> > > > > >> > > > > > > > > > > > > >> > execution.checkpointing.incremental.enabled >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > Actually, the incremental cp is something >> > like >> > > > > >> > choosing a >> > > > > >> > > > > mode >> > > > > >> > > > > > > for >> > > > > >> > > > > > > > doing >> > > > > >> > > > > > > > > > > > checkpoint instead of enabling a >> function. >> > So >> > > I >> > > > > >> think >> > > > > >> > an >> > > > > >> > > > > > > > enumeration >> > > > > >> > > > > > > > > > > option >> > > > > >> > > > > > > > > > > > `execution.checkpointing.mode` which can >> be >> > > 'full' >> > > > > >> > > > (default) >> > > > > >> > > > > or >> > > > > >> > > > > > > > > > > > 'incremental' would be better, WDYT? >> > > > > >> > > > > > > > > > > > And @Rui Fan @Yanfei What do you think >> about >> > > this? >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > On Tue, Dec 26, 2023 at 5:15 PM Lijie >> Wang < >> > > > > >> > > > > > > > wangdachui9...@gmail.com> >> > > > > >> > > > > > > > > > > > wrote: >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > Hi Zakelly, >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > Thanks for driving the discussion. >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > 1. >> > > > > >> > > > > > > > > > > > > >> But I'm not so sure since there is >> only >> > > one >> > > > > >> > > > > > > savepoint-related >> > > > > >> > > > > > > > > > > option. >> > > > > >> > > > > > > > > > > > > Maybe someone else could share some >> > thoughts >> > > > > here. >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > How about >> > > > > >> > > > > > > > > > > > > state.savepoints.dir -> >> > > > > >> > > > > execution.checkpointing.savepoint.dir >> > > > > >> > > > > > > > > > > > > state.checkpoints.dir -> >> > > > > >> > > > > > execution.checkpointing.checkpoint.dir >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > 2. We changed the >> > > > > >> execution.checkpointing.local-copy' >> > > > > >> > > to >> > > > > >> > > > > > > > > > > > > >> > > 'execution.checkpointing.local-copy.enabled'. >> > > > > >> Should >> > > > > >> > we >> > > > > >> > > > > also >> > > > > >> > > > > > > add >> > > > > >> > > > > > > > > > > > "enabled" >> > > > > >> > > > > > > > > > > > > suffix for other boolean type >> > configuration >> > > > > >> options ? >> > > > > >> > > For >> > > > > >> > > > > > > > example, >> > > > > >> > > > > > > > > > > > > execution.checkpointing.incremental -> >> > > > > >> > > > > > > > > > > > > >> > execution.checkpointing.incremental.enabled >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > In this way, the naming style of >> > > configuration >> > > > > >> > options >> > > > > >> > > is >> > > > > >> > > > > > > > unified, and >> > > > > >> > > > > > > > > > > it >> > > > > >> > > > > > > > > > > > > can avoid potential similar problems >> (for >> > > > > >> example, we >> > > > > >> > > may >> > > > > >> > > > > > need >> > > > > >> > > > > > > > to add >> > > > > >> > > > > > > > > > > > more >> > > > > >> > > > > > > > > > > > > options for incremental checkpoint in >> the >> > > > > future). >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > Best, >> > > > > >> > > > > > > > > > > > > Lijie >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > Yanfei Lei <fredia...@gmail.com> >> > > 于2023年12月26日周二 >> > > > > >> > > 12:05写道： >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > Hi Zakelly, >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > Thank you for creating the FLIP and >> > > starting >> > > > > the >> > > > > >> > > > > > discussion. >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > The current arrangement of these >> options >> > > is >> > > > > >> indeed >> > > > > >> > > > > somewhat >> > > > > >> > > > > > > > > > > haphazard, >> > > > > >> > > > > > > > > > > > > > and the new arrangement looks much >> > > better. I >> > > > > >> have >> > > > > >> > > some >> > > > > >> > > > > > > > questions >> > > > > >> > > > > > > > > > > about >> > > > > >> > > > > > > > > > > > > > the arrangement of some new >> > configuration >> > > > > >> options: >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > 1. For some state backends that do >> not >> > > support >> > > > > >> > > > > incremental >> > > > > >> > > > > > > > > > > checkpoint, >> > > > > >> > > > > > > > > > > > > > how does the >> > > > > >> > > execution.checkpointing.incrementaloption >> > > > > >> > > > > take >> > > > > >> > > > > > > > effect? >> > > > > >> > > > > > > > > > > Or >> > > > > >> > > > > > > > > > > > > > is it better to put incremental under >> > > > > >> > > > > > > > state.backend.xxx.incremental? >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > 2. I'm a little worried that putting >> all >> > > > > >> > > configurations >> > > > > >> > > > > > into >> > > > > >> > > > > > > > > > > > > > `ExecutionCheckpointingOptions` will >> > > introduce >> > > > > >> some >> > > > > >> > > > > > > dependency >> > > > > >> > > > > > > > > > > > > > problems. Some options would be used >> by >> > > > > >> > flink-runtime >> > > > > >> > > > > > module, >> > > > > >> > > > > > > > but >> > > > > >> > > > > > > > > > > > > > flink-runtime should not depend on >> > > > > >> > > > flink-streaming-java. >> > > > > >> > > > > > e.g. >> > > > > >> > > > > > > > > > > > > > FLINK-28286[1]. >> > > > > >> > > > > > > > > > > > > > So, I prefer to move configurations >> to >> > > > > >> > > > > > > `CheckpointingOptions`, >> > > > > >> > > > > > > > WDYT? >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > [1] >> > > > > >> > > https://issues.apache.org/jira/browse/FLINK-28286 >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > -- >> > > > > >> > > > > > > > > > > > > > Best, >> > > > > >> > > > > > > > > > > > > > Yanfei >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > Zakelly Lan <zakelly....@gmail.com> >> > > > > >> 于2023年12月25日周一 >> > > > > >> > > > > > 21:14写道： >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > Hi Rui Fan and Junrui, >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > Thanks for the reminder! I agree to >> > > change >> > > > > the >> > > > > >> > > > > > > > > > > > > > > >> 'execution.checkpointing.local-copy' >> > to >> > > > > >> > > > > > > > > > > > > > > >> > > > > 'execution.checkpointing.local-copy.enabled'. >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > And for other suggestions Rui >> > proposed: >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > 1. How about >> > > > > >> execution.checkpointing.storage.type >> > > > > >> > > > > instead >> > > > > >> > > > > > > > > > > > > > > > of >> execution.checkpointing.storage? >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > Ah, I missed something here. >> Actually >> > I >> > > > > >> suggest >> > > > > >> > we >> > > > > >> > > > > could >> > > > > >> > > > > > > > merge the >> > > > > >> > > > > > > > > > > > > > current >> > > > > >> > > > > > > > > > > > > > > 'state.checkpoints.dir' and >> > > > > >> > > > 'state.checkpoint-storage' >> > > > > >> > > > > > into >> > > > > >> > > > > > > > one URI >> > > > > >> > > > > > > > > > > > > > > configuration named >> > > > > >> > 'execution.checkpointing.dir'. >> > > > > >> > > > > WDYT? >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > 3. >> > > execution.checkpointing.savepoint.dir is >> > > > > a >> > > > > >> > > little >> > > > > >> > > > > > weird. >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > Yes, I think it is better to make >> > > > > 'savepoint' >> > > > > >> and >> > > > > >> > > > > > > > 'checkpoint' the >> > > > > >> > > > > > > > > > > > same >> > > > > >> > > > > > > > > > > > > > > level. But I'm not so sure since >> there >> > > is >> > > > > only >> > > > > >> > one >> > > > > >> > > > > > > > > > > savepoint-related >> > > > > >> > > > > > > > > > > > > > > option. Maybe someone else could >> share >> > > some >> > > > > >> > > thoughts >> > > > > >> > > > > > here. >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > 4. How about >> > > execution.recovery.claim-mode >> > > > > >> > instead >> > > > > >> > > of >> > > > > >> > > > > > > > > > > > > > > > execution.recovery.mode? >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > Agreed. That's more accurate. >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > Many thanks for your suggestions! >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > Best, >> > > > > >> > > > > > > > > > > > > > > Zakelly >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 8:18 PM >> Junrui >> > > Lee < >> > > > > >> > > > > > > > jrlee....@gmail.com> >> > > > > >> > > > > > > > > > > > > wrote: >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > Hi Zakelly, >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > Thanks for driving this. I agree >> > that >> > > the >> > > > > >> > > proposed >> > > > > >> > > > > > > > restructuring >> > > > > >> > > > > > > > > > > of >> > > > > >> > > > > > > > > > > > > the >> > > > > >> > > > > > > > > > > > > > > > configuration options is largely >> > > positive. >> > > > > >> It >> > > > > >> > > will >> > > > > >> > > > > make >> > > > > >> > > > > > > > > > > > understanding >> > > > > >> > > > > > > > > > > > > > and >> > > > > >> > > > > > > > > > > > > > > > working with Flink configurations >> > more >> > > > > >> > intuitive. >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > Most of the proposed changes look >> > > great. >> > > > > >> Just a >> > > > > >> > > > > > heads-up, >> > > > > >> > > > > > > > as Rui >> > > > > >> > > > > > > > > > > > Fan >> > > > > >> > > > > > > > > > > > > > > > mentioned, Flink currently >> requires >> > > that >> > > > > no >> > > > > >> > > > > > > configOption's >> > > > > >> > > > > > > > key be >> > > > > >> > > > > > > > > > > > the >> > > > > >> > > > > > > > > > > > > > > > prefix of another to avoid issues >> > > when we >> > > > > >> > > > eventually >> > > > > >> > > > > > > adopt >> > > > > >> > > > > > > > a >> > > > > >> > > > > > > > > > > > standard >> > > > > >> > > > > > > > > > > > > > YAML >> > > > > >> > > > > > > > > > > > > > > > parser, as detailed in >> FLINK-29372 ( >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > https://issues.apache.org/jira/browse/FLINK-29372 >> > > > > >> > > > ). >> > > > > >> > > > > > > > Therefore, >> > > > > >> > > > > > > > > > > > it's >> > > > > >> > > > > > > > > > > > > > better >> > > > > >> > > > > > > > > > > > > > > > to change the key >> > > > > >> > > > > 'execution.checkpointing.local-copy' >> > > > > >> > > > > > > > because it >> > > > > >> > > > > > > > > > > > > > serves as >> > > > > >> > > > > > > > > > > > > > > > a prefix to the key >> > > > > >> > > > > > > > 'execution.checkpointing.local-copy.dir'. >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > Best regards, >> > > > > >> > > > > > > > > > > > > > > > Junrui >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > Rui Fan <1996fan...@gmail.com> >> > > > > >> 于2023年12月25日周一 >> > > > > >> > > > > 19:11写道： >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > Hi Zakelly, >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > Thank you for driving this >> > proposal! >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > Overall good for me. I have >> some >> > > > > questions >> > > > > >> > > about >> > > > > >> > > > > > these >> > > > > >> > > > > > > > names. >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > 1. How about >> > > > > >> > > execution.checkpointing.storage.type >> > > > > >> > > > > > > > instead of >> > > > > >> > > > > > > > > > > > > > > > > >> execution.checkpointing.storage? >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > It's similar to >> > state.backend.type. >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > 2. How about >> > > > > >> > > > > > execution.checkpointing.local-copy.enabled >> > > > > >> > > > > > > > instead >> > > > > >> > > > > > > > > > > > of >> > > > > >> > > > > > > > > > > > > > > > > >> > execution.checkpointing.local-copy? >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > You added a new option: >> > > > > >> > > > > > > > execution.checkpointing.local-copy.dir. >> > > > > >> > > > > > > > > > > > > > > > > IIUC, one option name >> shouldn't be >> > > the >> > > > > >> prefix >> > > > > >> > > of >> > > > > >> > > > > > other >> > > > > >> > > > > > > > options. >> > > > > >> > > > > > > > > > > > > > > > > If you add a new option >> > > > > >> > > > > > > > execution.checkpointing.local-copy, >> > > > > >> > > > > > > > > > > > > > > > > flink CI will fail directly. >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > 3. >> > > execution.checkpointing.savepoint.dir >> > > > > >> is a >> > > > > >> > > > > little >> > > > > >> > > > > > > > weird. >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > For old options: >> > > state.savepoints.dir >> > > > > and >> > > > > >> > > > > > > > > > > state.checkpoints.dir, >> > > > > >> > > > > > > > > > > > > > > > > the savepoint and checkpoint >> are >> > the >> > > > > same >> > > > > >> > > level. >> > > > > >> > > > It >> > > > > >> > > > > > > means >> > > > > >> > > > > > > > > > > > > > > > > it's a checkpoint or savepoint. >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > The new option >> > > > > >> execution.checkpointing.dir is >> > > > > >> > > > fine >> > > > > >> > > > > > for >> > > > > >> > > > > > > > me. >> > > > > >> > > > > > > > > > > > > > > > > However, >> > > > > >> > execution.checkpointing.savepoint.dir >> > > > > >> > > > is a >> > > > > >> > > > > > > > little >> > > > > >> > > > > > > > > > > weird. >> > > > > >> > > > > > > > > > > > > > > > > I don't know which name is >> better >> > > now. >> > > > > >> Let us >> > > > > >> > > > think >> > > > > >> > > > > > > > about it >> > > > > >> > > > > > > > > > > > more. >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > 4. How about >> > > > > execution.recovery.claim-mode >> > > > > >> > > > instead >> > > > > >> > > > > of >> > > > > >> > > > > > > > > > > > > > > > > execution.recovery.mode? >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > The meaning of mode is too >> broad. >> > > The >> > > > > >> > > claim-mode >> > > > > >> > > > > may >> > > > > >> > > > > > > > > > > > > > > > > be more accurate for users. >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > WDYT? >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > Best, >> > > > > >> > > > > > > > > > > > > > > > > Rui >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > On Mon, Dec 25, 2023 at 5:14 PM >> > > Zakelly >> > > > > >> Lan < >> > > > > >> > > > > > > > > > > > zakelly....@gmail.com >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > wrote: >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > Hi devs, >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > I'd like to start a >> discussion >> > on >> > > > > >> FLIP-406: >> > > > > >> > > > > > > Reorganize >> > > > > >> > > > > > > > State >> > > > > >> > > > > > > > > > > & >> > > > > >> > > > > > > > > > > > > > > > > > Checkpointing & Recovery >> > > > > >> Configuration[1]. >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > Currently, the configuration >> > > options >> > > > > >> > > pertaining >> > > > > >> > > > > to >> > > > > >> > > > > > > > > > > > checkpointing, >> > > > > >> > > > > > > > > > > > > > > > > recovery, >> > > > > >> > > > > > > > > > > > > > > > > > and state management are >> > primarily >> > > > > >> grouped >> > > > > >> > > > under >> > > > > >> > > > > > the >> > > > > >> > > > > > > > > > > following >> > > > > >> > > > > > > > > > > > > > > > prefixes: >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > - state.backend.* : >> > > configurations >> > > > > >> > related >> > > > > >> > > > to >> > > > > >> > > > > > > state >> > > > > >> > > > > > > > > > > > accessing >> > > > > >> > > > > > > > > > > > > > and >> > > > > >> > > > > > > > > > > > > > > > > > checkpointing, as well as >> > > specific >> > > > > >> > options >> > > > > >> > > > for >> > > > > >> > > > > > > > individual >> > > > > >> > > > > > > > > > > > > state >> > > > > >> > > > > > > > > > > > > > > > > backends >> > > > > >> > > > > > > > > > > > > > > > > > - >> execution.checkpointing.* : >> > > > > >> > > configurations >> > > > > >> > > > > > > > associated >> > > > > >> > > > > > > > > > > with >> > > > > >> > > > > > > > > > > > > > > > > checkpoint >> > > > > >> > > > > > > > > > > > > > > > > > execution and recovery >> > > > > >> > > > > > > > > > > > > > > > > > - execution.savepoint.*: >> > > > > >> configurations >> > > > > >> > > for >> > > > > >> > > > > > > > recovery from >> > > > > >> > > > > > > > > > > > > > savepoint >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > In addition, there are >> several >> > > > > >> individual >> > > > > >> > > > options >> > > > > >> > > > > > > such >> > > > > >> > > > > > > > as ' >> > > > > >> > > > > > > > > > > > > > > > > > *state.checkpoint-storage*' >> and >> > > > > >> > > > > > > > '*state.checkpoints.dir*' >> > > > > >> > > > > > > > > > > that >> > > > > >> > > > > > > > > > > > > fall >> > > > > >> > > > > > > > > > > > > > > > > outside >> > > > > >> > > > > > > > > > > > > > > > > > of these prefixes. The >> current >> > > > > >> arrangement >> > > > > >> > of >> > > > > >> > > > > these >> > > > > >> > > > > > > > options, >> > > > > >> > > > > > > > > > > > > which >> > > > > >> > > > > > > > > > > > > > span >> > > > > >> > > > > > > > > > > > > > > > > > multiple modules, is somewhat >> > > > > haphazard >> > > > > >> and >> > > > > >> > > > > lacks a >> > > > > >> > > > > > > > > > > systematic >> > > > > >> > > > > > > > > > > > > > > > structure. >> > > > > >> > > > > > > > > > > > > > > > > > For example, the options >> under >> > the >> > > > > >> > > > > > > > '*CheckpointingOptions*' >> > > > > >> > > > > > > > > > > > and ' >> > > > > >> > > > > > > > > > > > > > > > > > >> *ExecutionCheckpointingOptions*' >> > > are >> > > > > >> > related >> > > > > >> > > > and >> > > > > >> > > > > > have >> > > > > >> > > > > > > > no >> > > > > >> > > > > > > > > > > clear >> > > > > >> > > > > > > > > > > > > > > > boundaries >> > > > > >> > > > > > > > > > > > > > > > > > from the user's perspective, >> but >> > > there >> > > > > >> is >> > > > > >> > no >> > > > > >> > > > > > unified >> > > > > >> > > > > > > > prefix >> > > > > >> > > > > > > > > > > for >> > > > > >> > > > > > > > > > > > > > them. >> > > > > >> > > > > > > > > > > > > > > > > With >> > > > > >> > > > > > > > > > > > > > > > > > the upcoming release of Flink >> > > 2.0, we >> > > > > >> have >> > > > > >> > an >> > > > > >> > > > > > > excellent >> > > > > >> > > > > > > > > > > > > > opportunity to >> > > > > >> > > > > > > > > > > > > > > > > > overhaul and restructure the >> > > > > >> configurations >> > > > > >> > > > > related >> > > > > >> > > > > > > to >> > > > > >> > > > > > > > > > > > > > checkpointing, >> > > > > >> > > > > > > > > > > > > > > > > > recovery, and state >> management. >> > > This >> > > > > >> FLIP >> > > > > >> > > > > proposes >> > > > > >> > > > > > to >> > > > > >> > > > > > > > > > > > reorganize >> > > > > >> > > > > > > > > > > > > > these >> > > > > >> > > > > > > > > > > > > > > > > > settings, making it more >> > coherent >> > > by >> > > > > >> > module, >> > > > > >> > > > > which >> > > > > >> > > > > > > > would >> > > > > >> > > > > > > > > > > > > > significantly >> > > > > >> > > > > > > > > > > > > > > > > > lower the barriers for >> > > understanding >> > > > > and >> > > > > >> > > reduce >> > > > > >> > > > > the >> > > > > >> > > > > > > > > > > development >> > > > > >> > > > > > > > > > > > > > costs >> > > > > >> > > > > > > > > > > > > > > > > > moving forward. >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > Looking forward to hearing >> from >> > > you! >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > [1] >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > >> > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789560 >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > Best, >> > > > > >> > > > > > > > > > > > > > > > > > Zakelly >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > > >> > > > > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> >> > > > > >> -- >> > > > > >> Best, >> > > > > >> Hangxiang. >> > > > > >> >> > > > > > >> > > > > >> > > > >> > > > >> > > > -- >> > > > Best, >> > > > Hangxiang. >> > > >> > > >> > >> >
