Hi, Ramin Thank you very much for initiating this FLIP. The proposed extension to the existing syntax is highly valuable and will significantly improve the user experience.
After carefully reviewing the FLIP, I have two questions: 1. Are the START_MODE and STATE_RETENTION clauses applicable only to CONTINUOUS mode? In FULL mode, since the job is stateless, do these clauses take effect at all? 2. Regarding the STATE_RETENTION clause: from your current design, it appears to control whether the state from operators *other than* the SourceOperator is retained. However, the source offset itself is also part of the job’s state. From a state management perspective, do START_MODE and STATE_RETENTION exhibit some functional overlap or ambiguity? For example, if STATE_RETENTION = NONE it means discard *all* state—including the source offset—then when START_MODE = RESUME_OR_FROM_NOW, would it still be possible to resume from the corresponding source offset? Best, Ron Timo Walther <[email protected]> 于2025年12月18日周四 19:46写道： > Hi Ramin, > > thank you for proposing this FLIP. It nicely extends the existing > semantics and improves the experience esp. for the CONTINUOUS mode. But > even in FULL mode, people don't want to pay for full processing and > would like to have more control over what will be reprocessed in case of > a query evolution. Sometimes also to skip malformed data. > > I was heavily involved in the FLIP discussions and it took us quite a > few iterations to end up with this concise syntax that nicely fits into > Flink semantics of savepoints and time travelling (FLIP-308), while > keeping new developments such as PTFs (FLIP-440) in the picture as well. > > +1 to this proposal. > > Cheers, > Timo > > > On 01.12.25 18:23, Ramin Gharib wrote: > > Hi everyone, > > > > I would like to start a discussion on FLIP-557: Granular Control over > Data > > Reprocessing and State Retention in Materialized Table Evolution [1]. > > > > Currently, ALTER MATERIALIZED TABLE forces a full job restart and > discards > > state, which is inefficient for many evolution scenarios. FLIP-557 > proposes > > decoupling data scope from state management by introducing two new > optional > > clauses: > > 1. START_MODE*:* Controls the data processing window (e.g., > FROM_BEGINNING, > > RESUME_OR_...). > > > > 2. STATE_RETENTION*:* Controls how existing state is handled (e.g., NONE, > > PTF_ONLY). > > > > This gives users explicit control over cost and correctness during table > > evolution. > > > > For more details, please refer to the FLIP [1]. > > > > Looking forward to your feedback and thoughts! > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-557%3A+Granular+Control+over+Data+Reprocessing+and+State+Retention+in+Materialized+Table+Evolution > > > > Best regards, > > > > Ramin Gharib > > > >
