Hi
Zuo and Lincoln
Thanks for your reply.
@Zuo
> detects whether the modified state is compatioble or not with the
previous state automatically ?
Automatic detection of state compatibility is technically feasible, but
currently, there is no ready-made interface to check whether the job before
and after modification is compatible. However, this issue may be beyond the
scope of this FLIP. Regarding the topic of Flink SQL state compatibility, I
believe a separate FLIP is needed to describe its behavior in more detail.
@Lincoln
> And for the default behavior of alter operation under continuous mode,
can you add an example of starting the new job with hints(similar to the
case in the User Story section)?
Thank you for the suggestion. The compatibility recovery for users is not
easy to judge. I think removing it is reasonable, as it is not suitable to
be provided as a public feature. I have already updated the relevant
content.
Best,
Feng Jin
On Thu, Dec 19, 2024 at 10:08 PM Lincoln Lee <lincoln.8...@gmail.com> wrote:
> Thanks Feng for driving this!
> Supporting modification is an important improvement for Materialized Table.
>
> Regarding the alter table reserve historic data, I have similar question
> with Ron,
> Users can't easily to judge whether a change is simple enough to keep state
> compatibility with old refresh job under the continuous mode. Therefore, I
> suggest removing the description of “Compatibile Recovery” from the public
> inferface section.
>
> And for the default behavior of alter operation under continuous mode, can
> you
> add an example of starting the new job with hints(similar to the case in
> the
> User Story section)?
>
> Best,
> Lincoln Lee
>
>
> Wei Zuo <1015766...@qq.com.invalid> 于2024年12月19日周四 14:05写道：
>
> > Hi, Feng
> >
> >
> > Is it possible that the framework detects whether the modified state is
> > compatioble or not with the previous state automatically? It would
> be
> > better to recognize query state compatibility automatically.
> >
> >
> > Best,
> >
> >
> > Zuo Wei
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人:
> > "dev"
> > <
> > ron9....@gmail.com>;
> > 发送时间: 2024年12月19日(星期四) 上午10:15
> > 收件人: "Feng Jin"<jinfeng1...@gmail.com>;
> > 抄送: "dev"<dev@flink.apache.org>;"Lincoln Lee"<
> > lincoln.8...@gmail.com>;
> > 主题: Re: [DISCUSS] FLIP-492: Support Query Modifications for
> > Materialized Tables.
> >
> >
> >
> > Hi, Feng
> >
> > The reply looks good to me. But I have one question: You mentioned the
> > `DESC MATERIALIZED TABLE` syntax in FLIP, but we didn't provide this
> syntax
> > until now. I think we should add it to this FLIP if needed.
> >
> > Best,
> > Ron
> >
> > Feng Jin <jinfeng1...@gmail.com> 于2024年12月18日周三 16:52写道：
> >
> > > Hi Ron
> > >
> > > Thanks for your reply.
> > >
> > > > Is it only possible to add columns at the end and not
> > anywhere in
> > > table schema, some databases have this limitation, does lake storage
> > such
> > > as Iceberg/Paimon have this limitation?
> > >
> > >
> > > Currently, we can restrict adding columns only to the end of
> > the schema.
> > > Although both Paimon and Iceberg already support adding columns
> > anywhere,
> > > there are still some systems that do not. I will include this in the
> > FLIP.
> > >
> > >
> > > > In the Refresh Task Behavior section you mention partition
> > hints, is it
> > > possible to clarify what it is in the FLIP？
> > >
> > >
> > > I have added the relevant details.
> > >
> > >
> > > > Are you able to articulate the default behavior?
> > >
> > >
> > > The detailed explanation for this part has been updated.
> > >
> > >
> > > > How users can determine if states are compatible？
> > >
> > >
> > > Users can only rely on their experience to make modifications.
> > Currently,
> > > the Flink framework does not guarantee that changes to SQL logic
> will
> > > maintain state compatibility.
> > >
> > > I think we can add some suggestions in the user documentation in the
> > > future. While the framework itself cannot ensure state
> compatibility,
> > some
> > > simple modification scenarios can indeed be compatible.
> > >
> > > For now, the responsibility is left to the users.
> > >
> > >
> > > Even if recovery ultimately fails, users still have the option to
> roll
> > > back to the original query or start consuming from a new offset by
> > > disabling recovery parameters.
> > >
> > >
> > >
> > >
> > > Best,
> > > Feng
> > >
> > >
> > > On Tue, Dec 17, 2024 at 10:37 AM Ron Liu <ron9....@gmail.com>
> > wrote:
> > >
> > >> Hi Feng
> > >>
> > >> Thanks for initiating this FLIP, in lakehouse, Schema Evolution
> > of tables
> > >> due to modification of business logic is a very common scenario,
> > so
> > >> Materialized Table's support for modification of Query can
> > greatly improve
> > >> flexibility and usability, and we've seen that other similar
> > products in
> > >> the industry also support this capability.
> > >>
> > >> I read the content of this FLIP and the overall design looks
> > good, +1.
> > >> However, I have some questions as follows:
> > >>
> > >> 1. By `ALTER MATERIALIZED TABLE ... AS select` statement to
> > realize the
> > >> add column logic, is it only possible to add columns at the end
> > and not
> > >> anywhere in table schema, some databases have this limitation,
> > does lake
> > >> storage such as Iceberg/Paimon have this limitation?
> > >> 2. In the Refresh Task Behavior section you mention partition
> > hints, is
> > >> it possible to clarify what it is in the FLIP？
> > >>
> > >> >>> *CONTINUOUS Mode: *Stops the old job and starts a
> > new one with the
> > >> updated query.
> > >>
> > >> - The initial position of the new job is
> > controlled by the source
> > >> parameters.
> > >> - For compatible logic changes, recovery
> > parameters
> > >> (execution.state-recovery.path) can be
> > manually set if state compatibility
> > >> is confirmed.
> > >>
> > >>
> > >> 4. Are you able to articulate the default behavior?
> > >> 5. How users can determine if states are compatible？
> > >>
> > >> Best,
> > >> Ron
> > >>
> > >> Feng Jin <jinfeng1...@gmail.com> 于2024年12月16日周一 10:49写道：
> > >>
> > >>> Hi, everyone,
> > >>>
> > >>> I’d like to initiate a discussion on FLIP-492: Support Query
> > >>> Modifications for Materialized Tables[1].
> > >>>
> > >>> In FLIP-435[2], we introduced *MATERIALIZED TABLES*. By
> > defining query
> > >>> logic and specifying data freshness requirements, users can
> > efficiently
> > >>> build data pipelines, greatly improving development
> > productivity.
> > >>> FLIP-492 builds on this by addressing a common need: the
> > ability to
> > >>> modify the query logic of an existing MATERIALIZED TABLE.
> Two
> > approaches
> > >>> are proposed:
> > >>>
> > >>>
> > >>> *1. Modifying the Query Logic: ALTER MATERIALIZED TABLE AS
> > <query>*
> > >>> Retain historical data while modifying the query logic:
> > >>>
> > >>> ```
> > >>> ALTER MATERIALIZED TABLE [catalog_name.][db_name.]table_name
> > AS <query>
> > >>> ```
> > >>>
> > >>>
> > >>> *2. Replacing the Table: CREATE OR REPLACE MATERIALIZED
> TABLE*
> > >>> Reconstruct the table with a new query, discarding all
> > historical data:
> > >>>
> > >>> ```
> > >>> CREATE [OR REPLACE] MATERIALIZED TABLE
> > >>> [catalog_name.][db_name.]table_name
> > >>> [ ([<table_constraint>]) ]
> > >>> [COMMENT table_comment]
> > >>> [PARTITIONED BY (partition_column_name1,
> > partition_column_name2, ...)]
> > >>> [WITH (key1=val1, key2=val2, ...)]
> > >>> FRESHNESS = INTERVAL '<num>' { SECOND | MINUTE | HOUR |
> > DAY }
> > >>> [REFRESH_MODE = { CONTINUOUS | FULL }]
> > >>> AS <select_statement>
> > >>> ```
> > >>>
> > >>> For a more detailed explanation of this proposal, please
> > refer to the
> > >>> FLIP-492[1] documentation.
> > >>> Your feedback and suggestions are highly appreciated to help
> > refine this
> > >>> proposal further.
> > >>>
> > >>> Lastly, I’d like to thank Ron and Lincoln (cc’d) for their
> > valuable
> > >>> input and suggestions during the drafting process.
> > >>>
> > >>> Looking forward to hearing your thoughts!
> > >>>
> > >>>
> > >>> Best,
> > >>> Feng Jin
> > >>>
> > >>>
> > >>> [1].
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-492%3A+Support+Query+Modifications+for+Materialized+Tables
> > >>>
> > <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-492%3A+Support+Query+Modifications+for+Materialized+Tables>>>
> >;
> > [2].
> > >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-435%3A+Introduce+a+New+Materialized+Table+for+Simplifying+Data+Pipelines
> > >>>
> > <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-435%3A+Introduce+a+New+Materialized+Table+for+Simplifying+Data+Pipelines>>>
> >
> > ;
> > >>
>
