Thanks Ron and Timo for your proposal!
Here is my ranking:
1. Derived table -> extend the persistent semantics of derived table in SQL
standard, with a strong association with query, and has industry
precedents
such as Google Looker.
2. Live Table -> an alternative for 'dynamic table'
3. Materialized Table -> combination of the Materialized View and Table,
but
still a table which accept data changes
4. Materialized View -> need to extend understanding of the view to accept
data changes
The reason for not adding 'Refresh Table' is I don't want to tell the user
to 'refresh a refresh table'.
Best,
Lincoln Lee
Ron liu <ron9....@gmail.com> 于2024年4月9日周二 20:11写道：
> Hi, Dev
>
> My rankings are:
>
> 1. Derived Table
> 2. Materialized Table
> 3. Live Table
> 4. Materialized View
>
> Best,
> Ron
>
>
>
> Ron liu <ron9....@gmail.com> 于2024年4月9日周二 20:07写道：
>
> > Hi, Dev
> >
> > After several rounds of discussion, there is currently no consensus on
> the
> > name of the new concept. Timo has proposed that we decide the name
> through
> > a vote. This is a good solution when there is no clear preference, so we
> > will adopt this approach.
> >
> > Regarding the name of the new concept, there are currently five
> candidates:
> > 1. Derived Table -> taken by SQL standard
> > 2. Materialized Table -> similar to SQL materialized view but a table
> > 3. Live Table -> similar to dynamic tables
> > 4. Refresh Table -> states what it does
> > 5. Materialized View -> needs to extend the standard to support modifying
> > data
> >
> > For the above five candidates, everyone can give your rankings based on
> > your preferences. You can choose up to five options or only choose some
> of
> > them.
> > We will use a scoring rule, where the* first rank gets 5 points, second
> > rank gets 4 points, third rank gets 3 points, fourth rank gets 2 points,
> > and fifth rank gets 1 point*.
> > After the voting closes, I will score all the candidates based on
> > everyone's votes, and the candidate with the highest score will be chosen
> > as the name for the new concept.
> >
> > The voting will last up to 72 hours and is expected to close this Friday.
> > I look forward to everyone voting on the name in this thread. Of course,
> we
> > also welcome new input regarding the name.
> >
> > Best,
> > Ron
> >
> > Ron liu <ron9....@gmail.com> 于2024年4月9日周二 19:49写道：
> >
> >> Hi, Dev
> >>
> >> Sorry for my previous statement was not quite accurate. We will hold a
> >> vote for the name within this thread.
> >>
> >> Best,
> >> Ron
> >>
> >>
> >> Ron liu <ron9....@gmail.com> 于2024年4月9日周二 19:29写道：
> >>
> >>> Hi, Timo
> >>>
> >>> Thanks for your reply.
> >>>
> >>> I agree with you that sometimes naming is more difficult. When no one
> >>> has a clear preference, voting on the name is a good solution, so I'll
> send
> >>> a separate email for the vote, clarify the rules for the vote, then let
> >>> everyone vote.
> >>>
> >>> One other point to confirm, in your ranking there is an option for
> >>> Materialized View, does it stand for the UPDATING Materialized View
> that
> >>> you mentioned earlier in the discussion? If using Materialized View I
> think
> >>> it is needed to extend it.
> >>>
> >>> Best,
> >>> Ron
> >>>
> >>> Timo Walther <twal...@apache.org> 于2024年4月9日周二 17:20写道：
> >>>
> >>>> Hi Ron,
> >>>>
> >>>> yes naming is hard. But it will have large impact on trainings,
> >>>> presentations, and the mental model of users. Maybe the easiest is to
> >>>> collect ranking by everyone with some short justification:
> >>>>
> >>>>
> >>>> My ranking (from good to not so good):
> >>>>
> >>>> 1. Refresh Table -> states what it does
> >>>> 2. Materialized Table -> similar to SQL materialized view but a table
> >>>> 3. Live Table -> nice buzzword, but maybe still too close to dynamic
> >>>> tables?
> >>>> 4. Materialized View -> a bit broader than standard but still very
> >>>> similar
> >>>> 5. Derived table -> taken by standard
> >>>>
> >>>> Regards,
> >>>> Timo
> >>>>
> >>>>
> >>>>
> >>>> On 07.04.24 11:34, Ron liu wrote:
> >>>> > Hi, Dev
> >>>> >
> >>>> > This is a summary letter. After several rounds of discussion, there
> >>>> is a
> >>>> > strong consensus about the FLIP proposal and the issues it aims to
> >>>> address.
> >>>> > The current point of disagreement is the naming of the new concept.
> I
> >>>> have
> >>>> > summarized the candidates as follows:
> >>>> >
> >>>> > 1. Derived Table (Inspired by Google Lookers)
> >>>> > - Pros: Google Lookers has introduced this concept, which is
> >>>> designed
> >>>> > for building Looker's automated modeling, aligning with our purpose
> >>>> for the
> >>>> > stream-batch automatic pipeline.
> >>>> >
> >>>> > - Cons: The SQL standard uses derived table term extensively,
> >>>> vendors
> >>>> > adopt this for simply referring to a table within a subclause.
> >>>> >
> >>>> > 2. Materialized Table: It means materialize the query result to
> table,
> >>>> > similar to Db2 MQT (Materialized Query Tables). In addition,
> Snowflake
> >>>> > Dynamic Table's predecessor is also called Materialized Table.
> >>>> >
> >>>> > 3. Updating Table (From Timo)
> >>>> >
> >>>> > 4. Updating Materialized View (From Timo)
> >>>> >
> >>>> > 5. Refresh/Live Table (From Martijn)
> >>>> >
> >>>> > As Martijn said, naming is a headache, looking forward to more
> >>>> valuable
> >>>> > input from everyone.
> >>>> >
> >>>> > [1]
> >>>> >
> >>>>
> https://cloud.google.com/looker/docs/derived-tables#persistent_derived_tables
> >>>> > [2]
> >>>> https://www.ibm.com/docs/en/db2/11.5?topic=tables-materialized-query
> >>>> > [3]
> >>>> >
> >>>>
> https://community.denodo.com/docs/html/browse/6.0/vdp/vql/materialized_tables/creating_materialized_tables/creating_materialized_tables
> >>>> >
> >>>> > Best,
> >>>> > Ron
> >>>> >
> >>>> > Ron liu <ron9....@gmail.com> 于2024年4月7日周日 15:55写道：
> >>>> >
> >>>> >> Hi, Lorenzo
> >>>> >>
> >>>> >> Thank you for your insightful input.
> >>>> >>
> >>>> >>>>> I think the 2 above twisted the materialized view concept to
> more
> >>>> than
> >>>> >> just an optimization for accessing pre-computed aggregates/filters.
> >>>> >> I think that concept (at least in my mind) is now adherent to the
> >>>> >> semantics of the words themselves ("materialized" and "view") than
> >>>> on its
> >>>> >> implementations in DBMs, as just a view on raw data that,
> hopefully,
> >>>> is
> >>>> >> constantly updated with fresh results.
> >>>> >> That's why I understand Timo's et al. objections.
> >>>> >>
> >>>> >> Your understanding of Materialized Views is correct. However, in
> our
> >>>> >> scenario, an important feature is the support for Update & Delete
> >>>> >> operations, which the current Materialized Views cannot fulfill. As
> >>>> we
> >>>> >> discussed with Timo before, if Materialized Views needs to support
> >>>> data
> >>>> >> modifications, it would require an extension of new keywords, such
> as
> >>>> >> CREATING xxx (UPDATING) MATERIALIZED VIEW.
> >>>> >>
> >>>> >>>>> Still, I don't understand why we need another type of special
> >>>> table.
> >>>> >> Could you dive deep into the reasons why not simply adding the
> >>>> FRESHNESS
> >>>> >> parameter to standard tables?
> >>>> >>
> >>>> >> Firstly, I need to emphasize that we cannot achieve the design goal
> >>>> of
> >>>> >> FLIP through the CREATE TABLE syntax combined with a FRESHNESS
> >>>> parameter.
> >>>> >> The proposal of this FLIP is to use Dynamic Table + Continuous
> >>>> Query, and
> >>>> >> combine it with FRESHNESS to realize a streaming-batch unification.
> >>>> >> However, CREATE TABLE is merely a metadata operation and cannot
> >>>> >> automatically start a background refresh job. To achieve the design
> >>>> goal of
> >>>> >> FLIP with standard tables, it would require extending the CTAS[1]
> >>>> syntax to
> >>>> >> introduce the FRESHNESS keyword. We considered this design
> >>>> initially, but
> >>>> >> it has following problems:
> >>>> >>
> >>>> >> 1. Distinguishing a table created through CTAS as a standard table
> >>>> or as a
> >>>> >> "special" standard table with an ongoing background refresh job
> >>>> using the
> >>>> >> FRESHNESS keyword is very obscure for users.
> >>>> >> 2. It intrudes on the semantics of the CTAS syntax. Currently,
> tables
> >>>> >> created using CTAS only add table metadata to the Catalog and do
> not
> >>>> record
> >>>> >> attributes such as query. There are also no ongoing background
> >>>> refresh
> >>>> >> jobs, and the data writing operation happens only once at table
> >>>> creation.
> >>>> >> 3. For the framework, when we perform a certain kind of Alter Table
> >>>> >> behavior for a table, for the table created by specifying FRESHNESS
> >>>> and did
> >>>> >> not specify the FRESHNESS created table behavior how to distinguish
> >>>> , which
> >>>> >> will also cause confusion.
> >>>> >>
> >>>> >> In terms of the design goal of combining Dynamic Table + Continuous
> >>>> Query,
> >>>> >> the FLIP proposal cannot be realized by only extending the current
> >>>> stardand
> >>>> >> tables, so a new kind of dynamic table needs to be introduced at
> the
> >>>> >> first-level concept.
> >>>> >>
> >>>> >> [1]
> >>>> >>
> >>>>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#as-select_statement
> >>>> >>
> >>>> >> Best,
> >>>> >> Ron
> >>>> >>
> >>>> >> <lorenzo.affe...@ververica.com.invalid> 于2024年4月3日周三 22:25写道：
> >>>> >>
> >>>> >>> Hello everybody!
> >>>> >>> Thanks for the FLIP as it looks amazing (and I think the prove is
> >>>> this
> >>>> >>> deep discussion it is provoking :))
> >>>> >>>
> >>>> >>> I have a couple of comments to add to this:
> >>>> >>>
> >>>> >>> Even though I get the reason why you rejected MATERIALIZED VIEW, I
> >>>> still
> >>>> >>> like it a lot, and I would like to provide pointers on how the
> >>>> materialized
> >>>> >>> view concept twisted in last years:
> >>>> >>>
> >>>> >>> • Materialize DB (https://materialize.com/)
> >>>> >>> • The famous talk by Martin Kleppmann "turning the database inside
> >>>> out" (
> >>>> >>> https://www.youtube.com/watch?v=fU9hR3kiOK0)
> >>>> >>>
> >>>> >>> I think the 2 above twisted the materialized view concept to more
> >>>> than
> >>>> >>> just an optimization for accessing pre-computed
> aggregates/filters.
> >>>> >>> I think that concept (at least in my mind) is now adherent to the
> >>>> >>> semantics of the words themselves ("materialized" and "view") than
> >>>> on its
> >>>> >>> implementations in DBMs, as just a view on raw data that,
> >>>> hopefully, is
> >>>> >>> constantly updated with fresh results.
> >>>> >>> That's why I understand Timo's et al. objections.
> >>>> >>> Still I understand there is no need to add confusion :)
> >>>> >>>
> >>>> >>> Still, I don't understand why we need another type of special
> table.
> >>>> >>> Could you dive deep into the reasons why not simply adding the
> >>>> FRESHNESS
> >>>> >>> parameter to standard tables?
> >>>> >>>
> >>>> >>> I would say that as a very seamless implementation with the goal
> of
> >>>> a
> >>>> >>> unification of batch and streaming.
> >>>> >>> If we stick to a unified world, I think that Flink should just
> >>>> provide 1
> >>>> >>> type of table that is inherently dynamic.
> >>>> >>> Now, depending on FRESHNESS objectives / connectors used in WITH,
> >>>> that
> >>>> >>> table can be backed by a stream or batch job as you explained in
> >>>> your FLIP.
> >>>> >>>
> >>>> >>> Maybe I am totally missing the point :)
> >>>> >>>
> >>>> >>> Thank you in advance,
> >>>> >>> Lorenzo
> >>>> >>> On Apr 3, 2024 at 15:25 +0200, Martijn Visser <
> >>>> martijnvis...@apache.org>,
> >>>> >>> wrote:
> >>>> >>>> Hi all,
> >>>> >>>>
> >>>> >>>> Thanks for the proposal. While the FLIP talks extensively on how
> >>>> >>> Snowflake
> >>>> >>>> has Dynamic Tables and Databricks has Delta Live Tables, my
> >>>> >>> understanding
> >>>> >>>> is that Databricks has CREATE STREAMING TABLE [1] which relates
> >>>> with
> >>>> >>> this
> >>>> >>>> proposal.
> >>>> >>>>
> >>>> >>>> I do have concerns about using CREATE DYNAMIC TABLE, specifically
> >>>> about
> >>>> >>>> confusing the users who are familiar with Snowflake's approach
> >>>> where you
> >>>> >>>> can't change the content via DML statements, while that is
> >>>> something
> >>>> >>> that
> >>>> >>>> would work in this proposal. Naming is hard of course, but I
> would
> >>>> >>> probably
> >>>> >>>> prefer something like CREATE CONTINUOUS TABLE, CREATE REFRESH
> >>>> TABLE or
> >>>> >>>> CREATE LIVE TABLE.
> >>>> >>>>
> >>>> >>>> Best regards,
> >>>> >>>>
> >>>> >>>> Martijn
> >>>> >>>>
> >>>> >>>> [1]
> >>>> >>>>
> >>>> >>>
> >>>>
> https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-streaming-table.html
> >>>> >>>>
> >>>> >>>> On Wed, Apr 3, 2024 at 5:19 AM Ron liu <ron9....@gmail.com>
> wrote:
> >>>> >>>>
> >>>> >>>>> Hi, dev
> >>>> >>>>>
> >>>> >>>>> After offline discussion with Becket Qin, Lincoln Lee and Jark
> >>>> Wu, we
> >>>> >>> have
> >>>> >>>>> improved some parts of the FLIP.
> >>>> >>>>>
> >>>> >>>>> 1. Add Full Refresh Mode section to clarify the semantics of
> full
> >>>> >>> refresh
> >>>> >>>>> mode.
> >>>> >>>>> 2. Add Future Improvement section explaining why query statement
> >>>> does
> >>>> >>> not
> >>>> >>>>> support references to temporary view and possible solutions.
> >>>> >>>>> 3. The Future Improvement section explains a possible future
> >>>> solution
> >>>> >>> for
> >>>> >>>>> dynamic table to support the modification of query statements to
> >>>> meet
> >>>> >>> the
> >>>> >>>>> common field-level schema evolution requirements of the
> lakehouse.
> >>>> >>>>> 4. The Refresh section emphasizes that the Refresh command and
> the
> >>>> >>>>> background refresh job can be executed in parallel, with no
> >>>> >>> restrictions at
> >>>> >>>>> the framework level.
> >>>> >>>>> 5. Convert RefreshHandler into a plug-in interface to support
> >>>> various
> >>>> >>>>> workflow schedulers.
> >>>> >>>>>
> >>>> >>>>> Best,
> >>>> >>>>> Ron
> >>>> >>>>>
> >>>> >>>>> Ron liu <ron9....@gmail.com> 于2024年4月2日周二 10:28写道：
> >>>> >>>>>
> >>>> >>>>>>> Hi, Venkata krishnan
> >>>> >>>>>>>
> >>>> >>>>>>> Thank you for your involvement and suggestions, and hope that
> >>>> the
> >>>> >>> design
> >>>> >>>>>>> goals of this FLIP will be helpful to your business.
> >>>> >>>>>>>
> >>>> >>>>>>>>>>>>> 1. In the proposed FLIP, given the example for the
> >>>> >>> dynamic table, do
> >>>> >>>>>>> the
> >>>> >>>>>>> data sources always come from a single lake storage such as
> >>>> >>> Paimon or
> >>>> >>>>> does
> >>>> >>>>>>> the same proposal solve for 2 disparate storage systems like
> >>>> >>> Kafka and
> >>>> >>>>>>> Iceberg where Kafka events are ETLed to Iceberg similar to
> >>>> Paimon?
> >>>> >>>>>>> Basically the lambda architecture that is mentioned in the
> FLIP
> >>>> >>> as well.
> >>>> >>>>>>> I'm wondering if it is possible to switch b/w sources based on
> >>>> the
> >>>> >>>>>>> execution mode, for eg: if it is backfill operation, switch
> to a
> >>>> >>> data
> >>>> >>>>> lake
> >>>> >>>>>>> storage system like Iceberg, otherwise an event streaming
> system
> >>>> >>> like
> >>>> >>>>>>> Kafka.
> >>>> >>>>>>>
> >>>> >>>>>>> Dynamic table is a design abstraction at the framework level
> and
> >>>> >>> is not
> >>>> >>>>>>> tied to the physical implementation of the connector. If a
> >>>> >>> connector
> >>>> >>>>>>> supports a combination of Kafka and lake storage, this works
> >>>> fine.
> >>>> >>>>>>>
> >>>> >>>>>>>>>>>>> 2. What happens in the context of a bootstrap (batch) +
> >>>> >>> nearline
> >>>> >>>>> update
> >>>> >>>>>>> (streaming) case that are stateful applications? What I mean
> by
> >>>> >>> that is,
> >>>> >>>>>>> will the state from the batch application be transferred to
> the
> >>>> >>> nearline
> >>>> >>>>>>> application after the bootstrap execution is complete?
> >>>> >>>>>>>
> >>>> >>>>>>> I think this is another orthogonal thing, something that
> >>>> FLIP-327
> >>>> >>> tries
> >>>> >>>>> to
> >>>> >>>>>>> address, not directly related to Dynamic Table.
> >>>> >>>>>>>
> >>>> >>>>>>> [1]
> >>>> >>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-327%3A+Support+switching+from+batch+to+stream+mode+to+improve+throughput+when+processing+backlog+data
> >>>> >>>>>>>
> >>>> >>>>>>> Best,
> >>>> >>>>>>> Ron
> >>>> >>>>>>>
> >>>> >>>>>>> Venkatakrishnan Sowrirajan <vsowr...@asu.edu> 于2024年3月30日周六
> >>>> >>> 07:06写道：
> >>>> >>>>>>>
> >>>> >>>>>>>>> Ron and Lincoln,
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> Great proposal and interesting discussion for adding support
> >>>> >>> for dynamic
> >>>> >>>>>>>>> tables within Flink.
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> At LinkedIn, we are also trying to solve compute/storage
> >>>> >>> convergence for
> >>>> >>>>>>>>> similar problems discussed as part of this FLIP,
> specifically
> >>>> >>> periodic
> >>>> >>>>>>>>> backfill, bootstrap + nearline update use cases using single
> >>>> >>>>>>>>> implementation
> >>>> >>>>>>>>> of business logic (single script).
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> Few clarifying questions:
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> 1. In the proposed FLIP, given the example for the dynamic
> >>>> >>> table, do the
> >>>> >>>>>>>>> data sources always come from a single lake storage such as
> >>>> >>> Paimon or
> >>>> >>>>> does
> >>>> >>>>>>>>> the same proposal solve for 2 disparate storage systems like
> >>>> >>> Kafka and
> >>>> >>>>>>>>> Iceberg where Kafka events are ETLed to Iceberg similar to
> >>>> >>> Paimon?
> >>>> >>>>>>>>> Basically the lambda architecture that is mentioned in the
> >>>> >>> FLIP as well.
> >>>> >>>>>>>>> I'm wondering if it is possible to switch b/w sources based
> on
> >>>> >>> the
> >>>> >>>>>>>>> execution mode, for eg: if it is backfill operation, switch
> to
> >>>> >>> a data
> >>>> >>>>> lake
> >>>> >>>>>>>>> storage system like Iceberg, otherwise an event streaming
> >>>> >>> system like
> >>>> >>>>>>>>> Kafka.
> >>>> >>>>>>>>> 2. What happens in the context of a bootstrap (batch) +
> >>>> >>> nearline update
> >>>> >>>>>>>>> (streaming) case that are stateful applications? What I mean
> >>>> >>> by that is,
> >>>> >>>>>>>>> will the state from the batch application be transferred to
> >>>> >>> the nearline
> >>>> >>>>>>>>> application after the bootstrap execution is complete?
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> Regards
> >>>> >>>>>>>>> Venkata krishnan
> >>>> >>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>>>>> On Mon, Mar 25, 2024 at 8:03 PM Ron liu <ron9....@gmail.com
> >
> >>>> >>> wrote:
> >>>> >>>>>>>>>
> >>>> >>>>>>>>>>> Hi, Timo
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> Thanks for your quick response, and your suggestion.
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> Yes, this discussion has turned into confirming whether
> >>>> >>> it's a special
> >>>> >>>>>>>>>>> table or a special MV.
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> 1. The key problem with MVs is that they don't support
> >>>> >>> modification,
> >>>> >>>>> so
> >>>> >>>>>>>>> I
> >>>> >>>>>>>>>>> prefer it to be a special table. Although the periodic
> >>>> >>> refresh
> >>>> >>>>> behavior
> >>>> >>>>>>>>> is
> >>>> >>>>>>>>>>> more characteristic of an MV, since we are already a
> >>>> >>> special table,
> >>>> >>>>>>>>>>> supporting periodic refresh behavior is quite natural,
> >>>> >>> similar to
> >>>> >>>>>>>>> Snowflake
> >>>> >>>>>>>>>>> dynamic tables.
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> 2. Regarding the keyword UPDATING, since the current
> >>>> >>> Regular Table is
> >>>> >>>>> a
> >>>> >>>>>>>>>>> Dynamic Table, which implies support for updating through
> >>>> >>> Continuous
> >>>> >>>>>>>>> Query,
> >>>> >>>>>>>>>>> I think it is redundant to add the keyword UPDATING. In
> >>>> >>> addition,
> >>>> >>>>>>>>> UPDATING
> >>>> >>>>>>>>>>> can not reflect the Continuous Query part, can not express
> >>>> >>> the purpose
> >>>> >>>>>>>>> we
> >>>> >>>>>>>>>>> want to simplify the data pipeline through Dynamic Table +
> >>>> >>> Continuous
> >>>> >>>>>>>>>>> Query.
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> 3. From the perspective of the SQL standard definition, I
> >>>> >>> can
> >>>> >>>>> understand
> >>>> >>>>>>>>>>> your concerns about Derived Table, but is it possible to
> >>>> >>> make a slight
> >>>> >>>>>>>>>>> adjustment to meet our needs? Additionally, as Lincoln
> >>>> >>> mentioned, the
> >>>> >>>>>>>>>>> Google Looker platform has introduced Persistent Derived
> >>>> >>> Table, and
> >>>> >>>>>>>>> there
> >>>> >>>>>>>>>>> are precedents in the industry; could Derived Table be a
> >>>> >>> candidate?
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> Of course, look forward to your better suggestions.
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> Best,
> >>>> >>>>>>>>>>> Ron
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>> Timo Walther <twal...@apache.org> 于2024年3月25日周一 18:49写道：
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>>>>> After thinking about this more, this discussion boils
> >>>> >>> down to
> >>>> >>>>> whether
> >>>> >>>>>>>>>>>>> this is a special table or a special materialized
> >>>> >>> view. In both
> >>>> >>>>> cases,
> >>>> >>>>>>>>>>>>> we would need to add a special keyword:
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>> Either
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>> CREATE UPDATING TABLE
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>> or
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>> CREATE UPDATING MATERIALIZED VIEW
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>> I still feel that the periodic refreshing behavior is
> >>>> >>> closer to a
> >>>> >>>>> MV.
> >>>> >>>>>>>>> If
> >>>> >>>>>>>>>>>>> we add a special keyword to MV, the optimizer would
> >>>> >>> know that the
> >>>> >>>>> data
> >>>> >>>>>>>>>>>>> cannot be used for query optimizations.
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>> I will ask more people for their opinion.
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>> Regards,
> >>>> >>>>>>>>>>>>> Timo
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>> On 25.03.24 10:45, Timo Walther wrote:
> >>>> >>>>>>>>>>>>>>> Hi Ron and Lincoln,
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> thanks for the quick response and the very
> >>>> >>> insightful discussion.
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>> we might limit future opportunities to
> >>>> >>> optimize queries
> >>>> >>>>>>>>>>>>>>>>> through automatic materialization rewriting by
> >>>> >>> allowing data
> >>>> >>>>>>>>>>>>>>>>> modifications, thus losing the potential for
> >>>> >>> such
> >>>> >>>>> optimizations.
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> This argument makes a lot of sense to me. Due to
> >>>> >>> the updates, the
> >>>> >>>>>>>>>>> system
> >>>> >>>>>>>>>>>>>>> is not in full control of the persisted data.
> >>>> >>> However, the system
> >>>> >>>>> is
> >>>> >>>>>>>>>>>>>>> still in full control of the job that powers the
> >>>> >>> refresh. So if
> >>>> >>>>> the
> >>>> >>>>>>>>>>>>>>> system manages all updating pipelines, it could
> >>>> >>> still leverage
> >>>> >>>>>>>>>>> automatic
> >>>> >>>>>>>>>>>>>>> materialization rewriting but without leveraging
> >>>> >>> the data at rest
> >>>> >>>>>>>>> (only
> >>>> >>>>>>>>>>>>>>> the data in flight).
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>> we are considering another candidate, Derived
> >>>> >>> Table, the term
> >>>> >>>>>>>>>>> 'derive'
> >>>> >>>>>>>>>>>>>>>>> suggests a query, and 'table' retains
> >>>> >>> modifiability. This
> >>>> >>>>>>>>> approach
> >>>> >>>>>>>>>>>>>>>>> would not disrupt our current concept of a
> >>>> >>> dynamic table
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> I did some research on this term. The SQL standard
> >>>> >>> uses the term
> >>>> >>>>>>>>>>>>>>> "derived table" extensively (defined in section
> >>>> >>> 4.17.3). Thus, a
> >>>> >>>>>>>>> lot of
> >>>> >>>>>>>>>>>>>>> vendors adopt this for simply referring to a table
> >>>> >>> within a
> >>>> >>>>>>>>> subclause:
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://dev.mysql.com/doc/refman/8.0/en/derived-tables.html__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j735ghdiMp$
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc32300.1600/doc/html/san1390612291252.html__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j737h1gRux$
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://www.c-sharpcorner.com/article/derived-tables-vs-common-table-expressions/__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j739bWIEcL$
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://stackoverflow.com/questions/26529804/what-are-the-derived-tables-in-my-explain-statement__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j739HnGtQf$
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://www.sqlservercentral.com/articles/sql-derived-tables__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j737DeBiqg$
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> Esp. the latter example is interesting, SQL Server
> >>>> >>> allows things
> >>>> >>>>>>>>> like
> >>>> >>>>>>>>>>>>>>> this on derived tables:
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> UPDATE T SET Name='Timo' FROM (SELECT * FROM
> >>>> >>> Product) AS T
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> SELECT * FROM Product;
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> Btw also Snowflake's dynamic table state:
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>> Because the content of a dynamic table is
> >>>> >>> fully determined
> >>>> >>>>>>>>>>>>>>>>> by the given query, the content cannot be
> >>>> >>> changed by using DML.
> >>>> >>>>>>>>>>>>>>>>> You don’t insert, update, or delete the rows
> >>>> >>> in a dynamic
> >>>> >>>>> table.
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> So a new term makes a lot of sense.
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> How about using `UPDATING`?
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> CREATE UPDATING TABLE
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> This reflects that modifications can be made and
> >>>> >>> from an
> >>>> >>>>>>>>>>>>>>> English-language perspective you can PAUSE or
> >>>> >>> RESUME the UPDATING.
> >>>> >>>>>>>>>>>>>>> Thus, a user can define UPDATING interval and mode?
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> Looking forward to your thoughts.
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> Regards,
> >>>> >>>>>>>>>>>>>>> Timo
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>> On 25.03.24 07:09, Ron liu wrote:
> >>>> >>>>>>>>>>>>>>>>> Hi, Ahmed
> >>>> >>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>> Thanks for your feedback.
> >>>> >>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>> Regarding your question:
> >>>> >>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>> I want to iterate on Timo's comments
> >>>> >>> regarding the confusion
> >>>> >>>>>>>>> between
> >>>> >>>>>>>>>>>>>>>>> "Dynamic Table" and current Flink "Table".
> >>>> >>> Should the refactoring
> >>>> >>>>>>>>> of
> >>>> >>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>> system happen in 2.0, should we rename it in
> >>>> >>> this Flip ( as the
> >>>> >>>>>>>>>>>>>>>>> suggestions
> >>>> >>>>>>>>>>>>>>>>> in the thread ) and address the holistic
> >>>> >>> changes in a separate
> >>>> >>>>> Flip
> >>>> >>>>>>>>>>>>>>>>> for 2.0?
> >>>> >>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>> Lincoln proposed a new concept in reply to
> >>>> >>> Timo: Derived Table,
> >>>> >>>>>>>>> which
> >>>> >>>>>>>>>>>>>>>>> is a
> >>>> >>>>>>>>>>>>>>>>> combination of Dynamic Table + Continuous
> >>>> >>> Query, and the use of
> >>>> >>>>>>>>>>> Derived
> >>>> >>>>>>>>>>>>>>>>> Table will not conflict with existing concepts,
> >>>> >>> what do you
> >>>> >>>>> think?
> >>>> >>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>> I feel confused with how it is further with
> >>>> >>> other components,
> >>>> >>>>> the
> >>>> >>>>>>>>>>>>>>>>> examples provided feel like a standalone ETL
> >>>> >>> job, could you
> >>>> >>>>>>>>> provide in
> >>>> >>>>>>>>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>> FLIP an example where the table is further used
> >>>> >>> in subsequent
> >>>> >>>>>>>>> queries
> >>>> >>>>>>>>>>>>>>>>> (specially in batch mode).
> >>>> >>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>> Thanks for your suggestion, I added how to use
> >>>> >>> Dynamic Table in
> >>>> >>>>>>>>> FLIP
> >>>> >>>>>>>>>>>>> user
> >>>> >>>>>>>>>>>>>>>>> story section, Dynamic Table can be referenced
> >>>> >>> by downstream
> >>>> >>>>>>>>> Dynamic
> >>>> >>>>>>>>>>>>>>>>> Table
> >>>> >>>>>>>>>>>>>>>>> and can also support OLAP queries.
> >>>> >>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>> Best,
> >>>> >>>>>>>>>>>>>>>>> Ron
> >>>> >>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>> Ron liu <ron9....@gmail.com> 于2024年3月23日周六
> >>>> >>> 10:35写道：
> >>>> >>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>> Hi, Feng
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>> Although currently we restrict users from
> >>>> >>> modifying the query,
> >>>> >>>>> I
> >>>> >>>>>>>>>>>>> wonder
> >>>> >>>>>>>>>>>>>>>>>>> if
> >>>> >>>>>>>>>>>>>>>>>>> we can provide a better way to help users
> >>>> >>> rebuild it without
> >>>> >>>>>>>>>>> affecting
> >>>> >>>>>>>>>>>>>>>>>>> downstream OLAP queries.
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>> Considering the problem of data consistency,
> >>>> >>> so in the first
> >>>> >>>>> step
> >>>> >>>>>>>>> we
> >>>> >>>>>>>>>>>>> are
> >>>> >>>>>>>>>>>>>>>>>>> strictly limited in semantics and do not
> >>>> >>> support modify the
> >>>> >>>>> query.
> >>>> >>>>>>>>>>>>>>>>>>> This is
> >>>> >>>>>>>>>>>>>>>>>>> really a good problem, one of my ideas is to
> >>>> >>> introduce a syntax
> >>>> >>>>>>>>>>>>>>>>>>> similar to
> >>>> >>>>>>>>>>>>>>>>>>> SWAP [1], which supports exchanging two
> >>>> >>> Dynamic Tables.
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>> From the documentation, the definitions
> >>>> >>> SQL and job
> >>>> >>>>> information
> >>>> >>>>>>>>> are
> >>>> >>>>>>>>>>>>>>>>>>> stored in the Catalog. Does this mean that
> >>>> >>> if a system needs to
> >>>> >>>>>>>>> adapt
> >>>> >>>>>>>>>>>>> to
> >>>> >>>>>>>>>>>>>>>>>>> Dynamic Tables, it also needs to store
> >>>> >>> Flink's job information
> >>>> >>>>> in
> >>>> >>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>> corresponding system?
> >>>> >>>>>>>>>>>>>>>>>>> For example, does MySQL's Catalog need to
> >>>> >>> store flink job
> >>>> >>>>>>>>> information
> >>>> >>>>>>>>>>>>> as
> >>>> >>>>>>>>>>>>>>>>>>> well?
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>> Yes, currently we need to rely on Catalog to
> >>>> >>> store refresh job
> >>>> >>>>>>>>>>>>>>>>>>> information.
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>> Users still need to consider how much
> >>>> >>> memory is being used, how
> >>>> >>>>>>>>>>> large
> >>>> >>>>>>>>>>>>>>>>>>> the concurrency is, which type of state
> >>>> >>> backend is being used,
> >>>> >>>>> and
> >>>> >>>>>>>>>>>>>>>>>>> may need
> >>>> >>>>>>>>>>>>>>>>>>> to set TTL expiration.
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>> Similar to the current practice, job
> >>>> >>> parameters can be set via
> >>>> >>>>> the
> >>>> >>>>>>>>>>>>> Flink
> >>>> >>>>>>>>>>>>>>>>>>> conf or SET commands
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>> When we submit a refresh command, can we
> >>>> >>> help users detect if
> >>>> >>>>>>>>> there
> >>>> >>>>>>>>>>>>> are
> >>>> >>>>>>>>>>>>>>>>>>> any
> >>>> >>>>>>>>>>>>>>>>>>> running jobs and automatically stop them
> >>>> >>> before executing the
> >>>> >>>>>>>>> refresh
> >>>> >>>>>>>>>>>>>>>>>>> command? Then wait for it to complete before
> >>>> >>> restarting the
> >>>> >>>>>>>>>>> background
> >>>> >>>>>>>>>>>>>>>>>>> streaming job?
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>> Purely from a technical implementation point
> >>>> >>> of view, your
> >>>> >>>>>>>>> proposal
> >>>> >>>>>>>>>>> is
> >>>> >>>>>>>>>>>>>>>>>>> doable, but it would be more costly. Also I
> >>>> >>> think data
> >>>> >>>>> consistency
> >>>> >>>>>>>>>>>>>>>>>>> itself
> >>>> >>>>>>>>>>>>>>>>>>> is the responsibility of the user, similar
> >>>> >>> to how Regular Table
> >>>> >>>>> is
> >>>> >>>>>>>>>>>>>>>>>>> now also
> >>>> >>>>>>>>>>>>>>>>>>> the responsibility of the user, so it's
> >>>> >>> consistent with its
> >>>> >>>>>>>>> behavior
> >>>> >>>>>>>>>>>>>>>>>>> and no
> >>>> >>>>>>>>>>>>>>>>>>> additional guarantees are made at the engine
> >>>> >>> level.
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>> Best,
> >>>> >>>>>>>>>>>>>>>>>>> Ron
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>> Ahmed Hamdy <hamdy10...@gmail.com>
> >>>> >>> 于2024年3月22日周五 23:50写道：
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>> Hi Ron,
> >>>> >>>>>>>>>>>>>>>>>>>>> Sorry for joining the discussion late,
> >>>> >>> thanks for the effort.
> >>>> >>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>> I think the base idea is great, however I
> >>>> >>> have a couple of
> >>>> >>>>>>>>> comments:
> >>>> >>>>>>>>>>>>>>>>>>>>> - I want to iterate on Timo's comments
> >>>> >>> regarding the confusion
> >>>> >>>>>>>>>>> between
> >>>> >>>>>>>>>>>>>>>>>>>>> "Dynamic Table" and current Flink
> >>>> >>> "Table". Should the
> >>>> >>>>>>>>> refactoring of
> >>>> >>>>>>>>>>>>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>> system happen in 2.0, should we rename it
> >>>> >>> in this Flip ( as the
> >>>> >>>>>>>>>>>>>>>>>>>>> suggestions
> >>>> >>>>>>>>>>>>>>>>>>>>> in the thread ) and address the holistic
> >>>> >>> changes in a separate
> >>>> >>>>>>>>> Flip
> >>>> >>>>>>>>>>>>> for
> >>>> >>>>>>>>>>>>>>>>>>>>> 2.0?
> >>>> >>>>>>>>>>>>>>>>>>>>> - I feel confused with how it is further
> >>>> >>> with other components,
> >>>> >>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>> examples provided feel like a standalone
> >>>> >>> ETL job, could you
> >>>> >>>>>>>>> provide
> >>>> >>>>>>>>>>>>>>>>>>>>> in the
> >>>> >>>>>>>>>>>>>>>>>>>>> FLIP an example where the table is
> >>>> >>> further used in subsequent
> >>>> >>>>>>>>>>> queries
> >>>> >>>>>>>>>>>>>>>>>>>>> (specially in batch mode).
> >>>> >>>>>>>>>>>>>>>>>>>>> - I really like the standard of keeping
> >>>> >>> the unified batch and
> >>>> >>>>>>>>>>>>> streaming
> >>>> >>>>>>>>>>>>>>>>>>>>> approach
> >>>> >>>>>>>>>>>>>>>>>>>>> Best Regards
> >>>> >>>>>>>>>>>>>>>>>>>>> Ahmed Hamdy
> >>>> >>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>> On Fri, 22 Mar 2024 at 12:07, Lincoln Lee
> >>>> >>> <
> >>>> >>>>>>>>> lincoln.8...@gmail.com>
> >>>> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>> >>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>> Hi Timo,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks for your thoughtful inputs!
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>> Yes, expanding the MATERIALIZED
> >>>> >>> VIEW(MV) could achieve the
> >>>> >>>>> same
> >>>> >>>>>>>>>>>>>>>>>>>>> function,
> >>>> >>>>>>>>>>>>>>>>>>>>>>> but our primary concern is that by
> >>>> >>> using a view, we might
> >>>> >>>>> limit
> >>>> >>>>>>>>>>>>> future
> >>>> >>>>>>>>>>>>>>>>>>>>>>> opportunities
> >>>> >>>>>>>>>>>>>>>>>>>>>>> to optimize queries through automatic
> >>>> >>> materialization
> >>>> >>>>> rewriting
> >>>> >>>>>>>>>>> [1],
> >>>> >>>>>>>>>>>>>>>>>>>>>>> leveraging
> >>>> >>>>>>>>>>>>>>>>>>>>>>> the support for MV by physical
> >>>> >>> storage. This is because we
> >>>> >>>>>>>>> would be
> >>>> >>>>>>>>>>>>>>>>>>>>>>> breaking
> >>>> >>>>>>>>>>>>>>>>>>>>>>> the intuitive semantics of a
> >>>> >>> materialized view (a materialized
> >>>> >>>>>>>>> view
> >>>> >>>>>>>>>>>>>>>>>>>>>>> represents
> >>>> >>>>>>>>>>>>>>>>>>>>>>> the result of a query) by allowing
> >>>> >>> data modifications, thus
> >>>> >>>>>>>>> losing
> >>>> >>>>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>> potential
> >>>> >>>>>>>>>>>>>>>>>>>>>>> for such optimizations.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>> With these considerations in mind, we
> >>>> >>> were inspired by Google
> >>>> >>>>>>>>>>>>> Looker's
> >>>> >>>>>>>>>>>>>>>>>>>>>>> Persistent
> >>>> >>>>>>>>>>>>>>>>>>>>>>> Derived Table [2]. PDT is designed for
> >>>> >>> building Looker's
> >>>> >>>>>>>>> automated
> >>>> >>>>>>>>>>>>>>>>>>>>>>> modeling,
> >>>> >>>>>>>>>>>>>>>>>>>>>>> aligning with our purpose for the
> >>>> >>> stream-batch automatic
> >>>> >>>>>>>>> pipeline.
> >>>> >>>>>>>>>>>>>>>>>>>>>>> Therefore,
> >>>> >>>>>>>>>>>>>>>>>>>>>>> we are considering another candidate,
> >>>> >>> Derived Table, the term
> >>>> >>>>>>>>>>>>> 'derive'
> >>>> >>>>>>>>>>>>>>>>>>>>>>> suggests a
> >>>> >>>>>>>>>>>>>>>>>>>>>>> query, and 'table' retains
> >>>> >>> modifiability. This approach would
> >>>> >>>>>>>>> not
> >>>> >>>>>>>>>>>>>>>>>>>>> disrupt
> >>>> >>>>>>>>>>>>>>>>>>>>>>> our current
> >>>> >>>>>>>>>>>>>>>>>>>>>>> concept of a dynamic table, preserving
> >>>> >>> the future utility of
> >>>> >>>>>>>>> MVs.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>> Conceptually, a Derived Table is a
> >>>> >>> Dynamic Table + Continuous
> >>>> >>>>>>>>>>>>>>>>>>>>>>> Query. By
> >>>> >>>>>>>>>>>>>>>>>>>>>>> introducing
> >>>> >>>>>>>>>>>>>>>>>>>>>>> a new concept Derived Table for this
> >>>> >>> FLIP, this makes all
> >>>> >>>>>>>>>>>>>>>>>>>>>>> concepts to
> >>>> >>>>>>>>>>>>>>>>>>>>> play
> >>>> >>>>>>>>>>>>>>>>>>>>>>> together nicely.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>> What do you think about this?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://calcite.apache.org/docs/materialized_views.html__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j73_NFf4D5$
> >>>> >>>>>>>>>>>>>>>>>>>>>>> [2]
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://cloud.google.com/looker/docs/derived-tables*persistent_derived_tables__;Iw!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j7382-2zI3$
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>> >>>>>>>>>>>>>>>>>>>>>>> Lincoln Lee
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>> Timo Walther <twal...@apache.org>
> >>>> >>> 于2024年3月22日周五 17:54写道：
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Ron,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> thanks for the detailed answer.
> >>>> >>> Sorry, for my late reply, we
> >>>> >>>>>>>>> had a
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> conference that kept me busy.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> In the current concept[1], it
> >>>> >>> actually includes: Dynamic
> >>>> >>>>>>>>>>> Tables
> >>>> >>>>>>>>>>>>> &
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> & Continuous Query. Dynamic
> >>>> >>> Table is just an abstract
> >>>> >>>>>>>>> logical
> >>>> >>>>>>>>>>>>>>>>>>>>> concept
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> This explanation makes sense to me.
> >>>> >>> But the docs also say "A
> >>>> >>>>>>>>>>>>>>>>>>>>> continuous
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> query is evaluated on the dynamic
> >>>> >>> table yielding a new
> >>>> >>>>> dynamic
> >>>> >>>>>>>>>>>>>>>>>>>>> table.".
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> So even our regular CREATE TABLEs
> >>>> >>> are considered dynamic
> >>>> >>>>>>>>> tables.
> >>>> >>>>>>>>>>>>> This
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> can also be seen in the diagram
> >>>> >>> "Dynamic Table -> Continuous
> >>>> >>>>>>>>> Query
> >>>> >>>>>>>>>>>>> ->
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> Dynamic Table". Currently, Flink
> >>>> >>> queries can only be executed
> >>>> >>>>>>>>> on
> >>>> >>>>>>>>>>>>>>>>>>>>> Dynamic
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> Tables.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> In essence, a materialized view
> >>>> >>> represents the result of
> >>>> >>>>> a
> >>>> >>>>>>>>>>>>> query.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> Isn't that what your proposal does
> >>>> >>> as well?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> the object of the suspend
> >>>> >>> operation is the refresh task
> >>>> >>>>> of
> >>>> >>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> dynamic table
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> I understand that Snowflake uses
> >>>> >>> the term [1] to merge their
> >>>> >>>>>>>>>>>>> concepts
> >>>> >>>>>>>>>>>>>>>>>>>>> of
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> STREAM, TASK, and TABLE into one
> >>>> >>> piece of concept. But Flink
> >>>> >>>>>>>>> has
> >>>> >>>>>>>>>>> no
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> concept of a "refresh task". Also,
> >>>> >>> they already introduced
> >>>> >>>>>>>>>>>>>>>>>>>>> MATERIALIZED
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> VIEW. Flink is in the convenient
> >>>> >>> position that the concept of
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> materialized views is not taken
> >>>> >>> (reserved maybe for exactly
> >>>> >>>>>>>>> this
> >>>> >>>>>>>>>>> use
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> case?). And SQL standard concept
> >>>> >>> could be "slightly adapted"
> >>>> >>>>> to
> >>>> >>>>>>>>>>> our
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> needs. Looking at other vendors
> >>>> >>> like Postgres[2], they also
> >>>> >>>>> use
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> `REFRESH` commands so why not
> >>>> >>> adding additional commands such
> >>>> >>>>>>>>> as
> >>>> >>>>>>>>>>>>>>>>>>>>> DELETE
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> or UPDATE. Oracle supports "ON
> >>>> >>> PREBUILT TABLE clause tells
> >>>> >>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>> database
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> to use an existing table
> >>>> >>> segment"[3] which comes closer to
> >>>> >>>>>>>>> what we
> >>>> >>>>>>>>>>>>>>>>>>>>> want
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> as well.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> it is not intended to support
> >>>> >>> data modification
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> This is an argument that I
> >>>> >>> understand. But we as Flink could
> >>>> >>>>>>>>> allow
> >>>> >>>>>>>>>>>>>>>>>>>>> data
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> modifications. This way we are only
> >>>> >>> extending the standard
> >>>> >>>>> and
> >>>> >>>>>>>>>>> don't
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> introduce new concepts.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> If we can't agree on using
> >>>> >>> MATERIALIZED VIEW concept. We
> >>>> >>>>> should
> >>>> >>>>>>>>>>> fix
> >>>> >>>>>>>>>>>>>>>>>>>>> our
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> syntax in a Flink 2.0 effort.
> >>>> >>> Making regular tables bounded
> >>>> >>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>> dynamic
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> tables unbounded. We would be
> >>>> >>> closer to the SQL standard with
> >>>> >>>>>>>>> this
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> pave the way for the future. I
> >>>> >>> would actually support this if
> >>>> >>>>>>>>> all
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> concepts play together nicely.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> In the future, we can consider
> >>>> >>> extending the statement
> >>>> >>>>> set
> >>>> >>>>>>>>>>>>> syntax
> >>>> >>>>>>>>>>>>>>>>>>>>> to
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> support the creation of multiple
> >>>> >>> dynamic tables.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> It's good that we called the
> >>>> >>> concept STATEMENT SET. This
> >>>> >>>>>>>>> allows us
> >>>> >>>>>>>>>>>>> to
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> defined CREATE TABLE within. Even
> >>>> >>> if it might look a bit
> >>>> >>>>>>>>>>> confusing.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> Regards,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> Timo
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://docs.snowflake.com/en/user-guide/dynamic-tables-about__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j73zexZBXu$
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> [2]
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://www.postgresql.org/docs/current/sql-creatematerializedview.html__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j73zbNhvS7$
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> [3]
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://oracle-base.com/articles/misc/materialized-views__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j739xS1kvD$
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> On 21.03.24 04:14, Feng Jin wrote:
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ron and Lincoln
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving this
> >>>> >>> discussion. I believe it will
> >>>> >>>>> greatly
> >>>> >>>>>>>>>>>>>>>>>>>>> improve
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> convenience of managing user
> >>>> >>> real-time pipelines.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I have some questions.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> *Regarding Limitations of
> >>>> >>> Dynamic Table:*
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does not support modifying
> >>>> >>> the select statement after the
> >>>> >>>>>>>>>>> dynamic
> >>>> >>>>>>>>>>>>>>>>>>>>>>> table
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> is created.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Although currently we restrict
> >>>> >>> users from modifying the
> >>>> >>>>>>>>> query, I
> >>>> >>>>>>>>>>>>>>>>>>>>> wonder
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> if
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> we can provide a better way to
> >>>> >>> help users rebuild it without
> >>>> >>>>>>>>>>>>>>>>>>>>> affecting
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> downstream OLAP queries.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> *Regarding the management of
> >>>> >>> background jobs:*
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> 1. From the documentation, the
> >>>> >>> definitions SQL and job
> >>>> >>>>>>>>>>> information
> >>>> >>>>>>>>>>>>>>>>>>>>> are
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> stored in the Catalog. Does this
> >>>> >>> mean that if a system needs
> >>>> >>>>>>>>> to
> >>>> >>>>>>>>>>>>>>>>>>>>> adapt
> >>>> >>>>>>>>>>>>>>>>>>>>>>> to
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Dynamic Tables, it also needs to
> >>>> >>> store Flink's job
> >>>> >>>>>>>>> information in
> >>>> >>>>>>>>>>>>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> corresponding system?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> For example, does MySQL's
> >>>> >>> Catalog need to store flink job
> >>>> >>>>>>>>>>>>>>>>>>>>> information
> >>>> >>>>>>>>>>>>>>>>>>>>>>> as
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> well?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Users still need to consider
> >>>> >>> how much memory is being
> >>>> >>>>> used,
> >>>> >>>>>>>>>>> how
> >>>> >>>>>>>>>>>>>>>>>>>>>>> large
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> the concurrency is, which type
> >>>> >>> of state backend is being
> >>>> >>>>> used,
> >>>> >>>>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>> may
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> need
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> to set TTL expiration.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> *Regarding the Refresh Part:*
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> If the refresh mode is
> >>>> >>> continuous and a background job is
> >>>> >>>>>>>>>>> running,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> caution should be taken with the
> >>>> >>> refresh command as it can
> >>>> >>>>>>>>> lead
> >>>> >>>>>>>>>>> to
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent data.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> When we submit a refresh
> >>>> >>> command, can we help users detect
> >>>> >>>>> if
> >>>> >>>>>>>>>>> there
> >>>> >>>>>>>>>>>>>>>>>>>>> are
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> any
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> running jobs and automatically
> >>>> >>> stop them before executing
> >>>> >>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>> refresh
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> command? Then wait for it to
> >>>> >>> complete before restarting the
> >>>> >>>>>>>>>>>>>>>>>>>>> background
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> streaming job?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Feng
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Mar 19, 2024 at 9:40 PM
> >>>> >>> Lincoln Lee <
> >>>> >>>>>>>>>>>>> lincoln.8...@gmail.com
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Yun,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you very much for your
> >>>> >>> valuable input!
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Incremental mode is indeed an
> >>>> >>> attractive idea, we have also
> >>>> >>>>>>>>>>>>>>>>>>>>> discussed
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this, but in the current
> >>>> >>> design,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> we first provided two refresh
> >>>> >>> modes: CONTINUOUS and
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> FULL. Incremental mode can be
> >>>> >>> introduced
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> once the execution layer has
> >>>> >>> the capability.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> My answer for the two
> >>>> >>> questions:
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, cascading is a good
> >>>> >>> question. Current proposal
> >>>> >>>>>>>>> provides a
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> freshness that defines a
> >>>> >>> dynamic
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> table relative to the base
> >>>> >>> table’s lag. If users need to
> >>>> >>>>>>>>>>> consider
> >>>> >>>>>>>>>>>>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> end-to-end freshness of
> >>>> >>> multiple
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> cascaded dynamic tables, he
> >>>> >>> can manually split them for
> >>>> >>>>> now.
> >>>> >>>>>>>>> Of
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> course, how to let multiple
> >>>> >>> cascaded
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> or dependent dynamic tables
> >>>> >>> complete the freshness
> >>>> >>>>>>>>> definition
> >>>> >>>>>>>>>>>>> in
> >>>> >>>>>>>>>>>>>>>>>>>>> a
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> simpler way, I think it can be
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> extended in the future.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cascading refresh is also a
> >>>> >>> part we focus on discussing. In
> >>>> >>>>>>>>> this
> >>>> >>>>>>>>>>>>>>>>>>>>> flip,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> we hope to focus as much as
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> possible on the core features
> >>>> >>> (as it already involves a lot
> >>>> >>>>>>>>>>>>>>>>>>>>> things),
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> so we did not directly
> >>>> >>> introduce related
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax. However, based on the
> >>>> >>> current design, combined
> >>>> >>>>>>>>> with
> >>>> >>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog and lineage,
> >>>> >>> theoretically,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> users can also finish the
> >>>> >>> cascading refresh.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lincoln Lee
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yun Tang <myas...@live.com>
> >>>> >>> 于2024年3月19日周二 13:45写道：
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lincoln,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving this
> >>>> >>> discussion, and I am so excited to
> >>>> >>>>>>>>> see
> >>>> >>>>>>>>>>>>>>>>>>>>> this
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> topic
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> being discussed in the
> >>>> >>> Flink community!
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From my point of view,
> >>>> >>> instead of the work of unifying
> >>>> >>>>>>>>>>>>> streaming
> >>>> >>>>>>>>>>>>>>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> batch
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in DataStream API [1],
> >>>> >>> this FLIP actually could make users
> >>>> >>>>>>>>>>>>> benefit
> >>>> >>>>>>>>>>>>>>>>>>>>>>> from
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> one
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine to rule batch &
> >>>> >>> streaming.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we treat this FLIP as
> >>>> >>> an open-source implementation of
> >>>> >>>>>>>>>>>>>>>>>>>>> Snowflake's
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamic tables [2], we
> >>>> >>> still lack an incremental refresh
> >>>> >>>>>>>>> mode
> >>>> >>>>>>>>>>> to
> >>>> >>>>>>>>>>>>>>>>>>>>> make
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ETL near real-time with a
> >>>> >>> much cheaper computation cost.
> >>>> >>>>>>>>>>> However,
> >>>> >>>>>>>>>>>>>>>>>>>>> I
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> think
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this could be done under
> >>>> >>> the current design by introducing
> >>>> >>>>>>>>>>>>> another
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> refresh
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode in the future.
> >>>> >>> Although the extra work of incremental
> >>>> >>>>>>>>> view
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> maintenance
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would be much larger.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the FLIP itself, I
> >>>> >>> have several questions below:
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It seems this FLIP does
> >>>> >>> not consider the lag of
> >>>> >>>>> refreshes
> >>>> >>>>>>>>>>>>>>>>>>>>> across
> >>>> >>>>>>>>>>>>>>>>>>>>>>> ETL
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> layers from ODS ---> DWD
> >>>> >>> ---> APP [3]. We currently only
> >>>> >>>>>>>>>>> consider
> >>>> >>>>>>>>>>>>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scheduler interval, which
> >>>> >>> means we cannot use lag to
> >>>> >>>>>>>>>>>>> automatically
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the upfront micro-batch
> >>>> >>> jobs to do the work.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. To support the
> >>>> >>> automagical refreshes, we should
> >>>> >>>>> consider
> >>>> >>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>> lineage
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the catalog or somewhere
> >>>> >>> else.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-134*3A*Batch*execution*for*the*DataStream*API__;JSsrKysrKw!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j7352JICzI$
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2]
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://docs.snowflake.com/en/user-guide/dynamic-tables-about__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j73zexZBXu$
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3]
> >>>> >>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j735ghqpxk$
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yun Tang
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>> ________________________________
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From: Lincoln Lee <
> >>>> >>> lincoln.8...@gmail.com>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: Thursday, March 14,
> >>>> >>> 2024 14:35
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@flink.apache.org <
> >>>> >>> dev@flink.apache.org>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS]
> >>>> >>> FLIP-435: Introduce a New Dynamic
> >>>> >>>>>>>>> Table
> >>>> >>>>>>>>>>>>> for
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Simplifying Data Pipelines
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jing,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your attention
> >>>> >>> to this flip! I'll try to answer
> >>>> >>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> following
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. How to define query
> >>>> >>> of dynamic table?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Use flink sql or
> >>>> >>> introducing new syntax?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If use flink sql, how
> >>>> >>> to handle the difference in SQL
> >>>> >>>>>>>>> between
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> streaming
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> batch processing?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, a query
> >>>> >>> including window aggregate based on
> >>>> >>>>>>>>>>>>>>>>>>>>> processing
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> time?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or a query including
> >>>> >>> global order by?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Similar to `CREATE TABLE
> >>>> >>> AS query`, here the `query` also
> >>>> >>>>>>>>> uses
> >>>> >>>>>>>>>>>>>>>>>>>>> Flink
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> sql
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> doesn't introduce a
> >>>> >>> totally new syntax.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We will not change the
> >>>> >>> status respect to
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the difference in
> >>>> >>> functionality of flink sql itself on
> >>>> >>>>>>>>>>> streaming
> >>>> >>>>>>>>>>>>>>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> batch, for example,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the proctime window agg on
> >>>> >>> streaming and global sort on
> >>>> >>>>>>>>> batch
> >>>> >>>>>>>>>>>>> that
> >>>> >>>>>>>>>>>>>>>>>>>>>>> you
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mentioned,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in fact, do not work
> >>>> >>> properly in the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other mode, so when the
> >>>> >>> user modifies the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> refresh mode of a dynamic
> >>>> >>> table that is not supported, we
> >>>> >>>>>>>>> will
> >>>> >>>>>>>>>>>>>>>>>>>>> throw
> >>>> >>>>>>>>>>>>>>>>>>>>>>> an
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Whether modify the
> >>>> >>> query of dynamic table is allowed?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Or we could only
> >>>> >>> refresh a dynamic table based on the
> >>>> >>>>>>>>> initial
> >>>> >>>>>>>>>>>>>>>>>>>>> query?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, in the current
> >>>> >>> design, the query definition of the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamic table is not
> >>>> >>> allowed
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to be modified, and you
> >>>> >>> can only refresh the data based
> >>>> >>>>>>>>> on
> >>>> >>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initial definition.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. How to use dynamic
> >>>> >>> table?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The dynamic table seems
> >>>> >>> to be similar to the materialized
> >>>> >>>>>>>>>>> view.
> >>>> >>>>>>>>>>>>>>>>>>>>>>> Will
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something like
> >>>> >>> materialized view rewriting during the
> >>>> >>>>>>>>>>>>>>>>>>>>> optimization?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's true that dynamic
> >>>> >>> table and materialized view
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are similar in some ways,
> >>>> >>> but as Ron
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explains
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there are differences. In
> >>>> >>> terms of optimization, automated
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization discovery
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> similar to that supported
> >>>> >>> by calcite is also a potential
> >>>> >>>>>>>>>>>>>>>>>>>>> possibility,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> perhaps with the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> addition of automated
> >>>> >>> rewriting in the future.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lincoln Lee
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ron liu <
> >>>> >>> ron9....@gmail.com> 于2024年3月14日周四 14:01写道：
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Timo
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for later
> >>>> >>> response, thanks for your feedback.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding your
> >>>> >>> questions:
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Flink has introduced
> >>>> >>> the concept of Dynamic Tables many
> >>>> >>>>>>>>> years
> >>>> >>>>>>>>>>>>>>>>>>>>> ago.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> How
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does the term "Dynamic
> >>>> >>> Table" fit into Flink's regular
> >>>> >>>>>>>>> tables
> >>>> >>>>>>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>>>> also
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how does it relate to
> >>>> >>> Table API?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I fear that adding
> >>>> >>> the DYNAMIC TABLE keyword could cause
> >>>> >>>>>>>>>>>>>>>>>>>>> confusion
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users, because a
> >>>> >>> term for regular CREATE TABLE (that can
> >>>> >>>>>>>>> be
> >>>> >>>>>>>>>>>>>>>>>>>>> "kind
> >>>> >>>>>>>>>>>>>>>>>>>>>>> of
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamic" as well and
> >>>> >>> is backed by a changelog) is then
> >>>> >>>>>>>>>>> missing.
> >>>> >>>>>>>>>>>>>>>>>>>>>>> Also
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> given that we call
> >>>> >>> our connectors for those tables,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DynamicTableSource
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and DynamicTableSink.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In general, I find
> >>>> >>> it contradicting that a TABLE can be
> >>>> >>>>>>>>>>>>>>>>>>>>> "paused" or
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "resumed". From an
> >>>> >>> English language perspective, this
> >>>> >>>>> does
> >>>> >>>>>>>>>>>>> sound
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> incorrect. In my
> >>>> >>> opinion (without much research yet), a
> >>>> >>>>>>>>>>>>>>>>>>>>> continuous
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating trigger
> >>>> >>> should rather be modelled as a CREATE
> >>>> >>>>>>>>>>>>>>>>>>>>> MATERIALIZED
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VIEW
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which users are
> >>>> >>> familiar with?) or a new concept such
> >>>> >>>>> as
> >>>> >>>>>>>>> a
> >>>> >>>>>>>>>>>>>>>>>>>>> CREATE
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> TASK
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (that can be paused
> >>>> >>> and resumed?).
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the current
> >>>> >>> concept[1], it actually includes: Dynamic
> >>>> >>>>>>>>>>> Tables
> >>>> >>>>>>>>>>>>> &
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Continuous Query.
> >>>> >>> Dynamic Table is just an abstract
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> logical concept
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> , which in its physical
> >>>> >>> form represents either a table
> >>>> >>>>> or a
> >>>> >>>>>>>>>>>>>>>>>>>>>>> changelog
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stream. It requires the
> >>>> >>> combination with Continuous Query
> >>>> >>>>>>>>> to
> >>>> >>>>>>>>>>>>>>>>>>>>> achieve
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamic updates of the
> >>>> >>> target table similar to a
> >>>> >>>>> database’s
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Materialized View.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We hope to upgrade the
> >>>> >>> Dynamic Table to a real entity
> >>>> >>>>> that
> >>>> >>>>>>>>>>> users
> >>>> >>>>>>>>>>>>>>>>>>>>> can
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operate, which combines
> >>>> >>> the logical concepts of Dynamic
> >>>> >>>>>>>>>>> Tables +
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Continuous Query. By
> >>>> >>> integrating the definition of tables
> >>>> >>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>>>> queries,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it can achieve
> >>>> >>> functions similar to Materialized Views,
> >>>> >>>>>>>>>>>>>>>>>>>>> simplifying
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users' data processing
> >>>> >>> pipelines.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, the object of the
> >>>> >>> suspend operation is the refresh
> >>>> >>>>>>>>> task of
> >>>> >>>>>>>>>>>>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamic table. The
> >>>> >>> command `ALTER DYNAMIC TABLE
> >>>> >>>>> table_name
> >>>> >>>>>>>>>>>>>>>>>>>>> SUSPEND
> >>>> >>>>>>>>>>>>>>>>>>>>>>> `
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is actually a shorthand
> >>>> >>> for `ALTER DYNAMIC TABLE
> >>>> >>>>> table_name
> >>>> >>>>>>>>>>>>>>>>>>>>> SUSPEND
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> REFRESH` (if written in
> >>>> >>> full for clarity, we can also
> >>>> >>>>>>>>> modify
> >>>> >>>>>>>>>>>>> it).
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Initially, we also
> >>>> >>> considered Materialized Views
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> , but ultimately
> >>>> >>> decided against them. Materialized views
> >>>> >>>>>>>>> are
> >>>> >>>>>>>>>>>>>>>>>>>>>>> designed
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to enhance query
> >>>> >>> performance for workloads that consist
> >>>> >>>>> of
> >>>> >>>>>>>>>>>>>>>>>>>>> common,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> repetitive query
> >>>> >>> patterns. In essence, a materialized
> >>>> >>>>> view
> >>>> >>>>>>>>>>>>>>>>>>>>>>> represents
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the result of a query.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, it is not
> >>>> >>> intended to support data modification.
> >>>> >>>>>>>>> For
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lakehouse scenarios,
> >>>> >>> where the ability to delete or
> >>>> >>>>> update
> >>>> >>>>>>>>>>> data
> >>>> >>>>>>>>>>>>>>>>>>>>> is
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> crucial (such as
> >>>> >>> compliance with GDPR, FLIP-2),
> >>>> >>>>>>>>> materialized
> >>>> >>>>>>>>>>>>>>>>>>>>> views
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fall short.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Compared to CREATE
> >>>> >>> (regular) TABLE, CREATE DYNAMIC TABLE
> >>>> >>>>>>>>> not
> >>>> >>>>>>>>>>>>> only
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defines metadata in the
> >>>> >>> catalog but also automatically
> >>>> >>>>>>>>>>> initiates
> >>>> >>>>>>>>>>>>>>>>>>>>> a
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data refresh task based
> >>>> >>> on the query specified during
> >>>> >>>>> table
> >>>> >>>>>>>>>>>>>>>>>>>>>>> creation.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It dynamically executes
> >>>> >>> data updates. Users can focus on
> >>>> >>>>>>>>> data
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dependencies and data
> >>>> >>> generation logic.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The new dynamic table
> >>>> >>> does not conflict with the existing
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DynamicTableSource and
> >>>> >>> DynamicTableSink interfaces. For
> >>>> >>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>> developer,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all that needs to be
> >>>> >>> implemented is the new
> >>>> >>>>>>>>>>> CatalogDynamicTable,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> without changing the
> >>>> >>> implementation of source and sink.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5. For now, the FLIP
> >>>> >>> does not consider supporting Table
> >>>> >>>>> API
> >>>> >>>>>>>>>>>>>>>>>>>>>>> operations
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dynamic Table
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> . However, once the SQL
> >>>> >>> syntax is finalized, we can
> >>>> >>>>> discuss
> >>>> >>>>>>>>>>> this
> >>>> >>>>>>>>>>>>>>>>>>>>> in
> >>>> >>>>>>>>>>>>>>>>>>>>>>> a
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> separate FLIP.
> >>>> >>> Currently, I have a rough idea: the Table
> >>>> >>>>>>>>> API
> >>>> >>>>>>>>>>>>>>>>>>>>> should
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also introduce
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DynamicTable operation
> >>>> >>> interfaces
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> corresponding to the
> >>>> >>> existing Table interfaces.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The TableEnvironment
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will provide relevant
> >>>> >>> methods to support various
> >>>> >>>>> dynamic
> >>>> >>>>>>>>>>>>> table
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operations. The goal
> >>>> >>> for the new Dynamic Table is to
> >>>> >>>>> offer
> >>>> >>>>>>>>>>> users
> >>>> >>>>>>>>>>>>>>>>>>>>> an
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> experience similar to
> >>>> >>> using a database, which is why we
> >>>> >>>>>>>>>>>>>>>>>>>>> prioritize
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL-based approaches
> >>>> >>> initially.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How do you envision
> >>>> >>> re-adding the functionality of a
> >>>> >>>>>>>>>>> statement
> >>>> >>>>>>>>>>>>>>>>>>>>> set,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fans out to multiple
> >>>> >>> tables? This is a very important
> >>>> >>>>> use
> >>>> >>>>>>>>>>> case
> >>>> >>>>>>>>>>>>>>>>>>>>> for
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> data
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pipelines.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Multi-tables is indeed
> >>>> >>> a very important user scenario. In
> >>>> >>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>> future,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we can consider
> >>>> >>> extending the statement set syntax to
> >>>> >>>>>>>>> support
> >>>> >>>>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creation of multiple
> >>>> >>> dynamic tables.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since the early
> >>>> >>> days of Flink SQL, we were discussing
> >>>> >>>>>>>>>>> `SELECT
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> STREAM
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FROM T EMIT 5
> >>>> >>> MINUTES`. Your proposal seems to rephrase
> >>>> >>>>>>>>>>> STREAM
> >>>> >>>>>>>>>>>>>>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> EMIT,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into other keywords
> >>>> >>> DYNAMIC TABLE and FRESHNESS. But the
> >>>> >>>>>>>>> core
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functionality is
> >>>> >>> still there. I'm wondering if we should
> >>>> >>>>>>>>>>> widen
> >>>> >>>>>>>>>>>>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scope
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (maybe not part of
> >>>> >>> this FLIP but a new FLIP) to follow
> >>>> >>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>> standard
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> closely. Making
> >>>> >>> `SELECT * FROM t` bounded by default and
> >>>> >>>>>>>>> use
> >>>> >>>>>>>>>>>>> new
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for the dynamic
> >>>> >>> behavior. Flink 2.0 would be the perfect
> >>>> >>>>>>>>> time
> >>>> >>>>>>>>>>>>>>>>>>>>> for
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> however, it would
> >>>> >>> require careful discussions. What do
> >>>> >>>>> you
> >>>> >>>>>>>>>>>>>>>>>>>>> think?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The query part indeed
> >>>> >>> requires a separate FLIP
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for discussion, as it
> >>>> >>> involves changes to the default
> >>>> >>>>>>>>>>> behavior.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>>
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/dynamic_tables__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j73477_wHn$
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ron
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jing Zhang <
> >>>> >>> beyond1...@gmail.com> 于2024年3月13日周三 15:19写道：
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Lincoln & Ron,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the
> >>>> >>> proposal.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the
> >>>> >>> question raised by Timo.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Besides, I have some
> >>>> >>> other questions.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. How to define
> >>>> >>> query of dynamic table?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Use flink sql or
> >>>> >>> introducing new syntax?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If use flink sql,
> >>>> >>> how to handle the difference in SQL
> >>>> >>>>>>>>> between
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> streaming
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> batch processing?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, a query
> >>>> >>> including window aggregate based on
> >>>> >>>>>>>>>>>>>>>>>>>>> processing
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or a query including
> >>>> >>> global order by?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Whether modify
> >>>> >>> the query of dynamic table is allowed?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Or we could only
> >>>> >>> refresh a dynamic table based on
> >>>> >>>>> initial
> >>>> >>>>>>>>>>>>> query?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. How to use
> >>>> >>> dynamic table?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The dynamic table
> >>>> >>> seems to be similar with materialized
> >>>> >>>>>>>>> view.
> >>>> >>>>>>>>>>>>>>>>>>>>> Will
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something like
> >>>> >>> materialized view rewriting during the
> >>>> >>>>>>>>>>>>>>>>>>>>> optimization?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jing Zhang
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Timo Walther <
> >>>> >>> twal...@apache.org> 于2024年3月13日周三 01:24写
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 道：
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lincoln & Ron,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for
> >>>> >>> proposing this FLIP. I think a design
> >>>> >>>>> similar
> >>>> >>>>>>>>> to
> >>>> >>>>>>>>>>>>>>>>>>>>> what
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> you
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> propose has been
> >>>> >>> in the heads of many people, however,
> >>>> >>>>>>>>> I'm
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wondering
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this will fit
> >>>> >>> into the bigger picture.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I haven't deeply
> >>>> >>> reviewed the FLIP yet, but would like
> >>>> >>>>> to
> >>>> >>>>>>>>>>> ask
> >>>> >>>>>>>>>>>>>>>>>>>>> some
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initial questions:
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Flink has
> >>>> >>> introduced the concept of Dynamic Tables many
> >>>> >>>>>>>>>>> years
> >>>> >>>>>>>>>>>>>>>>>>>>> ago.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does the term
> >>>> >>> "Dynamic Table" fit into Flink's regular
> >>>> >>>>>>>>>>> tables
> >>>> >>>>>>>>>>>>>>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how does it
> >>>> >>> relate to Table API?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I fear that
> >>>> >>> adding the DYNAMIC TABLE keyword could
> >>>> >>>>> cause
> >>>> >>>>>>>>>>>>>>>>>>>>> confusion
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users, because a
> >>>> >>> term for regular CREATE TABLE (that
> >>>> >>>>> can
> >>>> >>>>>>>>> be
> >>>> >>>>>>>>>>>>>>>>>>>>> "kind
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamic" as well
> >>>> >>> and is backed by a changelog) is then
> >>>> >>>>>>>>>>>>> missing.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> given that we
> >>>> >>> call our connectors for those tables,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DynamicTableSource
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>> >>> DynamicTableSink.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In general, I
> >>>> >>> find it contradicting that a TABLE can be
> >>>> >>>>>>>>>>>>>>>>>>>>> "paused"
> >>>> >>>>>>>>>>>>>>>>>>>>>>> or
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "resumed". From
> >>>> >>> an English language perspective, this
> >>>> >>>>>>>>> does
> >>>> >>>>>>>>>>>>>>>>>>>>> sound
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> incorrect. In my
> >>>> >>> opinion (without much research yet), a
> >>>> >>>>>>>>>>>>>>>>>>>>> continuous
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating trigger
> >>>> >>> should rather be modelled as a CREATE
> >>>> >>>>>>>>>>>>>>>>>>>>>>> MATERIALIZED
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VIEW
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which users are
> >>>> >>> familiar with?) or a new concept such
> >>>> >>>>>>>>> as a
> >>>> >>>>>>>>>>>>>>>>>>>>> CREATE
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TASK
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (that can be
> >>>> >>> paused and resumed?).
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How do you
> >>>> >>> envision re-adding the functionality of a
> >>>> >>>>>>>>>>> statement
> >>>> >>>>>>>>>>>>>>>>>>>>>>> set,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fans out to
> >>>> >>> multiple tables? This is a very important
> >>>> >>>>> use
> >>>> >>>>>>>>>>> case
> >>>> >>>>>>>>>>>>>>>>>>>>> for
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pipelines.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since the early
> >>>> >>> days of Flink SQL, we were discussing
> >>>> >>>>>>>>>>> `SELECT
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> STREAM
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FROM T EMIT 5
> >>>> >>> MINUTES`. Your proposal seems to rephrase
> >>>> >>>>>>>>>>> STREAM
> >>>> >>>>>>>>>>>>>>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> EMIT,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into other
> >>>> >>> keywords DYNAMIC TABLE and FRESHNESS. But
> >>>> >>>>> the
> >>>> >>>>>>>>>>> core
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functionality is
> >>>> >>> still there. I'm wondering if we
> >>>> >>>>> should
> >>>> >>>>>>>>>>> widen
> >>>> >>>>>>>>>>>>>>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scope
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (maybe not part
> >>>> >>> of this FLIP but a new FLIP) to follow
> >>>> >>>>>>>>> the
> >>>> >>>>>>>>>>>>>>>>>>>>>>> standard
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> closely. Making
> >>>> >>> `SELECT * FROM t` bounded by default
> >>>> >>>>> and
> >>>> >>>>>>>>> use
> >>>> >>>>>>>>>>>>>>>>>>>>> new
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for the dynamic
> >>>> >>> behavior. Flink 2.0 would be the
> >>>> >>>>> perfect
> >>>> >>>>>>>>>>> time
> >>>> >>>>>>>>>>>>>>>>>>>>> for
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> however, it would
> >>>> >>> require careful discussions. What do
> >>>> >>>>>>>>> you
> >>>> >>>>>>>>>>>>>>>>>>>>> think?
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Timo
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 11.03.24
> >>>> >>> 08:23, Ron liu wrote:
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Dev
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lincoln Lee
> >>>> >>> and I would like to start a discussion
> >>>> >>>>> about
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> FLIP-435:
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Introduce a
> >>>> >>> New Dynamic Table for Simplifying Data
> >>>> >>>>>>>>>>>>> Pipelines.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This FLIP is
> >>>> >>> designed to simplify the development of
> >>>> >>>>>>>>> data
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> processing
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pipelines.
> >>>> >>> With Dynamic Tables with uniform SQL
> >>>> >>>>>>>>> statements
> >>>> >>>>>>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freshness,
> >>>> >>> users can define batch and streaming
> >>>> >>>>>>>>>>>>>>>>>>>>> transformations
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data in the
> >>>> >>> same way, accelerate ETL pipeline
> >>>> >>>>>>>>> development,
> >>>> >>>>>>>>>>>>> and
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> manage
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> task
> >>>> >>> scheduling automatically.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For more
> >>>> >>> details, see FLIP-435 [1]. Looking forward to
> >>>> >>>>>>>>> your
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback.
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lincoln & Ron
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>>>
> >>>> >>>>>>>>>>>
> >>>> >>>>>>>>>
> >>>> >>>>>>>
> >>>> >>>>>
> >>>> >>>
> >>>> >>
> >>>> >
> >>>>
> >>>>
>
