Sorry for coming very late to this thread. I have not contributed much to Flink publicly for quite some time but I have been involved with Flink, daily, for years now and I'm keenly interested in where we take Flink SQL going forward.
Thanks for the proposal!! I think it's definitely a step in the right direction and I'm thrilled this is happening. The end state I have in mind is that we get rid of execution modes as something users have to think about and instead make sure the SQL a user writes completely describes their intent. In the case of this proposal the intent a user has is that the system continually maintains an object (whatever we decide to call it) that is the result of their query and further that these can be easily chained together into declarative data pipelines. I would think it would be very unsurprising to users to call this a MATERIALIZED VIEW, except for the fact that this object can also accept updates via one-off DML statements. However, I don't think this object *should* accept updates via one-off DML statements so I may be the odd man out here. I would like to dive into this a little more if at all possible. The reasoning I've seen mentioned is GDPR requirements so can we dig into that specifically? I am not terribly familiar with the exact GDPR requirements but I should think that the solution to deleting data is to delete it in the upstream tables which would appropriately update any downstream MVs (or whatever we call it). So, with that context and the desire to explore the GDPR requirements a little more I would vote like so: (1) Materialized View, it should work as expected to SQL users, no new concept, no direct updates, dig into GDPR requirements though. (2) Dynamic Table, just follow the Snowflake precedent. I'm actually against all of the other proposed names so I rank them equally last. I don't think we need yet another new concept for this. I think that will just add to users' confusion and learning curve which is already substantial with Flink. We need to make things easier rather than harder. All of that said, I think these discussions may be a bit easier if they were part of a shared longer term vision for Flink SQL overall. You can see this from the little bits of side discussion that come up even in this thread. I'm not quite sure how to address that though. I will however give an example. I think that longer term the Flink SQL query text alone should dictate everything the system should do and we shouldn't rely on things like runtime execution modes at all. This means, for example, that a SELECT statement should always be a point in time query and immediately return results over the current data set and terminate. This also holds for an INSERT INTO query for that matter, and CTAS. A continuous query that perpetually maintains some view in the background should really have a distinct syntax. Basically Flink SQL should behave in a way that is unsurprising to users of existing database systems. Anyway, the point is that maybe we need a high level sketch of where we're going so we can make sure it all hangs together nicely. That all said I do think CREATE MATERIALIZED is a step in the right direction but we should figure out the GDPR stuff and the overall direction for Flink SQL going forward as well. On Wed, Apr 10, 2024 at 6:16 AM Dawid Wysakowicz <dwysakow...@apache.org> wrote: > Hi all, > I thought I'd cast my vote as well to give extra data: > > > 1. Materialized Table > 2. Materialized View (generally speaking I am not too concerned with > using a View here, but since there are concerns around updating a view I > put it second) > > I think what is suggested in this FLIP is really close to what MATERIALIZED > VIEWS do already, that's why I very much prefer any of the two options > above over any of the remaining candidates, but if I were to order them it > would be: > > 3. Refresh Table (it says what it does) > 4. Live Table - a new concept to explain, "live" can be interpreted in many > ways > 5. Derived Table - does not say much > > Best, > Dawid > > On Wed, 10 Apr 2024 at 04:50, Jark Wu <imj...@gmail.com> wrote: > > > I have been following up on the discussion, it's a great FLIP to further > > unify stream and batch ETL pipelines. Thanks for the proposal! > > > > Here is my ranking: > > > > 1. Materialized Table -> "The table materializes the results of a query > > that you specify", this can reflect what we want and doesn't conflict > with > > any SQL standard. > > 2. Derived Table -> easy to understand and write, but need to extend the > > standard > > 3. Live Table -> looks too much like Databrick's Delta Live Table. > > 4. Materialized View -> looks weird to insert/update a view. > > > > > > Best, > > Jark > > > > > > > > > > On Wed, 10 Apr 2024 at 09:57, Becket Qin <becket....@gmail.com> wrote: > > > > > Thanks for the proposal. I like the FLIP. > > > > > > My ranking: > > > > > > 1. Refresh(ing) / Live Table -> easy to understand and implies the > > dynamic > > > characteristic > > > > > > 2. Derived Table -> easy to understand. > > > > > > 3. Materialized Table -> sounds like just a table with physical data > > stored > > > somewhere. > > > > > > 4. Materialized View -> modifying a view directly is a little weird. > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > On Tue, Apr 9, 2024 at 5:46 AM Lincoln Lee <lincoln.8...@gmail.com> > > wrote: > > > > > > > Thanks Ron and Timo for your proposal! > > > > > > > > Here is my ranking: > > > > > > > > 1. Derived table -> extend the persistent semantics of derived table > in > > > SQL > > > > standard, with a strong association with query, and has industry > > > > precedents > > > > such as Google Looker. > > > > > > > > 2. Live Table -> an alternative for 'dynamic table' > > > > > > > > 3. Materialized Table -> combination of the Materialized View and > > Table, > > > > but > > > > still a table which accept data changes > > > > > > > > 4. Materialized View -> need to extend understanding of the view to > > > accept > > > > data changes > > > > > > > > The reason for not adding 'Refresh Table' is I don't want to tell the > > > user > > > > to 'refresh a refresh table'. > > > > > > > > > > > > Best, > > > > Lincoln Lee > > > > > > > > > > > > Ron liu <ron9....@gmail.com> 于2024年4月9日周二 20:11写道： > > > > > > > > > Hi, Dev > > > > > > > > > > My rankings are: > > > > > > > > > > 1. Derived Table > > > > > 2. Materialized Table > > > > > 3. Live Table > > > > > 4. Materialized View > > > > > > > > > > Best, > > > > > Ron > > > > > > > > > > > > > > > > > > > > Ron liu <ron9....@gmail.com> 于2024年4月9日周二 20:07写道： > > > > > > > > > > > Hi, Dev > > > > > > > > > > > > After several rounds of discussion, there is currently no > consensus > > > on > > > > > the > > > > > > name of the new concept. Timo has proposed that we decide the > name > > > > > through > > > > > > a vote. This is a good solution when there is no clear > preference, > > so > > > > we > > > > > > will adopt this approach. > > > > > > > > > > > > Regarding the name of the new concept, there are currently five > > > > > candidates: > > > > > > 1. Derived Table -> taken by SQL standard > > > > > > 2. Materialized Table -> similar to SQL materialized view but a > > table > > > > > > 3. Live Table -> similar to dynamic tables > > > > > > 4. Refresh Table -> states what it does > > > > > > 5. Materialized View -> needs to extend the standard to support > > > > modifying > > > > > > data > > > > > > > > > > > > For the above five candidates, everyone can give your rankings > > based > > > on > > > > > > your preferences. You can choose up to five options or only > choose > > > some > > > > > of > > > > > > them. > > > > > > We will use a scoring rule, where the* first rank gets 5 points, > > > second > > > > > > rank gets 4 points, third rank gets 3 points, fourth rank gets 2 > > > > points, > > > > > > and fifth rank gets 1 point*. > > > > > > After the voting closes, I will score all the candidates based on > > > > > > everyone's votes, and the candidate with the highest score will > be > > > > chosen > > > > > > as the name for the new concept. > > > > > > > > > > > > The voting will last up to 72 hours and is expected to close this > > > > Friday. > > > > > > I look forward to everyone voting on the name in this thread. Of > > > > course, > > > > > we > > > > > > also welcome new input regarding the name. > > > > > > > > > > > > Best, > > > > > > Ron > > > > > > > > > > > > Ron liu <ron9....@gmail.com> 于2024年4月9日周二 19:49写道： > > > > > > > > > > > >> Hi, Dev > > > > > >> > > > > > >> Sorry for my previous statement was not quite accurate. We will > > > hold a > > > > > >> vote for the name within this thread. > > > > > >> > > > > > >> Best, > > > > > >> Ron > > > > > >> > > > > > >> > > > > > >> Ron liu <ron9....@gmail.com> 于2024年4月9日周二 19:29写道： > > > > > >> > > > > > >>> Hi, Timo > > > > > >>> > > > > > >>> Thanks for your reply. > > > > > >>> > > > > > >>> I agree with you that sometimes naming is more difficult. When > no > > > one > > > > > >>> has a clear preference, voting on the name is a good solution, > so > > > > I'll > > > > > send > > > > > >>> a separate email for the vote, clarify the rules for the vote, > > then > > > > let > > > > > >>> everyone vote. > > > > > >>> > > > > > >>> One other point to confirm, in your ranking there is an option > > for > > > > > >>> Materialized View, does it stand for the UPDATING Materialized > > View > > > > > that > > > > > >>> you mentioned earlier in the discussion? If using Materialized > > > View I > > > > > think > > > > > >>> it is needed to extend it. > > > > > >>> > > > > > >>> Best, > > > > > >>> Ron > > > > > >>> > > > > > >>> Timo Walther <twal...@apache.org> 于2024年4月9日周二 17:20写道： > > > > > >>> > > > > > >>>> Hi Ron, > > > > > >>>> > > > > > >>>> yes naming is hard. But it will have large impact on > trainings, > > > > > >>>> presentations, and the mental model of users. Maybe the > easiest > > is > > > > to > > > > > >>>> collect ranking by everyone with some short justification: > > > > > >>>> > > > > > >>>> > > > > > >>>> My ranking (from good to not so good): > > > > > >>>> > > > > > >>>> 1. Refresh Table -> states what it does > > > > > >>>> 2. Materialized Table -> similar to SQL materialized view but > a > > > > table > > > > > >>>> 3. Live Table -> nice buzzword, but maybe still too close to > > > dynamic > > > > > >>>> tables? > > > > > >>>> 4. Materialized View -> a bit broader than standard but still > > very > > > > > >>>> similar > > > > > >>>> 5. Derived table -> taken by standard > > > > > >>>> > > > > > >>>> Regards, > > > > > >>>> Timo > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> On 07.04.24 11:34, Ron liu wrote: > > > > > >>>> > Hi, Dev > > > > > >>>> > > > > > > >>>> > This is a summary letter. After several rounds of > discussion, > > > > there > > > > > >>>> is a > > > > > >>>> > strong consensus about the FLIP proposal and the issues it > > aims > > > to > > > > > >>>> address. > > > > > >>>> > The current point of disagreement is the naming of the new > > > > concept. > > > > > I > > > > > >>>> have > > > > > >>>> > summarized the candidates as follows: > > > > > >>>> > > > > > > >>>> > 1. Derived Table (Inspired by Google Lookers) > > > > > >>>> > - Pros: Google Lookers has introduced this concept, > which > > > is > > > > > >>>> designed > > > > > >>>> > for building Looker's automated modeling, aligning with our > > > > purpose > > > > > >>>> for the > > > > > >>>> > stream-batch automatic pipeline. > > > > > >>>> > > > > > > >>>> > - Cons: The SQL standard uses derived table term > > > extensively, > > > > > >>>> vendors > > > > > >>>> > adopt this for simply referring to a table within a > subclause. > > > > > >>>> > > > > > > >>>> > 2. Materialized Table: It means materialize the query result > > to > > > > > table, > > > > > >>>> > similar to Db2 MQT (Materialized Query Tables). In addition, > > > > > Snowflake > > > > > >>>> > Dynamic Table's predecessor is also called Materialized > Table. > > > > > >>>> > > > > > > >>>> > 3. Updating Table (From Timo) > > > > > >>>> > > > > > > >>>> > 4. Updating Materialized View (From Timo) > > > > > >>>> > > > > > > >>>> > 5. Refresh/Live Table (From Martijn) > > > > > >>>> > > > > > > >>>> > As Martijn said, naming is a headache, looking forward to > more > > > > > >>>> valuable > > > > > >>>> > input from everyone. > > > > > >>>> > > > > > > >>>> > [1] > > > > > >>>> > > > > > > >>>> > > > > > > > > > > > > > > > https://cloud.google.com/looker/docs/derived-tables#persistent_derived_tables > > > > > >>>> > [2] > > > > > >>>> > > > > https://www.ibm.com/docs/en/db2/11.5?topic=tables-materialized-query > > > > > >>>> > [3] > > > > > >>>> > > > > > > >>>> > > > > > > > > > > > > > > > https://community.denodo.com/docs/html/browse/6.0/vdp/vql/materialized_tables/creating_materialized_tables/creating_materialized_tables > > > > > >>>> > > > > > > >>>> > Best, > > > > > >>>> > Ron > > > > > >>>> > > > > > > >>>> > Ron liu <ron9....@gmail.com> 于2024年4月7日周日 15:55写道： > > > > > >>>> > > > > > > >>>> >> Hi, Lorenzo > > > > > >>>> >> > > > > > >>>> >> Thank you for your insightful input. > > > > > >>>> >> > > > > > >>>> >>>>> I think the 2 above twisted the materialized view > concept > > to > > > > > more > > > > > >>>> than > > > > > >>>> >> just an optimization for accessing pre-computed > > > > aggregates/filters. > > > > > >>>> >> I think that concept (at least in my mind) is now adherent > to > > > the > > > > > >>>> >> semantics of the words themselves ("materialized" and > "view") > > > > than > > > > > >>>> on its > > > > > >>>> >> implementations in DBMs, as just a view on raw data that, > > > > > hopefully, > > > > > >>>> is > > > > > >>>> >> constantly updated with fresh results. > > > > > >>>> >> That's why I understand Timo's et al. objections. > > > > > >>>> >> > > > > > >>>> >> Your understanding of Materialized Views is correct. > However, > > > in > > > > > our > > > > > >>>> >> scenario, an important feature is the support for Update & > > > Delete > > > > > >>>> >> operations, which the current Materialized Views cannot > > > fulfill. > > > > As > > > > > >>>> we > > > > > >>>> >> discussed with Timo before, if Materialized Views needs to > > > > support > > > > > >>>> data > > > > > >>>> >> modifications, it would require an extension of new > keywords, > > > > such > > > > > as > > > > > >>>> >> CREATING xxx (UPDATING) MATERIALIZED VIEW. > > > > > >>>> >> > > > > > >>>> >>>>> Still, I don't understand why we need another type of > > > special > > > > > >>>> table. > > > > > >>>> >> Could you dive deep into the reasons why not simply adding > > the > > > > > >>>> FRESHNESS > > > > > >>>> >> parameter to standard tables? > > > > > >>>> >> > > > > > >>>> >> Firstly, I need to emphasize that we cannot achieve the > > design > > > > goal > > > > > >>>> of > > > > > >>>> >> FLIP through the CREATE TABLE syntax combined with a > > FRESHNESS > > > > > >>>> parameter. > > > > > >>>> >> The proposal of this FLIP is to use Dynamic Table + > > Continuous > > > > > >>>> Query, and > > > > > >>>> >> combine it with FRESHNESS to realize a streaming-batch > > > > unification. > > > > > >>>> >> However, CREATE TABLE is merely a metadata operation and > > cannot > > > > > >>>> >> automatically start a background refresh job. To achieve > the > > > > design > > > > > >>>> goal of > > > > > >>>> >> FLIP with standard tables, it would require extending the > > > CTAS[1] > > > > > >>>> syntax to > > > > > >>>> >> introduce the FRESHNESS keyword. We considered this design > > > > > >>>> initially, but > > > > > >>>> >> it has following problems: > > > > > >>>> >> > > > > > >>>> >> 1. Distinguishing a table created through CTAS as a > standard > > > > table > > > > > >>>> or as a > > > > > >>>> >> "special" standard table with an ongoing background refresh > > job > > > > > >>>> using the > > > > > >>>> >> FRESHNESS keyword is very obscure for users. > > > > > >>>> >> 2. It intrudes on the semantics of the CTAS syntax. > > Currently, > > > > > tables > > > > > >>>> >> created using CTAS only add table metadata to the Catalog > and > > > do > > > > > not > > > > > >>>> record > > > > > >>>> >> attributes such as query. There are also no ongoing > > background > > > > > >>>> refresh > > > > > >>>> >> jobs, and the data writing operation happens only once at > > table > > > > > >>>> creation. > > > > > >>>> >> 3. For the framework, when we perform a certain kind of > Alter > > > > Table > > > > > >>>> >> behavior for a table, for the table created by specifying > > > > FRESHNESS > > > > > >>>> and did > > > > > >>>> >> not specify the FRESHNESS created table behavior how to > > > > distinguish > > > > > >>>> , which > > > > > >>>> >> will also cause confusion. > > > > > >>>> >> > > > > > >>>> >> In terms of the design goal of combining Dynamic Table + > > > > Continuous > > > > > >>>> Query, > > > > > >>>> >> the FLIP proposal cannot be realized by only extending the > > > > current > > > > > >>>> stardand > > > > > >>>> >> tables, so a new kind of dynamic table needs to be > introduced > > > at > > > > > the > > > > > >>>> >> first-level concept. > > > > > >>>> >> > > > > > >>>> >> [1] > > > > > >>>> >> > > > > > >>>> > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#as-select_statement > > > > > >>>> >> > > > > > >>>> >> Best, > > > > > >>>> >> Ron > > > > > >>>> >> > > > > > >>>> >> <lorenzo.affe...@ververica.com.invalid> 于2024年4月3日周三 > > 22:25写道： > > > > > >>>> >> > > > > > >>>> >>> Hello everybody! > > > > > >>>> >>> Thanks for the FLIP as it looks amazing (and I think the > > prove > > > > is > > > > > >>>> this > > > > > >>>> >>> deep discussion it is provoking :)) > > > > > >>>> >>> > > > > > >>>> >>> I have a couple of comments to add to this: > > > > > >>>> >>> > > > > > >>>> >>> Even though I get the reason why you rejected MATERIALIZED > > > > VIEW, I > > > > > >>>> still > > > > > >>>> >>> like it a lot, and I would like to provide pointers on how > > the > > > > > >>>> materialized > > > > > >>>> >>> view concept twisted in last years: > > > > > >>>> >>> > > > > > >>>> >>> • Materialize DB (https://materialize.com/) > > > > > >>>> >>> • The famous talk by Martin Kleppmann "turning the > database > > > > inside > > > > > >>>> out" ( > > > > > >>>> >>> https://www.youtube.com/watch?v=fU9hR3kiOK0) > > > > > >>>> >>> > > > > > >>>> >>> I think the 2 above twisted the materialized view concept > to > > > > more > > > > > >>>> than > > > > > >>>> >>> just an optimization for accessing pre-computed > > > > > aggregates/filters. > > > > > >>>> >>> I think that concept (at least in my mind) is now adherent > > to > > > > the > > > > > >>>> >>> semantics of the words themselves ("materialized" and > > "view") > > > > than > > > > > >>>> on its > > > > > >>>> >>> implementations in DBMs, as just a view on raw data that, > > > > > >>>> hopefully, is > > > > > >>>> >>> constantly updated with fresh results. > > > > > >>>> >>> That's why I understand Timo's et al. objections. > > > > > >>>> >>> Still I understand there is no need to add confusion :) > > > > > >>>> >>> > > > > > >>>> >>> Still, I don't understand why we need another type of > > special > > > > > table. > > > > > >>>> >>> Could you dive deep into the reasons why not simply adding > > the > > > > > >>>> FRESHNESS > > > > > >>>> >>> parameter to standard tables? > > > > > >>>> >>> > > > > > >>>> >>> I would say that as a very seamless implementation with > the > > > goal > > > > > of > > > > > >>>> a > > > > > >>>> >>> unification of batch and streaming. > > > > > >>>> >>> If we stick to a unified world, I think that Flink should > > just > > > > > >>>> provide 1 > > > > > >>>> >>> type of table that is inherently dynamic. > > > > > >>>> >>> Now, depending on FRESHNESS objectives / connectors used > in > > > > WITH, > > > > > >>>> that > > > > > >>>> >>> table can be backed by a stream or batch job as you > > explained > > > in > > > > > >>>> your FLIP. > > > > > >>>> >>> > > > > > >>>> >>> Maybe I am totally missing the point :) > > > > > >>>> >>> > > > > > >>>> >>> Thank you in advance, > > > > > >>>> >>> Lorenzo > > > > > >>>> >>> On Apr 3, 2024 at 15:25 +0200, Martijn Visser < > > > > > >>>> martijnvis...@apache.org>, > > > > > >>>> >>> wrote: > > > > > >>>> >>>> Hi all, > > > > > >>>> >>>> > > > > > >>>> >>>> Thanks for the proposal. While the FLIP talks extensively > > on > > > > how > > > > > >>>> >>> Snowflake > > > > > >>>> >>>> has Dynamic Tables and Databricks has Delta Live Tables, > my > > > > > >>>> >>> understanding > > > > > >>>> >>>> is that Databricks has CREATE STREAMING TABLE [1] which > > > relates > > > > > >>>> with > > > > > >>>> >>> this > > > > > >>>> >>>> proposal. > > > > > >>>> >>>> > > > > > >>>> >>>> I do have concerns about using CREATE DYNAMIC TABLE, > > > > specifically > > > > > >>>> about > > > > > >>>> >>>> confusing the users who are familiar with Snowflake's > > > approach > > > > > >>>> where you > > > > > >>>> >>>> can't change the content via DML statements, while that > is > > > > > >>>> something > > > > > >>>> >>> that > > > > > >>>> >>>> would work in this proposal. Naming is hard of course, > but > > I > > > > > would > > > > > >>>> >>> probably > > > > > >>>> >>>> prefer something like CREATE CONTINUOUS TABLE, CREATE > > REFRESH > > > > > >>>> TABLE or > > > > > >>>> >>>> CREATE LIVE TABLE. > > > > > >>>> >>>> > > > > > >>>> >>>> Best regards, > > > > > >>>> >>>> > > > > > >>>> >>>> Martijn > > > > > >>>> >>>> > > > > > >>>> >>>> [1] > > > > > >>>> >>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-streaming-table.html > > > > > >>>> >>>> > > > > > >>>> >>>> On Wed, Apr 3, 2024 at 5:19 AM Ron liu < > ron9....@gmail.com > > > > > > > > wrote: > > > > > >>>> >>>> > > > > > >>>> >>>>> Hi, dev > > > > > >>>> >>>>> > > > > > >>>> >>>>> After offline discussion with Becket Qin, Lincoln Lee > and > > > Jark > > > > > >>>> Wu, we > > > > > >>>> >>> have > > > > > >>>> >>>>> improved some parts of the FLIP. > > > > > >>>> >>>>> > > > > > >>>> >>>>> 1. Add Full Refresh Mode section to clarify the > semantics > > of > > > > > full > > > > > >>>> >>> refresh > > > > > >>>> >>>>> mode. > > > > > >>>> >>>>> 2. Add Future Improvement section explaining why query > > > > statement > > > > > >>>> does > > > > > >>>> >>> not > > > > > >>>> >>>>> support references to temporary view and possible > > solutions. > > > > > >>>> >>>>> 3. The Future Improvement section explains a possible > > future > > > > > >>>> solution > > > > > >>>> >>> for > > > > > >>>> >>>>> dynamic table to support the modification of query > > > statements > > > > to > > > > > >>>> meet > > > > > >>>> >>> the > > > > > >>>> >>>>> common field-level schema evolution requirements of the > > > > > lakehouse. > > > > > >>>> >>>>> 4. The Refresh section emphasizes that the Refresh > command > > > and > > > > > the > > > > > >>>> >>>>> background refresh job can be executed in parallel, with > > no > > > > > >>>> >>> restrictions at > > > > > >>>> >>>>> the framework level. > > > > > >>>> >>>>> 5. Convert RefreshHandler into a plug-in interface to > > > support > > > > > >>>> various > > > > > >>>> >>>>> workflow schedulers. > > > > > >>>> >>>>> > > > > > >>>> >>>>> Best, > > > > > >>>> >>>>> Ron > > > > > >>>> >>>>> > > > > > >>>> >>>>> Ron liu <ron9....@gmail.com> 于2024年4月2日周二 10:28写道： > > > > > >>>> >>>>> > > > > > >>>> >>>>>>> Hi, Venkata krishnan > > > > > >>>> >>>>>>> > > > > > >>>> >>>>>>> Thank you for your involvement and suggestions, and > hope > > > > that > > > > > >>>> the > > > > > >>>> >>> design > > > > > >>>> >>>>>>> goals of this FLIP will be helpful to your business. > > > > > >>>> >>>>>>> > > > > > >>>> >>>>>>>>>>>>> 1. In the proposed FLIP, given the example for > the > > > > > >>>> >>> dynamic table, do > > > > > >>>> >>>>>>> the > > > > > >>>> >>>>>>> data sources always come from a single lake storage > such > > > as > > > > > >>>> >>> Paimon or > > > > > >>>> >>>>> does > > > > > >>>> >>>>>>> the same proposal solve for 2 disparate storage > systems > > > like > > > > > >>>> >>> Kafka and > > > > > >>>> >>>>>>> Iceberg where Kafka events are ETLed to Iceberg > similar > > to > > > > > >>>> Paimon? > > > > > >>>> >>>>>>> Basically the lambda architecture that is mentioned in > > the > > > > > FLIP > > > > > >>>> >>> as well. > > > > > >>>> >>>>>>> I'm wondering if it is possible to switch b/w sources > > > based > > > > on > > > > > >>>> the > > > > > >>>> >>>>>>> execution mode, for eg: if it is backfill operation, > > > switch > > > > > to a > > > > > >>>> >>> data > > > > > >>>> >>>>> lake > > > > > >>>> >>>>>>> storage system like Iceberg, otherwise an event > > streaming > > > > > system > > > > > >>>> >>> like > > > > > >>>> >>>>>>> Kafka. > > > > > >>>> >>>>>>> > > > > > >>>> >>>>>>> Dynamic table is a design abstraction at the framework > > > level > > > > > and > > > > > >>>> >>> is not > > > > > >>>> >>>>>>> tied to the physical implementation of the connector. > > If a > > > > > >>>> >>> connector > > > > > >>>> >>>>>>> supports a combination of Kafka and lake storage, this > > > works > > > > > >>>> fine. > > > > > >>>> >>>>>>> > > > > > >>>> >>>>>>>>>>>>> 2. What happens in the context of a bootstrap > > > (batch) > > > > + > > > > > >>>> >>> nearline > > > > > >>>> >>>>> update > > > > > >>>> >>>>>>> (streaming) case that are stateful applications? What > I > > > mean > > > > > by > > > > > >>>> >>> that is, > > > > > >>>> >>>>>>> will the state from the batch application be > transferred > > > to > > > > > the > > > > > >>>> >>> nearline > > > > > >>>> >>>>>>> application after the bootstrap execution is complete? > > > > > >>>> >>>>>>> > > > > > >>>> >>>>>>> I think this is another orthogonal thing, something > that > > > > > >>>> FLIP-327 > > > > > >>>> >>> tries > > > > > >>>> >>>>> to > > > > > >>>> >>>>>>> address, not directly related to Dynamic Table. > > > > > >>>> >>>>>>> > > > > > >>>> >>>>>>> [1] > > > > > >>>> >>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-327%3A+Support+switching+from+batch+to+stream+mode+to+improve+throughput+when+processing+backlog+data > > > > > >>>> >>>>>>> > > > > > >>>> >>>>>>> Best, > > > > > >>>> >>>>>>> Ron > > > > > >>>> >>>>>>> > > > > > >>>> >>>>>>> Venkatakrishnan Sowrirajan <vsowr...@asu.edu> > > > 于2024年3月30日周六 > > > > > >>>> >>> 07:06写道： > > > > > >>>> >>>>>>> > > > > > >>>> >>>>>>>>> Ron and Lincoln, > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>>>>>> Great proposal and interesting discussion for adding > > > > support > > > > > >>>> >>> for dynamic > > > > > >>>> >>>>>>>>> tables within Flink. > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>>>>>> At LinkedIn, we are also trying to solve > > compute/storage > > > > > >>>> >>> convergence for > > > > > >>>> >>>>>>>>> similar problems discussed as part of this FLIP, > > > > > specifically > > > > > >>>> >>> periodic > > > > > >>>> >>>>>>>>> backfill, bootstrap + nearline update use cases > using > > > > single > > > > > >>>> >>>>>>>>> implementation > > > > > >>>> >>>>>>>>> of business logic (single script). > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>>>>>> Few clarifying questions: > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>>>>>> 1. In the proposed FLIP, given the example for the > > > dynamic > > > > > >>>> >>> table, do the > > > > > >>>> >>>>>>>>> data sources always come from a single lake storage > > such > > > > as > > > > > >>>> >>> Paimon or > > > > > >>>> >>>>> does > > > > > >>>> >>>>>>>>> the same proposal solve for 2 disparate storage > > systems > > > > like > > > > > >>>> >>> Kafka and > > > > > >>>> >>>>>>>>> Iceberg where Kafka events are ETLed to Iceberg > > similar > > > to > > > > > >>>> >>> Paimon? > > > > > >>>> >>>>>>>>> Basically the lambda architecture that is mentioned > in > > > the > > > > > >>>> >>> FLIP as well. > > > > > >>>> >>>>>>>>> I'm wondering if it is possible to switch b/w > sources > > > > based > > > > > on > > > > > >>>> >>> the > > > > > >>>> >>>>>>>>> execution mode, for eg: if it is backfill operation, > > > > switch > > > > > to > > > > > >>>> >>> a data > > > > > >>>> >>>>> lake > > > > > >>>> >>>>>>>>> storage system like Iceberg, otherwise an event > > > streaming > > > > > >>>> >>> system like > > > > > >>>> >>>>>>>>> Kafka. > > > > > >>>> >>>>>>>>> 2. What happens in the context of a bootstrap > (batch) > > + > > > > > >>>> >>> nearline update > > > > > >>>> >>>>>>>>> (streaming) case that are stateful applications? > What > > I > > > > mean > > > > > >>>> >>> by that is, > > > > > >>>> >>>>>>>>> will the state from the batch application be > > transferred > > > > to > > > > > >>>> >>> the nearline > > > > > >>>> >>>>>>>>> application after the bootstrap execution is > complete? > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>>>>>> Regards > > > > > >>>> >>>>>>>>> Venkata krishnan > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>>>>>> On Mon, Mar 25, 2024 at 8:03 PM Ron liu < > > > > ron9....@gmail.com > > > > > > > > > > > >>>> >>> wrote: > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>>>>>>>> Hi, Timo > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> Thanks for your quick response, and your > suggestion. > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> Yes, this discussion has turned into confirming > > > whether > > > > > >>>> >>> it's a special > > > > > >>>> >>>>>>>>>>> table or a special MV. > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> 1. The key problem with MVs is that they don't > > support > > > > > >>>> >>> modification, > > > > > >>>> >>>>> so > > > > > >>>> >>>>>>>>> I > > > > > >>>> >>>>>>>>>>> prefer it to be a special table. Although the > > periodic > > > > > >>>> >>> refresh > > > > > >>>> >>>>> behavior > > > > > >>>> >>>>>>>>> is > > > > > >>>> >>>>>>>>>>> more characteristic of an MV, since we are > already a > > > > > >>>> >>> special table, > > > > > >>>> >>>>>>>>>>> supporting periodic refresh behavior is quite > > natural, > > > > > >>>> >>> similar to > > > > > >>>> >>>>>>>>> Snowflake > > > > > >>>> >>>>>>>>>>> dynamic tables. > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> 2. Regarding the keyword UPDATING, since the > current > > > > > >>>> >>> Regular Table is > > > > > >>>> >>>>> a > > > > > >>>> >>>>>>>>>>> Dynamic Table, which implies support for updating > > > > through > > > > > >>>> >>> Continuous > > > > > >>>> >>>>>>>>> Query, > > > > > >>>> >>>>>>>>>>> I think it is redundant to add the keyword > UPDATING. > > > In > > > > > >>>> >>> addition, > > > > > >>>> >>>>>>>>> UPDATING > > > > > >>>> >>>>>>>>>>> can not reflect the Continuous Query part, can not > > > > express > > > > > >>>> >>> the purpose > > > > > >>>> >>>>>>>>> we > > > > > >>>> >>>>>>>>>>> want to simplify the data pipeline through Dynamic > > > > Table + > > > > > >>>> >>> Continuous > > > > > >>>> >>>>>>>>>>> Query. > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> 3. From the perspective of the SQL standard > > > definition, > > > > I > > > > > >>>> >>> can > > > > > >>>> >>>>> understand > > > > > >>>> >>>>>>>>>>> your concerns about Derived Table, but is it > > possible > > > to > > > > > >>>> >>> make a slight > > > > > >>>> >>>>>>>>>>> adjustment to meet our needs? Additionally, as > > Lincoln > > > > > >>>> >>> mentioned, the > > > > > >>>> >>>>>>>>>>> Google Looker platform has introduced Persistent > > > Derived > > > > > >>>> >>> Table, and > > > > > >>>> >>>>>>>>> there > > > > > >>>> >>>>>>>>>>> are precedents in the industry; could Derived > Table > > > be a > > > > > >>>> >>> candidate? > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> Of course, look forward to your better > suggestions. > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> Best, > > > > > >>>> >>>>>>>>>>> Ron > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> Timo Walther <twal...@apache.org> 于2024年3月25日周一 > > > > 18:49写道： > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> After thinking about this more, this discussion > > > boils > > > > > >>>> >>> down to > > > > > >>>> >>>>> whether > > > > > >>>> >>>>>>>>>>>>> this is a special table or a special > materialized > > > > > >>>> >>> view. In both > > > > > >>>> >>>>> cases, > > > > > >>>> >>>>>>>>>>>>> we would need to add a special keyword: > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> Either > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> CREATE UPDATING TABLE > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> or > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> CREATE UPDATING MATERIALIZED VIEW > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> I still feel that the periodic refreshing > behavior > > > is > > > > > >>>> >>> closer to a > > > > > >>>> >>>>> MV. > > > > > >>>> >>>>>>>>> If > > > > > >>>> >>>>>>>>>>>>> we add a special keyword to MV, the optimizer > > would > > > > > >>>> >>> know that the > > > > > >>>> >>>>> data > > > > > >>>> >>>>>>>>>>>>> cannot be used for query optimizations. > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> I will ask more people for their opinion. > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> Regards, > > > > > >>>> >>>>>>>>>>>>> Timo > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> On 25.03.24 10:45, Timo Walther wrote: > > > > > >>>> >>>>>>>>>>>>>>> Hi Ron and Lincoln, > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> thanks for the quick response and the very > > > > > >>>> >>> insightful discussion. > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>> we might limit future opportunities to > > > > > >>>> >>> optimize queries > > > > > >>>> >>>>>>>>>>>>>>>>> through automatic materialization rewriting > by > > > > > >>>> >>> allowing data > > > > > >>>> >>>>>>>>>>>>>>>>> modifications, thus losing the potential for > > > > > >>>> >>> such > > > > > >>>> >>>>> optimizations. > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> This argument makes a lot of sense to me. Due > to > > > > > >>>> >>> the updates, the > > > > > >>>> >>>>>>>>>>> system > > > > > >>>> >>>>>>>>>>>>>>> is not in full control of the persisted data. > > > > > >>>> >>> However, the system > > > > > >>>> >>>>> is > > > > > >>>> >>>>>>>>>>>>>>> still in full control of the job that powers > the > > > > > >>>> >>> refresh. So if > > > > > >>>> >>>>> the > > > > > >>>> >>>>>>>>>>>>>>> system manages all updating pipelines, it > could > > > > > >>>> >>> still leverage > > > > > >>>> >>>>>>>>>>> automatic > > > > > >>>> >>>>>>>>>>>>>>> materialization rewriting but without > leveraging > > > > > >>>> >>> the data at rest > > > > > >>>> >>>>>>>>> (only > > > > > >>>> >>>>>>>>>>>>>>> the data in flight). > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>> we are considering another candidate, > Derived > > > > > >>>> >>> Table, the term > > > > > >>>> >>>>>>>>>>> 'derive' > > > > > >>>> >>>>>>>>>>>>>>>>> suggests a query, and 'table' retains > > > > > >>>> >>> modifiability. This > > > > > >>>> >>>>>>>>> approach > > > > > >>>> >>>>>>>>>>>>>>>>> would not disrupt our current concept of a > > > > > >>>> >>> dynamic table > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> I did some research on this term. The SQL > > standard > > > > > >>>> >>> uses the term > > > > > >>>> >>>>>>>>>>>>>>> "derived table" extensively (defined in > section > > > > > >>>> >>> 4.17.3). Thus, a > > > > > >>>> >>>>>>>>> lot of > > > > > >>>> >>>>>>>>>>>>>>> vendors adopt this for simply referring to a > > table > > > > > >>>> >>> within a > > > > > >>>> >>>>>>>>> subclause: > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://dev.mysql.com/doc/refman/8.0/en/derived-tables.html__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j735ghdiMp$ > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://infocenter.sybase.com/help/topic/com.sybase.infocenter.dc32300.1600/doc/html/san1390612291252.html__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j737h1gRux$ > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://www.c-sharpcorner.com/article/derived-tables-vs-common-table-expressions/__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j739bWIEcL$ > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://stackoverflow.com/questions/26529804/what-are-the-derived-tables-in-my-explain-statement__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j739HnGtQf$ > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://www.sqlservercentral.com/articles/sql-derived-tables__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j737DeBiqg$ > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> Esp. the latter example is interesting, SQL > > Server > > > > > >>>> >>> allows things > > > > > >>>> >>>>>>>>> like > > > > > >>>> >>>>>>>>>>>>>>> this on derived tables: > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> UPDATE T SET Name='Timo' FROM (SELECT * FROM > > > > > >>>> >>> Product) AS T > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> SELECT * FROM Product; > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> Btw also Snowflake's dynamic table state: > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>> Because the content of a dynamic table is > > > > > >>>> >>> fully determined > > > > > >>>> >>>>>>>>>>>>>>>>> by the given query, the content cannot be > > > > > >>>> >>> changed by using DML. > > > > > >>>> >>>>>>>>>>>>>>>>> You don’t insert, update, or delete the rows > > > > > >>>> >>> in a dynamic > > > > > >>>> >>>>> table. > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> So a new term makes a lot of sense. > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> How about using `UPDATING`? > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> CREATE UPDATING TABLE > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> This reflects that modifications can be made > and > > > > > >>>> >>> from an > > > > > >>>> >>>>>>>>>>>>>>> English-language perspective you can PAUSE or > > > > > >>>> >>> RESUME the UPDATING. > > > > > >>>> >>>>>>>>>>>>>>> Thus, a user can define UPDATING interval and > > > mode? > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> Looking forward to your thoughts. > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> Regards, > > > > > >>>> >>>>>>>>>>>>>>> Timo > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> On 25.03.24 07:09, Ron liu wrote: > > > > > >>>> >>>>>>>>>>>>>>>>> Hi, Ahmed > > > > > >>>> >>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>> Thanks for your feedback. > > > > > >>>> >>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>> Regarding your question: > > > > > >>>> >>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>> I want to iterate on Timo's comments > > > > > >>>> >>> regarding the confusion > > > > > >>>> >>>>>>>>> between > > > > > >>>> >>>>>>>>>>>>>>>>> "Dynamic Table" and current Flink "Table". > > > > > >>>> >>> Should the refactoring > > > > > >>>> >>>>>>>>> of > > > > > >>>> >>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>> system happen in 2.0, should we rename it in > > > > > >>>> >>> this Flip ( as the > > > > > >>>> >>>>>>>>>>>>>>>>> suggestions > > > > > >>>> >>>>>>>>>>>>>>>>> in the thread ) and address the holistic > > > > > >>>> >>> changes in a separate > > > > > >>>> >>>>> Flip > > > > > >>>> >>>>>>>>>>>>>>>>> for 2.0? > > > > > >>>> >>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>> Lincoln proposed a new concept in reply to > > > > > >>>> >>> Timo: Derived Table, > > > > > >>>> >>>>>>>>> which > > > > > >>>> >>>>>>>>>>>>>>>>> is a > > > > > >>>> >>>>>>>>>>>>>>>>> combination of Dynamic Table + Continuous > > > > > >>>> >>> Query, and the use of > > > > > >>>> >>>>>>>>>>> Derived > > > > > >>>> >>>>>>>>>>>>>>>>> Table will not conflict with existing > > concepts, > > > > > >>>> >>> what do you > > > > > >>>> >>>>> think? > > > > > >>>> >>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>> I feel confused with how it is further > with > > > > > >>>> >>> other components, > > > > > >>>> >>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>> examples provided feel like a standalone ETL > > > > > >>>> >>> job, could you > > > > > >>>> >>>>>>>>> provide in > > > > > >>>> >>>>>>>>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>> FLIP an example where the table is further > > used > > > > > >>>> >>> in subsequent > > > > > >>>> >>>>>>>>> queries > > > > > >>>> >>>>>>>>>>>>>>>>> (specially in batch mode). > > > > > >>>> >>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>> Thanks for your suggestion, I added how to > use > > > > > >>>> >>> Dynamic Table in > > > > > >>>> >>>>>>>>> FLIP > > > > > >>>> >>>>>>>>>>>>> user > > > > > >>>> >>>>>>>>>>>>>>>>> story section, Dynamic Table can be > referenced > > > > > >>>> >>> by downstream > > > > > >>>> >>>>>>>>> Dynamic > > > > > >>>> >>>>>>>>>>>>>>>>> Table > > > > > >>>> >>>>>>>>>>>>>>>>> and can also support OLAP queries. > > > > > >>>> >>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>> Best, > > > > > >>>> >>>>>>>>>>>>>>>>> Ron > > > > > >>>> >>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>> Ron liu <ron9....@gmail.com> 于2024年3月23日周六 > > > > > >>>> >>> 10:35写道： > > > > > >>>> >>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>> Hi, Feng > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>> Thanks for your feedback. > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> Although currently we restrict users > from > > > > > >>>> >>> modifying the query, > > > > > >>>> >>>>> I > > > > > >>>> >>>>>>>>>>>>> wonder > > > > > >>>> >>>>>>>>>>>>>>>>>>> if > > > > > >>>> >>>>>>>>>>>>>>>>>>> we can provide a better way to help users > > > > > >>>> >>> rebuild it without > > > > > >>>> >>>>>>>>>>> affecting > > > > > >>>> >>>>>>>>>>>>>>>>>>> downstream OLAP queries. > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>> Considering the problem of data > consistency, > > > > > >>>> >>> so in the first > > > > > >>>> >>>>> step > > > > > >>>> >>>>>>>>> we > > > > > >>>> >>>>>>>>>>>>> are > > > > > >>>> >>>>>>>>>>>>>>>>>>> strictly limited in semantics and do not > > > > > >>>> >>> support modify the > > > > > >>>> >>>>> query. > > > > > >>>> >>>>>>>>>>>>>>>>>>> This is > > > > > >>>> >>>>>>>>>>>>>>>>>>> really a good problem, one of my ideas is > to > > > > > >>>> >>> introduce a syntax > > > > > >>>> >>>>>>>>>>>>>>>>>>> similar to > > > > > >>>> >>>>>>>>>>>>>>>>>>> SWAP [1], which supports exchanging two > > > > > >>>> >>> Dynamic Tables. > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> From the documentation, the definitions > > > > > >>>> >>> SQL and job > > > > > >>>> >>>>> information > > > > > >>>> >>>>>>>>> are > > > > > >>>> >>>>>>>>>>>>>>>>>>> stored in the Catalog. Does this mean that > > > > > >>>> >>> if a system needs to > > > > > >>>> >>>>>>>>> adapt > > > > > >>>> >>>>>>>>>>>>> to > > > > > >>>> >>>>>>>>>>>>>>>>>>> Dynamic Tables, it also needs to store > > > > > >>>> >>> Flink's job information > > > > > >>>> >>>>> in > > > > > >>>> >>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>> corresponding system? > > > > > >>>> >>>>>>>>>>>>>>>>>>> For example, does MySQL's Catalog need to > > > > > >>>> >>> store flink job > > > > > >>>> >>>>>>>>> information > > > > > >>>> >>>>>>>>>>>>> as > > > > > >>>> >>>>>>>>>>>>>>>>>>> well? > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>> Yes, currently we need to rely on Catalog > to > > > > > >>>> >>> store refresh job > > > > > >>>> >>>>>>>>>>>>>>>>>>> information. > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> Users still need to consider how much > > > > > >>>> >>> memory is being used, how > > > > > >>>> >>>>>>>>>>> large > > > > > >>>> >>>>>>>>>>>>>>>>>>> the concurrency is, which type of state > > > > > >>>> >>> backend is being used, > > > > > >>>> >>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>> may need > > > > > >>>> >>>>>>>>>>>>>>>>>>> to set TTL expiration. > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>> Similar to the current practice, job > > > > > >>>> >>> parameters can be set via > > > > > >>>> >>>>> the > > > > > >>>> >>>>>>>>>>>>> Flink > > > > > >>>> >>>>>>>>>>>>>>>>>>> conf or SET commands > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> When we submit a refresh command, can we > > > > > >>>> >>> help users detect if > > > > > >>>> >>>>>>>>> there > > > > > >>>> >>>>>>>>>>>>> are > > > > > >>>> >>>>>>>>>>>>>>>>>>> any > > > > > >>>> >>>>>>>>>>>>>>>>>>> running jobs and automatically stop them > > > > > >>>> >>> before executing the > > > > > >>>> >>>>>>>>> refresh > > > > > >>>> >>>>>>>>>>>>>>>>>>> command? Then wait for it to complete > before > > > > > >>>> >>> restarting the > > > > > >>>> >>>>>>>>>>> background > > > > > >>>> >>>>>>>>>>>>>>>>>>> streaming job? > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>> Purely from a technical implementation > point > > > > > >>>> >>> of view, your > > > > > >>>> >>>>>>>>> proposal > > > > > >>>> >>>>>>>>>>> is > > > > > >>>> >>>>>>>>>>>>>>>>>>> doable, but it would be more costly. Also > I > > > > > >>>> >>> think data > > > > > >>>> >>>>> consistency > > > > > >>>> >>>>>>>>>>>>>>>>>>> itself > > > > > >>>> >>>>>>>>>>>>>>>>>>> is the responsibility of the user, similar > > > > > >>>> >>> to how Regular Table > > > > > >>>> >>>>> is > > > > > >>>> >>>>>>>>>>>>>>>>>>> now also > > > > > >>>> >>>>>>>>>>>>>>>>>>> the responsibility of the user, so it's > > > > > >>>> >>> consistent with its > > > > > >>>> >>>>>>>>> behavior > > > > > >>>> >>>>>>>>>>>>>>>>>>> and no > > > > > >>>> >>>>>>>>>>>>>>>>>>> additional guarantees are made at the > engine > > > > > >>>> >>> level. > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>> Best, > > > > > >>>> >>>>>>>>>>>>>>>>>>> Ron > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>> Ahmed Hamdy <hamdy10...@gmail.com> > > > > > >>>> >>> 于2024年3月22日周五 23:50写道： > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> Hi Ron, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> Sorry for joining the discussion late, > > > > > >>>> >>> thanks for the effort. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> I think the base idea is great, however > I > > > > > >>>> >>> have a couple of > > > > > >>>> >>>>>>>>> comments: > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> - I want to iterate on Timo's comments > > > > > >>>> >>> regarding the confusion > > > > > >>>> >>>>>>>>>>> between > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> "Dynamic Table" and current Flink > > > > > >>>> >>> "Table". Should the > > > > > >>>> >>>>>>>>> refactoring of > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> system happen in 2.0, should we rename > it > > > > > >>>> >>> in this Flip ( as the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> suggestions > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> in the thread ) and address the holistic > > > > > >>>> >>> changes in a separate > > > > > >>>> >>>>>>>>> Flip > > > > > >>>> >>>>>>>>>>>>> for > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> 2.0? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> - I feel confused with how it is further > > > > > >>>> >>> with other components, > > > > > >>>> >>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> examples provided feel like a standalone > > > > > >>>> >>> ETL job, could you > > > > > >>>> >>>>>>>>> provide > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> in the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> FLIP an example where the table is > > > > > >>>> >>> further used in subsequent > > > > > >>>> >>>>>>>>>>> queries > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> (specially in batch mode). > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> - I really like the standard of keeping > > > > > >>>> >>> the unified batch and > > > > > >>>> >>>>>>>>>>>>> streaming > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> approach > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> Best Regards > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> Ahmed Hamdy > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> On Fri, 22 Mar 2024 at 12:07, Lincoln > Lee > > > > > >>>> >>> < > > > > > >>>> >>>>>>>>> lincoln.8...@gmail.com> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> wrote: > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> Hi Timo, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks for your thoughtful inputs! > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> Yes, expanding the MATERIALIZED > > > > > >>>> >>> VIEW(MV) could achieve the > > > > > >>>> >>>>> same > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> function, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> but our primary concern is that by > > > > > >>>> >>> using a view, we might > > > > > >>>> >>>>> limit > > > > > >>>> >>>>>>>>>>>>> future > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> opportunities > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> to optimize queries through automatic > > > > > >>>> >>> materialization > > > > > >>>> >>>>> rewriting > > > > > >>>> >>>>>>>>>>> [1], > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> leveraging > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> the support for MV by physical > > > > > >>>> >>> storage. This is because we > > > > > >>>> >>>>>>>>> would be > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> breaking > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> the intuitive semantics of a > > > > > >>>> >>> materialized view (a materialized > > > > > >>>> >>>>>>>>> view > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> represents > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> the result of a query) by allowing > > > > > >>>> >>> data modifications, thus > > > > > >>>> >>>>>>>>> losing > > > > > >>>> >>>>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> potential > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> for such optimizations. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> With these considerations in mind, we > > > > > >>>> >>> were inspired by Google > > > > > >>>> >>>>>>>>>>>>> Looker's > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> Persistent > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> Derived Table [2]. PDT is designed for > > > > > >>>> >>> building Looker's > > > > > >>>> >>>>>>>>> automated > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> modeling, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> aligning with our purpose for the > > > > > >>>> >>> stream-batch automatic > > > > > >>>> >>>>>>>>> pipeline. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> Therefore, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> we are considering another candidate, > > > > > >>>> >>> Derived Table, the term > > > > > >>>> >>>>>>>>>>>>> 'derive' > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> suggests a > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> query, and 'table' retains > > > > > >>>> >>> modifiability. This approach would > > > > > >>>> >>>>>>>>> not > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> disrupt > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> our current > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> concept of a dynamic table, preserving > > > > > >>>> >>> the future utility of > > > > > >>>> >>>>>>>>> MVs. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> Conceptually, a Derived Table is a > > > > > >>>> >>> Dynamic Table + Continuous > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> Query. By > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> introducing > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> a new concept Derived Table for this > > > > > >>>> >>> FLIP, this makes all > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> concepts to > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> play > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> together nicely. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> What do you think about this? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> [1] > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://calcite.apache.org/docs/materialized_views.html__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j73_NFf4D5$ > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> [2] > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://cloud.google.com/looker/docs/derived-tables*persistent_derived_tables__;Iw!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j7382-2zI3$ > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> Lincoln Lee > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> Timo Walther <twal...@apache.org> > > > > > >>>> >>> 于2024年3月22日周五 17:54写道： > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Ron, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> thanks for the detailed answer. > > > > > >>>> >>> Sorry, for my late reply, we > > > > > >>>> >>>>>>>>> had a > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> conference that kept me busy. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> In the current concept[1], it > > > > > >>>> >>> actually includes: Dynamic > > > > > >>>> >>>>>>>>>>> Tables > > > > > >>>> >>>>>>>>>>>>> & > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> & Continuous Query. Dynamic > > > > > >>>> >>> Table is just an abstract > > > > > >>>> >>>>>>>>> logical > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> concept > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> This explanation makes sense to me. > > > > > >>>> >>> But the docs also say "A > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> continuous > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> query is evaluated on the dynamic > > > > > >>>> >>> table yielding a new > > > > > >>>> >>>>> dynamic > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> table.". > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> So even our regular CREATE TABLEs > > > > > >>>> >>> are considered dynamic > > > > > >>>> >>>>>>>>> tables. > > > > > >>>> >>>>>>>>>>>>> This > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> can also be seen in the diagram > > > > > >>>> >>> "Dynamic Table -> Continuous > > > > > >>>> >>>>>>>>> Query > > > > > >>>> >>>>>>>>>>>>> -> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> Dynamic Table". Currently, Flink > > > > > >>>> >>> queries can only be executed > > > > > >>>> >>>>>>>>> on > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> Dynamic > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> Tables. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> In essence, a materialized view > > > > > >>>> >>> represents the result of > > > > > >>>> >>>>> a > > > > > >>>> >>>>>>>>>>>>> query. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> Isn't that what your proposal does > > > > > >>>> >>> as well? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> the object of the suspend > > > > > >>>> >>> operation is the refresh task > > > > > >>>> >>>>> of > > > > > >>>> >>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> dynamic table > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> I understand that Snowflake uses > > > > > >>>> >>> the term [1] to merge their > > > > > >>>> >>>>>>>>>>>>> concepts > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> of > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> STREAM, TASK, and TABLE into one > > > > > >>>> >>> piece of concept. But Flink > > > > > >>>> >>>>>>>>> has > > > > > >>>> >>>>>>>>>>> no > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> concept of a "refresh task". Also, > > > > > >>>> >>> they already introduced > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> MATERIALIZED > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> VIEW. Flink is in the convenient > > > > > >>>> >>> position that the concept of > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> materialized views is not taken > > > > > >>>> >>> (reserved maybe for exactly > > > > > >>>> >>>>>>>>> this > > > > > >>>> >>>>>>>>>>> use > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> case?). And SQL standard concept > > > > > >>>> >>> could be "slightly adapted" > > > > > >>>> >>>>> to > > > > > >>>> >>>>>>>>>>> our > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> needs. Looking at other vendors > > > > > >>>> >>> like Postgres[2], they also > > > > > >>>> >>>>> use > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> `REFRESH` commands so why not > > > > > >>>> >>> adding additional commands such > > > > > >>>> >>>>>>>>> as > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> DELETE > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> or UPDATE. Oracle supports "ON > > > > > >>>> >>> PREBUILT TABLE clause tells > > > > > >>>> >>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> database > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> to use an existing table > > > > > >>>> >>> segment"[3] which comes closer to > > > > > >>>> >>>>>>>>> what we > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> want > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> as well. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> it is not intended to support > > > > > >>>> >>> data modification > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> This is an argument that I > > > > > >>>> >>> understand. But we as Flink could > > > > > >>>> >>>>>>>>> allow > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> data > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> modifications. This way we are only > > > > > >>>> >>> extending the standard > > > > > >>>> >>>>> and > > > > > >>>> >>>>>>>>>>> don't > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> introduce new concepts. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> If we can't agree on using > > > > > >>>> >>> MATERIALIZED VIEW concept. We > > > > > >>>> >>>>> should > > > > > >>>> >>>>>>>>>>> fix > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> our > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> syntax in a Flink 2.0 effort. > > > > > >>>> >>> Making regular tables bounded > > > > > >>>> >>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> dynamic > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> tables unbounded. We would be > > > > > >>>> >>> closer to the SQL standard with > > > > > >>>> >>>>>>>>> this > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> pave the way for the future. I > > > > > >>>> >>> would actually support this if > > > > > >>>> >>>>>>>>> all > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> concepts play together nicely. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> In the future, we can consider > > > > > >>>> >>> extending the statement > > > > > >>>> >>>>> set > > > > > >>>> >>>>>>>>>>>>> syntax > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> to > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> support the creation of multiple > > > > > >>>> >>> dynamic tables. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> It's good that we called the > > > > > >>>> >>> concept STATEMENT SET. This > > > > > >>>> >>>>>>>>> allows us > > > > > >>>> >>>>>>>>>>>>> to > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> defined CREATE TABLE within. Even > > > > > >>>> >>> if it might look a bit > > > > > >>>> >>>>>>>>>>> confusing. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> Regards, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> Timo > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> [1] > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://docs.snowflake.com/en/user-guide/dynamic-tables-about__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j73zexZBXu$ > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> [2] > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://www.postgresql.org/docs/current/sql-creatematerializedview.html__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j73zbNhvS7$ > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> [3] > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://oracle-base.com/articles/misc/materialized-views__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j739xS1kvD$ > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> On 21.03.24 04:14, Feng Jin wrote: > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ron and Lincoln > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving this > > > > > >>>> >>> discussion. I believe it will > > > > > >>>> >>>>> greatly > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> improve > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> convenience of managing user > > > > > >>>> >>> real-time pipelines. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I have some questions. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> *Regarding Limitations of > > > > > >>>> >>> Dynamic Table:* > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does not support modifying > > > > > >>>> >>> the select statement after the > > > > > >>>> >>>>>>>>>>> dynamic > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> table > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> is created. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Although currently we restrict > > > > > >>>> >>> users from modifying the > > > > > >>>> >>>>>>>>> query, I > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> wonder > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> if > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> we can provide a better way to > > > > > >>>> >>> help users rebuild it without > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> affecting > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> downstream OLAP queries. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> *Regarding the management of > > > > > >>>> >>> background jobs:* > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> 1. From the documentation, the > > > > > >>>> >>> definitions SQL and job > > > > > >>>> >>>>>>>>>>> information > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> are > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> stored in the Catalog. Does this > > > > > >>>> >>> mean that if a system needs > > > > > >>>> >>>>>>>>> to > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> adapt > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> to > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Dynamic Tables, it also needs to > > > > > >>>> >>> store Flink's job > > > > > >>>> >>>>>>>>> information in > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> corresponding system? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> For example, does MySQL's > > > > > >>>> >>> Catalog need to store flink job > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> information > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> as > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> well? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Users still need to consider > > > > > >>>> >>> how much memory is being > > > > > >>>> >>>>> used, > > > > > >>>> >>>>>>>>>>> how > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> large > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> the concurrency is, which type > > > > > >>>> >>> of state backend is being > > > > > >>>> >>>>> used, > > > > > >>>> >>>>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> may > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> need > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> to set TTL expiration. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> *Regarding the Refresh Part:* > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> If the refresh mode is > > > > > >>>> >>> continuous and a background job is > > > > > >>>> >>>>>>>>>>> running, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> caution should be taken with the > > > > > >>>> >>> refresh command as it can > > > > > >>>> >>>>>>>>> lead > > > > > >>>> >>>>>>>>>>> to > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> inconsistent data. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> When we submit a refresh > > > > > >>>> >>> command, can we help users detect > > > > > >>>> >>>>> if > > > > > >>>> >>>>>>>>>>> there > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> are > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> any > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> running jobs and automatically > > > > > >>>> >>> stop them before executing > > > > > >>>> >>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> refresh > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> command? Then wait for it to > > > > > >>>> >>> complete before restarting the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> background > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> streaming job? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Feng > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Mar 19, 2024 at 9:40 PM > > > > > >>>> >>> Lincoln Lee < > > > > > >>>> >>>>>>>>>>>>> lincoln.8...@gmail.com > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Yun, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you very much for your > > > > > >>>> >>> valuable input! > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Incremental mode is indeed an > > > > > >>>> >>> attractive idea, we have also > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> discussed > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this, but in the current > > > > > >>>> >>> design, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> we first provided two refresh > > > > > >>>> >>> modes: CONTINUOUS and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> FULL. Incremental mode can be > > > > > >>>> >>> introduced > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> once the execution layer has > > > > > >>>> >>> the capability. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> My answer for the two > > > > > >>>> >>> questions: > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, cascading is a good > > > > > >>>> >>> question. Current proposal > > > > > >>>> >>>>>>>>> provides a > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> freshness that defines a > > > > > >>>> >>> dynamic > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> table relative to the base > > > > > >>>> >>> table’s lag. If users need to > > > > > >>>> >>>>>>>>>>> consider > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> end-to-end freshness of > > > > > >>>> >>> multiple > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> cascaded dynamic tables, he > > > > > >>>> >>> can manually split them for > > > > > >>>> >>>>> now. > > > > > >>>> >>>>>>>>> Of > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> course, how to let multiple > > > > > >>>> >>> cascaded > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> or dependent dynamic tables > > > > > >>>> >>> complete the freshness > > > > > >>>> >>>>>>>>> definition > > > > > >>>> >>>>>>>>>>>>> in > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> a > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> simpler way, I think it can be > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> extended in the future. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cascading refresh is also a > > > > > >>>> >>> part we focus on discussing. In > > > > > >>>> >>>>>>>>> this > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> flip, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> we hope to focus as much as > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> possible on the core features > > > > > >>>> >>> (as it already involves a lot > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> things), > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> so we did not directly > > > > > >>>> >>> introduce related > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax. However, based on the > > > > > >>>> >>> current design, combined > > > > > >>>> >>>>>>>>> with > > > > > >>>> >>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> catalog and lineage, > > > > > >>>> >>> theoretically, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> users can also finish the > > > > > >>>> >>> cascading refresh. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lincoln Lee > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yun Tang <myas...@live.com> > > > > > >>>> >>> 于2024年3月19日周二 13:45写道： > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lincoln, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving this > > > > > >>>> >>> discussion, and I am so excited to > > > > > >>>> >>>>>>>>> see > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> this > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> topic > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> being discussed in the > > > > > >>>> >>> Flink community! > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From my point of view, > > > > > >>>> >>> instead of the work of unifying > > > > > >>>> >>>>>>>>>>>>> streaming > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> batch > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in DataStream API [1], > > > > > >>>> >>> this FLIP actually could make users > > > > > >>>> >>>>>>>>>>>>> benefit > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> from > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> one > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engine to rule batch & > > > > > >>>> >>> streaming. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we treat this FLIP as > > > > > >>>> >>> an open-source implementation of > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> Snowflake's > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamic tables [2], we > > > > > >>>> >>> still lack an incremental refresh > > > > > >>>> >>>>>>>>> mode > > > > > >>>> >>>>>>>>>>> to > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> make > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ETL near real-time with a > > > > > >>>> >>> much cheaper computation cost. > > > > > >>>> >>>>>>>>>>> However, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> I > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> think > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this could be done under > > > > > >>>> >>> the current design by introducing > > > > > >>>> >>>>>>>>>>>>> another > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> refresh > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode in the future. > > > > > >>>> >>> Although the extra work of incremental > > > > > >>>> >>>>>>>>> view > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> maintenance > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would be much larger. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the FLIP itself, I > > > > > >>>> >>> have several questions below: > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It seems this FLIP does > > > > > >>>> >>> not consider the lag of > > > > > >>>> >>>>> refreshes > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> across > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> ETL > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> layers from ODS ---> DWD > > > > > >>>> >>> ---> APP [3]. We currently only > > > > > >>>> >>>>>>>>>>> consider > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scheduler interval, which > > > > > >>>> >>> means we cannot use lag to > > > > > >>>> >>>>>>>>>>>>> automatically > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the upfront micro-batch > > > > > >>>> >>> jobs to do the work. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. To support the > > > > > >>>> >>> automagical refreshes, we should > > > > > >>>> >>>>> consider > > > > > >>>> >>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> lineage > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> in > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the catalog or somewhere > > > > > >>>> >>> else. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-134*3A*Batch*execution*for*the*DataStream*API__;JSsrKysrKw!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j7352JICzI$ > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2] > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://docs.snowflake.com/en/user-guide/dynamic-tables-about__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j73zexZBXu$ > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3] > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j735ghqpxk$ > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yun Tang > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>> ________________________________ > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From: Lincoln Lee < > > > > > >>>> >>> lincoln.8...@gmail.com> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent: Thursday, March 14, > > > > > >>>> >>> 2024 14:35 > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@flink.apache.org < > > > > > >>>> >>> dev@flink.apache.org> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] > > > > > >>>> >>> FLIP-435: Introduce a New Dynamic > > > > > >>>> >>>>>>>>> Table > > > > > >>>> >>>>>>>>>>>>> for > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Simplifying Data Pipelines > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jing, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your attention > > > > > >>>> >>> to this flip! I'll try to answer > > > > > >>>> >>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> following > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. How to define query > > > > > >>>> >>> of dynamic table? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Use flink sql or > > > > > >>>> >>> introducing new syntax? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If use flink sql, how > > > > > >>>> >>> to handle the difference in SQL > > > > > >>>> >>>>>>>>> between > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> streaming > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> batch processing? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, a query > > > > > >>>> >>> including window aggregate based on > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> processing > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> time? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or a query including > > > > > >>>> >>> global order by? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Similar to `CREATE TABLE > > > > > >>>> >>> AS query`, here the `query` also > > > > > >>>> >>>>>>>>> uses > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> Flink > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> sql > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> doesn't introduce a > > > > > >>>> >>> totally new syntax. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We will not change the > > > > > >>>> >>> status respect to > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the difference in > > > > > >>>> >>> functionality of flink sql itself on > > > > > >>>> >>>>>>>>>>> streaming > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> batch, for example, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the proctime window agg on > > > > > >>>> >>> streaming and global sort on > > > > > >>>> >>>>>>>>> batch > > > > > >>>> >>>>>>>>>>>>> that > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> you > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mentioned, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in fact, do not work > > > > > >>>> >>> properly in the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other mode, so when the > > > > > >>>> >>> user modifies the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> refresh mode of a dynamic > > > > > >>>> >>> table that is not supported, we > > > > > >>>> >>>>>>>>> will > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> throw > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> an > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Whether modify the > > > > > >>>> >>> query of dynamic table is allowed? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Or we could only > > > > > >>>> >>> refresh a dynamic table based on the > > > > > >>>> >>>>>>>>> initial > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> query? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes, in the current > > > > > >>>> >>> design, the query definition of the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamic table is not > > > > > >>>> >>> allowed > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to be modified, and you > > > > > >>>> >>> can only refresh the data based > > > > > >>>> >>>>>>>>> on > > > > > >>>> >>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initial definition. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. How to use dynamic > > > > > >>>> >>> table? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The dynamic table seems > > > > > >>>> >>> to be similar to the materialized > > > > > >>>> >>>>>>>>>>> view. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> Will > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> we > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something like > > > > > >>>> >>> materialized view rewriting during the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> optimization? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's true that dynamic > > > > > >>>> >>> table and materialized view > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are similar in some ways, > > > > > >>>> >>> but as Ron > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explains > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there are differences. In > > > > > >>>> >>> terms of optimization, automated > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization discovery > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> similar to that supported > > > > > >>>> >>> by calcite is also a potential > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> possibility, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> perhaps with the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> addition of automated > > > > > >>>> >>> rewriting in the future. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lincoln Lee > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ron liu < > > > > > >>>> >>> ron9....@gmail.com> 于2024年3月14日周四 14:01写道： > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Timo > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for later > > > > > >>>> >>> response, thanks for your feedback. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding your > > > > > >>>> >>> questions: > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Flink has introduced > > > > > >>>> >>> the concept of Dynamic Tables many > > > > > >>>> >>>>>>>>> years > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> ago. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> How > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does the term "Dynamic > > > > > >>>> >>> Table" fit into Flink's regular > > > > > >>>> >>>>>>>>> tables > > > > > >>>> >>>>>>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> also > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how does it relate to > > > > > >>>> >>> Table API? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I fear that adding > > > > > >>>> >>> the DYNAMIC TABLE keyword could cause > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> confusion > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users, because a > > > > > >>>> >>> term for regular CREATE TABLE (that can > > > > > >>>> >>>>>>>>> be > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> "kind > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> of > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamic" as well and > > > > > >>>> >>> is backed by a changelog) is then > > > > > >>>> >>>>>>>>>>> missing. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> Also > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> given that we call > > > > > >>>> >>> our connectors for those tables, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DynamicTableSource > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and DynamicTableSink. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In general, I find > > > > > >>>> >>> it contradicting that a TABLE can be > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> "paused" or > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "resumed". From an > > > > > >>>> >>> English language perspective, this > > > > > >>>> >>>>> does > > > > > >>>> >>>>>>>>>>>>> sound > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> incorrect. In my > > > > > >>>> >>> opinion (without much research yet), a > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> continuous > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating trigger > > > > > >>>> >>> should rather be modelled as a CREATE > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> MATERIALIZED > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VIEW > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which users are > > > > > >>>> >>> familiar with?) or a new concept such > > > > > >>>> >>>>> as > > > > > >>>> >>>>>>>>> a > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> CREATE > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> TASK > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (that can be paused > > > > > >>>> >>> and resumed?). > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In the current > > > > > >>>> >>> concept[1], it actually includes: Dynamic > > > > > >>>> >>>>>>>>>>> Tables > > > > > >>>> >>>>>>>>>>>>> & > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Continuous Query. > > > > > >>>> >>> Dynamic Table is just an abstract > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> logical concept > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> , which in its physical > > > > > >>>> >>> form represents either a table > > > > > >>>> >>>>> or a > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> changelog > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stream. It requires the > > > > > >>>> >>> combination with Continuous Query > > > > > >>>> >>>>>>>>> to > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> achieve > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamic updates of the > > > > > >>>> >>> target table similar to a > > > > > >>>> >>>>> database’s > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Materialized View. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We hope to upgrade the > > > > > >>>> >>> Dynamic Table to a real entity > > > > > >>>> >>>>> that > > > > > >>>> >>>>>>>>>>> users > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> can > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operate, which combines > > > > > >>>> >>> the logical concepts of Dynamic > > > > > >>>> >>>>>>>>>>> Tables + > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Continuous Query. By > > > > > >>>> >>> integrating the definition of tables > > > > > >>>> >>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> queries, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it can achieve > > > > > >>>> >>> functions similar to Materialized Views, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> simplifying > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users' data processing > > > > > >>>> >>> pipelines. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, the object of the > > > > > >>>> >>> suspend operation is the refresh > > > > > >>>> >>>>>>>>> task of > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamic table. The > > > > > >>>> >>> command `ALTER DYNAMIC TABLE > > > > > >>>> >>>>> table_name > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> SUSPEND > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> ` > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is actually a shorthand > > > > > >>>> >>> for `ALTER DYNAMIC TABLE > > > > > >>>> >>>>> table_name > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> SUSPEND > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> REFRESH` (if written in > > > > > >>>> >>> full for clarity, we can also > > > > > >>>> >>>>>>>>> modify > > > > > >>>> >>>>>>>>>>>>> it). > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Initially, we also > > > > > >>>> >>> considered Materialized Views > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> , but ultimately > > > > > >>>> >>> decided against them. Materialized views > > > > > >>>> >>>>>>>>> are > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> designed > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to enhance query > > > > > >>>> >>> performance for workloads that consist > > > > > >>>> >>>>> of > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> common, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> repetitive query > > > > > >>>> >>> patterns. In essence, a materialized > > > > > >>>> >>>>> view > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> represents > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the result of a query. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, it is not > > > > > >>>> >>> intended to support data modification. > > > > > >>>> >>>>>>>>> For > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lakehouse scenarios, > > > > > >>>> >>> where the ability to delete or > > > > > >>>> >>>>> update > > > > > >>>> >>>>>>>>>>> data > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> is > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> crucial (such as > > > > > >>>> >>> compliance with GDPR, FLIP-2), > > > > > >>>> >>>>>>>>> materialized > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> views > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fall short. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Compared to CREATE > > > > > >>>> >>> (regular) TABLE, CREATE DYNAMIC TABLE > > > > > >>>> >>>>>>>>> not > > > > > >>>> >>>>>>>>>>>>> only > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> defines metadata in the > > > > > >>>> >>> catalog but also automatically > > > > > >>>> >>>>>>>>>>> initiates > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> a > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data refresh task based > > > > > >>>> >>> on the query specified during > > > > > >>>> >>>>> table > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> creation. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It dynamically executes > > > > > >>>> >>> data updates. Users can focus on > > > > > >>>> >>>>>>>>> data > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dependencies and data > > > > > >>>> >>> generation logic. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The new dynamic table > > > > > >>>> >>> does not conflict with the existing > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DynamicTableSource and > > > > > >>>> >>> DynamicTableSink interfaces. For > > > > > >>>> >>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> developer, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all that needs to be > > > > > >>>> >>> implemented is the new > > > > > >>>> >>>>>>>>>>> CatalogDynamicTable, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> without changing the > > > > > >>>> >>> implementation of source and sink. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5. For now, the FLIP > > > > > >>>> >>> does not consider supporting Table > > > > > >>>> >>>>> API > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> operations > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> on > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dynamic Table > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> . However, once the SQL > > > > > >>>> >>> syntax is finalized, we can > > > > > >>>> >>>>> discuss > > > > > >>>> >>>>>>>>>>> this > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> in > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> a > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> separate FLIP. > > > > > >>>> >>> Currently, I have a rough idea: the Table > > > > > >>>> >>>>>>>>> API > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> should > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also introduce > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DynamicTable operation > > > > > >>>> >>> interfaces > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> corresponding to the > > > > > >>>> >>> existing Table interfaces. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The TableEnvironment > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will provide relevant > > > > > >>>> >>> methods to support various > > > > > >>>> >>>>> dynamic > > > > > >>>> >>>>>>>>>>>>> table > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operations. The goal > > > > > >>>> >>> for the new Dynamic Table is to > > > > > >>>> >>>>> offer > > > > > >>>> >>>>>>>>>>> users > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> an > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> experience similar to > > > > > >>>> >>> using a database, which is why we > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> prioritize > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SQL-based approaches > > > > > >>>> >>> initially. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How do you envision > > > > > >>>> >>> re-adding the functionality of a > > > > > >>>> >>>>>>>>>>> statement > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> set, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fans out to multiple > > > > > >>>> >>> tables? This is a very important > > > > > >>>> >>>>> use > > > > > >>>> >>>>>>>>>>> case > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> for > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> data > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pipelines. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Multi-tables is indeed > > > > > >>>> >>> a very important user scenario. In > > > > > >>>> >>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> future, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we can consider > > > > > >>>> >>> extending the statement set syntax to > > > > > >>>> >>>>>>>>> support > > > > > >>>> >>>>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creation of multiple > > > > > >>>> >>> dynamic tables. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since the early > > > > > >>>> >>> days of Flink SQL, we were discussing > > > > > >>>> >>>>>>>>>>> `SELECT > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> STREAM > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FROM T EMIT 5 > > > > > >>>> >>> MINUTES`. Your proposal seems to rephrase > > > > > >>>> >>>>>>>>>>> STREAM > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> EMIT, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into other keywords > > > > > >>>> >>> DYNAMIC TABLE and FRESHNESS. But the > > > > > >>>> >>>>>>>>> core > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functionality is > > > > > >>>> >>> still there. I'm wondering if we should > > > > > >>>> >>>>>>>>>>> widen > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scope > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (maybe not part of > > > > > >>>> >>> this FLIP but a new FLIP) to follow > > > > > >>>> >>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> standard > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> closely. Making > > > > > >>>> >>> `SELECT * FROM t` bounded by default and > > > > > >>>> >>>>>>>>> use > > > > > >>>> >>>>>>>>>>>>> new > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for the dynamic > > > > > >>>> >>> behavior. Flink 2.0 would be the perfect > > > > > >>>> >>>>>>>>> time > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> for > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> however, it would > > > > > >>>> >>> require careful discussions. What do > > > > > >>>> >>>>> you > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> think? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The query part indeed > > > > > >>>> >>> requires a separate FLIP > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for discussion, as it > > > > > >>>> >>> involves changes to the default > > > > > >>>> >>>>>>>>>>> behavior. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/dynamic_tables__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j73477_wHn$ > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ron > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jing Zhang < > > > > > >>>> >>> beyond1...@gmail.com> 于2024年3月13日周三 15:19写道： > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Lincoln & Ron, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the > > > > > >>>> >>> proposal. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the > > > > > >>>> >>> question raised by Timo. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Besides, I have some > > > > > >>>> >>> other questions. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. How to define > > > > > >>>> >>> query of dynamic table? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Use flink sql or > > > > > >>>> >>> introducing new syntax? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If use flink sql, > > > > > >>>> >>> how to handle the difference in SQL > > > > > >>>> >>>>>>>>> between > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> streaming > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> batch processing? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, a query > > > > > >>>> >>> including window aggregate based on > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> processing > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or a query including > > > > > >>>> >>> global order by? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Whether modify > > > > > >>>> >>> the query of dynamic table is allowed? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Or we could only > > > > > >>>> >>> refresh a dynamic table based on > > > > > >>>> >>>>> initial > > > > > >>>> >>>>>>>>>>>>> query? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. How to use > > > > > >>>> >>> dynamic table? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The dynamic table > > > > > >>>> >>> seems to be similar with materialized > > > > > >>>> >>>>>>>>> view. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> Will > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> we > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something like > > > > > >>>> >>> materialized view rewriting during the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> optimization? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jing Zhang > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Timo Walther < > > > > > >>>> >>> twal...@apache.org> 于2024年3月13日周三 01:24写 > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 道： > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Lincoln & Ron, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for > > > > > >>>> >>> proposing this FLIP. I think a design > > > > > >>>> >>>>> similar > > > > > >>>> >>>>>>>>> to > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> what > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> you > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> propose has been > > > > > >>>> >>> in the heads of many people, however, > > > > > >>>> >>>>>>>>> I'm > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wondering > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this will fit > > > > > >>>> >>> into the bigger picture. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I haven't deeply > > > > > >>>> >>> reviewed the FLIP yet, but would like > > > > > >>>> >>>>> to > > > > > >>>> >>>>>>>>>>> ask > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> some > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> initial questions: > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Flink has > > > > > >>>> >>> introduced the concept of Dynamic Tables many > > > > > >>>> >>>>>>>>>>> years > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> ago. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does the term > > > > > >>>> >>> "Dynamic Table" fit into Flink's regular > > > > > >>>> >>>>>>>>>>> tables > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how does it > > > > > >>>> >>> relate to Table API? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I fear that > > > > > >>>> >>> adding the DYNAMIC TABLE keyword could > > > > > >>>> >>>>> cause > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> confusion > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users, because a > > > > > >>>> >>> term for regular CREATE TABLE (that > > > > > >>>> >>>>> can > > > > > >>>> >>>>>>>>> be > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> "kind > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> of > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dynamic" as well > > > > > >>>> >>> and is backed by a changelog) is then > > > > > >>>> >>>>>>>>>>>>> missing. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> given that we > > > > > >>>> >>> call our connectors for those tables, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DynamicTableSource > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and > > > > > >>>> >>> DynamicTableSink. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In general, I > > > > > >>>> >>> find it contradicting that a TABLE can be > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> "paused" > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> or > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "resumed". From > > > > > >>>> >>> an English language perspective, this > > > > > >>>> >>>>>>>>> does > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> sound > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> incorrect. In my > > > > > >>>> >>> opinion (without much research yet), a > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> continuous > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating trigger > > > > > >>>> >>> should rather be modelled as a CREATE > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> MATERIALIZED > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> VIEW > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which users are > > > > > >>>> >>> familiar with?) or a new concept such > > > > > >>>> >>>>>>>>> as a > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> CREATE > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TASK > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (that can be > > > > > >>>> >>> paused and resumed?). > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How do you > > > > > >>>> >>> envision re-adding the functionality of a > > > > > >>>> >>>>>>>>>>> statement > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> set, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fans out to > > > > > >>>> >>> multiple tables? This is a very important > > > > > >>>> >>>>> use > > > > > >>>> >>>>>>>>>>> case > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> for > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pipelines. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since the early > > > > > >>>> >>> days of Flink SQL, we were discussing > > > > > >>>> >>>>>>>>>>> `SELECT > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> STREAM > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FROM T EMIT 5 > > > > > >>>> >>> MINUTES`. Your proposal seems to rephrase > > > > > >>>> >>>>>>>>>>> STREAM > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> EMIT, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into other > > > > > >>>> >>> keywords DYNAMIC TABLE and FRESHNESS. But > > > > > >>>> >>>>> the > > > > > >>>> >>>>>>>>>>> core > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> functionality is > > > > > >>>> >>> still there. I'm wondering if we > > > > > >>>> >>>>> should > > > > > >>>> >>>>>>>>>>> widen > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scope > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (maybe not part > > > > > >>>> >>> of this FLIP but a new FLIP) to follow > > > > > >>>> >>>>>>>>> the > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> standard > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> closely. Making > > > > > >>>> >>> `SELECT * FROM t` bounded by default > > > > > >>>> >>>>> and > > > > > >>>> >>>>>>>>> use > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> new > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> syntax > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for the dynamic > > > > > >>>> >>> behavior. Flink 2.0 would be the > > > > > >>>> >>>>> perfect > > > > > >>>> >>>>>>>>>>> time > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> for > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> however, it would > > > > > >>>> >>> require careful discussions. What do > > > > > >>>> >>>>>>>>> you > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> think? > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Timo > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 11.03.24 > > > > > >>>> >>> 08:23, Ron liu wrote: > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Dev > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lincoln Lee > > > > > >>>> >>> and I would like to start a discussion > > > > > >>>> >>>>> about > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> FLIP-435: > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Introduce a > > > > > >>>> >>> New Dynamic Table for Simplifying Data > > > > > >>>> >>>>>>>>>>>>> Pipelines. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This FLIP is > > > > > >>>> >>> designed to simplify the development of > > > > > >>>> >>>>>>>>> data > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> processing > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pipelines. > > > > > >>>> >>> With Dynamic Tables with uniform SQL > > > > > >>>> >>>>>>>>> statements > > > > > >>>> >>>>>>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> freshness, > > > > > >>>> >>> users can define batch and streaming > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> transformations > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data in the > > > > > >>>> >>> same way, accelerate ETL pipeline > > > > > >>>> >>>>>>>>> development, > > > > > >>>> >>>>>>>>>>>>> and > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> manage > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> task > > > > > >>>> >>> scheduling automatically. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For more > > > > > >>>> >>> details, see FLIP-435 [1]. Looking forward to > > > > > >>>> >>>>>>>>> your > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback. > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lincoln & Ron > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>>>> > > > > > >>>> >>>>>>>>>>> > > > > > >>>> >>>>>>>>> > > > > > >>>> >>>>>>> > > > > > >>>> >>>>> > > > > > >>>> >>> > > > > > >>>> >> > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > > > > > > > > > > > >
