Thanks Jamie for your thoughtful inputs!
Regarding the necessity of supporting updates, I'd like to add a scenario
where the base table for the 'view' created by the user does not entirely
come from a database(under the relational model).
For example, in a specific scenario: personalized user rec
>
> In the SQL standard regarding the definition of View, there are the
> following restrictions:
1. Partitioned view is not supported.
I agree that partitioned "views" don't really make sense since there is no
storage to partition. However, a bit of a search reveals that partitioned
"material
FLIP-282 [1] has also introduced Update & Delete API for modifying table.
1.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
Best,
Ron
Ron liu 于2024年4月12日周五 19:49写道：
> Hi, jgrier
>
> Thanks for your insightful input.
>
> First of all, very much agree with you that it
Hi, jgrier
Thanks for your insightful input.
First of all, very much agree with you that it is a right direction that we
should strive towards making Flink SQL more user-friendly, including
simplifying the job execution parameters, execution modes, data processing
pipeline definitions and mainten
Sorry for coming very late to this thread. I have not contributed much to
Flink publicly for quite some time but I have been involved with Flink,
daily, for years now and I'm keenly interested in where we take Flink SQL
going forward.
Thanks for the proposal!! I think it's definitely a step in t
Hi all,
I thought I'd cast my vote as well to give extra data:
1. Materialized Table
2. Materialized View (generally speaking I am not too concerned with
using a View here, but since there are concerns around updating a view I
put it second)
I think what is suggested in this FLIP is
I have been following up on the discussion, it's a great FLIP to further
unify stream and batch ETL pipelines. Thanks for the proposal!
Here is my ranking:
1. Materialized Table -> "The table materializes the results of a query
that you specify", this can reflect what we want and doesn't conflic
Thanks for the proposal. I like the FLIP.
My ranking:
1. Refresh(ing) / Live Table -> easy to understand and implies the dynamic
characteristic
2. Derived Table -> easy to understand.
3. Materialized Table -> sounds like just a table with physical data stored
somewhere.
4. Materialized View ->
Thanks Ron and Timo for your proposal!
Here is my ranking:
1. Derived table -> extend the persistent semantics of derived table in SQL
standard, with a strong association with query, and has industry
precedents
such as Google Looker.
2. Live Table -> an alternative for 'dynamic table'
3.
Hi, Dev
My rankings are:
1. Derived Table
2. Materialized Table
3. Live Table
4. Materialized View
Best,
Ron
Ron liu 于2024年4月9日周二 20:07写道：
> Hi, Dev
>
> After several rounds of discussion, there is currently no consensus on the
> name of the new concept. Timo has proposed that we decide the
Hi, Dev
After several rounds of discussion, there is currently no consensus on the
name of the new concept. Timo has proposed that we decide the name through
a vote. This is a good solution when there is no clear preference, so we
will adopt this approach.
Regarding the name of the new concept, t
;>>>> system happen in 2.0, should we rename it
>> >>> in this Flip ( as the
>> >>>>>>>>>>>>>>>>>>>>> suggestions
>> >>>>>>>>>>>>>>>>>>>&g
t;>>>> Best Regards
> >>>>>>>>>>>>>>>>>>>>> Ahmed Hamdy
> >>>>>>>>>>>>>>>>>>>>>
&g
[2]
https://urldefense.com/v3/__https://docs.snowflake.com/en/user-guide/dynamic-tables-about__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j73zexZBXu$
[3]
https://urldefense.com/v3/__https://docs.snowflake.com/en/user-guide/dynamic-tables-re
Hi, Dev
This is a summary letter. After several rounds of discussion, there is a
strong consensus about the FLIP proposal and the issues it aims to address.
The current point of disagreement is the naming of the new concept. I have
summarized the candidates as follows:
1. Derived Table (Inspired
gt;> > > >>>> concept
> > > > > > > > > > > > > >> > > >>>>>>
> > > > > > > > > > > > > >> > > >>>>>> This explanation makes sense to me.
> But the docs a
Hello everybody!
Thanks for the FLIP as it looks amazing (and I think the prove is this deep
discussion it is provoking :))
I have a couple of comments to add to this:
Even though I get the reason why you rejected MATERIALIZED VIEW, I still like
it a lot, and I would like to provide pointers on
t;>>>
> >> > > >>>>>> This is an argument that I understand. But we as Flink could
> >> allow
> >> > > >>>> data
> >> > > >>>>>> modifications. This way we ar
> > >>>> to
>> > > >>>>>> support the creation of multiple dynamic tables.
>> > > >>>>>>
>> > > >>>>>> It's good that we called the concept STATEMENT SET. This
>> allows us
s/misc/materialized-views__;!!IKRxdwAv5BmarQ!dVYcp9PUyjpBGzkYFxb2sdnmB0E22koc-YLdxY2LidExEHUJKRkyvRbAveqjlYFKWevFvmE1Z-j739xS1kvD$
> > > >>>>>>
> > > >>>>>> On 21.03.24 04:14, Feng Jin wrote:
> > > >>>>>>> Hi Ron and
t;>>>>
> > >>>>>>> Although currently we restrict users from modifying the query, I
> > >>>> wonder
> > >>>>>> if
> > >>>>>>> we can provide a better way to help users rebuild it without
>
tions SQL and job information
> >>>> are
> >>>>>>> stored in the Catalog. Does this mean that if a system needs to
> >>>> adapt
> >>>>> to
> >>>>>>> Dynamic Tables, it also needs to store Flink's jo
catalog or somewhere else.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-134%3A+Batch+execution+for+the+DataStream+API
[2] https://docs.snowflake.com/en/user-guide/dynamic-tables-about
[3]
https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh
Best
Yun Tang
er the
lineage
in
the catalog or somewhere else.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-134%3A+Batch+execution+for+the+DataStream+API
[2] https://docs.snowflake.com/en/user-guide/dynamic-tables-about
[3]
https://docs.snowflake.com/en/user-guide/dynamic-tables-ref
>> table relative to the base table’s lag. If users need to consider
>> the
>> > > >> end-to-end freshness of multiple
>> > > >> cascaded dynamic tables, he can manually split them for now. Of
>> > > >> course, how to let mu
e related
> > > >> syntax. However, based on the current design, combined with the
> > > >> catalog and lineage, theoretically,
> > > >> users can also finish the cascading refresh.
> > > >>
> > > >>
> > > >>
ic
> > >>> being discussed in the Flink community!
> > >>>
> > >>> From my point of view, instead of the work of unifying streaming and
> > >> batch
> > >>> in DataStream API [1], this FLIP actually could make users benef
FLIP as an open-source implementation of Snowflake's
> >>> dynamic tables [2], we still lack an incremental refresh mode to make
> the
> >>> ETL near real-time with a much cheaper computation cost. However, I
> think
> >>> this could be done unde
FLIP-134%3A+Batch+execution+for+the+DataStream+API
[2] https://docs.snowflake.com/en/user-guide/dynamic-tables-about
[3] https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh
Best
Yun Tang
________________
From: Lincoln Lee
Sent: Thursday, March 14, 2024 14:35
To: dev@fl
rt the automagical refreshes, we should consider the lineage
> in
> > the catalog or somewhere else.
> >
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-134%3A+Batch+execution+for+the+DataStream+API
> > [2] https://docs.snowflake.com/en/user-guide/
-refresh
>
> Best
> Yun Tang
>
>
> ____________
> From: Lincoln Lee
> Sent: Thursday, March 14, 2024 14:35
> To: dev@flink.apache.org
> Subject: Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for
> Simplifying Data Pipelines
>
>
March 14, 2024 14:35
To: dev@flink.apache.org
Subject: Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying
Data Pipelines
Hi Jing,
Thanks for your attention to this flip! I'll try to answer the following
questions.
> 1. How to define query of dynamic table?
> Use
Hi Jing,
Thanks for your attention to this flip! I'll try to answer the following
questions.
> 1. How to define query of dynamic table?
> Use flink sql or introducing new syntax?
> If use flink sql, how to handle the difference in SQL between streaming
and
> batch processing?
> For example, a que
Hi, Timo
Sorry for later response, thanks for your feedback.
Regarding your questions:
> Flink has introduced the concept of Dynamic Tables many years ago. How
does the term "Dynamic Table" fit into Flink's regular tables and also
how does it relate to Table API?
> I fear that adding the DYN
Hi, Lincoln & Ron,
Thanks for the proposal.
I agree with the question raised by Timo.
Besides, I have some other questions.
1. How to define query of dynamic table?
Use flink sql or introducing new syntax?
If use flink sql, how to handle the difference in SQL between streaming and
batch processi
Hi Lincoln & Ron,
thanks for proposing this FLIP. I think a design similar to what you
propose has been in the heads of many people, however, I'm wondering how
this will fit into the bigger picture.
I haven't deeply reviewed the FLIP yet, but would like to ask some
initial questions:
Flink
Hi, Dev
Lincoln Lee and I would like to start a discussion about FLIP-435:
Introduce a New Dynamic Table for Simplifying Data Pipelines.
This FLIP is designed to simplify the development of data processing
pipelines. With Dynamic Tables with uniform SQL statements and
freshness, users can defi
37 matches
Mail list logo
