> >
> > On 07.11.24 16:23, David Radley wrote:
> >> Hi Timo,
> >> Thank you for the well structured and referenced Flip. This looks
> >> really useful.
> >>
> >> Some thoughts:
> >>
> >>
> >>1. I wonder wha
? Is business as
usual with the watermark generator or is there also something else the
PTF can do?
2. When it says “in case of onWatermark the output is rowtime =
TimeContext.followingWatermark() - 1” . It would be good to state in
the Flip the thinking behind this.
3. I agree with
ith the watermark generator or is there also something else the
PTF can do?
2. When it says “in case of onWatermark the output is rowtime =
TimeContext.followingWatermark() - 1” . It would be good to state in
the Flip the thinking behind this.
3. I agree with Fabian the currentWatermark
ark generator or is there also something else the PTF can do?
2. When it says “in case of onWatermark the output is rowtime =
TimeContext.followingWatermark() - 1” . It would be good to state in the Flip
the thinking behind this.
3. I agree with Fabian the currentWatermark and followingWat
, Duration
period); what is relativeTime? relativeTime sounds like a duration.
Kind regards, David.
From: Fabian Hüske
Date: Monday, 4 November 2024 at 15:41
To: dev@flink.apache.org
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-440: User-defined SQL operators /
ProcessTableFunction (PTF)
Hi Timo
Hi Timo,
Thanks for the detailed and very well structured FLIP document!
This is an important feature and will enable many more use-cases for Flink
SQL and Table API.
I have a few questions / suggestions:
1. "Scoping and Simplifications", "Partition and Order Semantics":
"By default, we requir
Hi David, Hi Shengkai,
> can I apply a PTF to a stream that doesn't have a time attribute?
Yes, time attributes are optional. This is why the
REQUIRES_TIME_ATTRIBUTE argument trait exists. If no on_time has been
specified in the SQL call and the REQUIRES_TIME_ATTRIBUTE trait is not
present, t
Hi, Timo.
Thanks for your proposal. This FLIP greatly extends the ease of use of SQL!
But I have some questions about this FLIP:
1. How do users register PTF to Flink? It looks like users can use CREATE
FUNCTION to register the PTF.
2. Can the input parameter of PTF be a view? If the input param
Timo, thanks for the response. I have a few more questions.
as mentioned in "Scoping and Simplifications" a PTF will not support
> late events. It will filter them out. We have to solve the late events
> topic at an earlier stage in the SQL pipeline. This is a different FLIP
> discussion. Not ever
Hi David,
as mentioned in "Scoping and Simplifications" a PTF will not support
late events. It will filter them out. We have to solve the late events
topic at an earlier stage in the SQL pipeline. This is a different FLIP
discussion. Not every SQL operators should deal with late events in a
d
> 3. Change of interfaces for multiple output tables
> Currently, I think using a STATEMENT SET should be enough for side
> output semantics. I have added an example in section 5.2.3.2 for that.
I question whether this really works. Is there a guarantee that
watermarking will be applied upstream o
Hi Xuyang,
thanks for the good questions.
1. What happens if the TTLs for these different StateHints are not the same?
The eval() fully determines available state and their TTL. Helper
methods such as onTimer() and finish() can references a subset of
declared state. It is not necessary that t
Hi, Timo.
Thank you for this great work! When I previously introduced the session window
TVF, I was contemplating
how to enable users to define a PTF in SQL. I'm glad to see this work being
discussed and that it has
improved the integration with the DataStream API.
After reading the entire
Hi everyone,
thanks for all the feedback I received so far. I had very healthy
discussions with various people both online and offline at Current and
Flink Forward Berlin. The general user responses were also very
positive. The FLIP should be ready to start a VOTE thread.
This is the last ca
Hi Jim,
3. Multiple output tables
> Does the target_table need to be specified in the SELECT clause?
No. Similar to reading from a regular table. The filter column must not
be part of SELECT part.
> It seems like the two target_table could have separate schemas defined.
That is true. The SE
Hi Timo,
Thank you for the answers. I have a few clarifications inlined.
On Mon, Oct 14, 2024 at 8:07 AM Timo Walther wrote:
> 3. Change of interfaces for multiple output tables
> Currently, I think using a STATEMENT SET should be enough for side
> output semantics. I have added an example in
Hi everyone,
thank you, Timo, for this proposal. ProccessTableFunction to me is about
taking the best thing about DataStream API and combining it with all the
good things of Table API (type system, optimizer, backwards compatibility).
Big +1 on this direction and proposal.
Cheers,
Konstantin
A
Hey Timo,
I think this will be a very valuable addition to Flink SQL. I think it's
necessary to drive SQL and Table API adoption further since it can help
stateful upgrades. This will close a few gaps between SQL and DataStream
enabling more people to use SQL type system and catalog ecosystem which
Hi Jim,
thanks for taking a deep look into the FLIP. Happy to answer your questions:
1. Pass through semantics and partition columns
We won't support pass through semantics in the first version. But I
don't think that columns are duplicated unless they use a different
name. E.g. the PTF can of
Hi Timo,
After thinking about it more today, I have some additionals questions /
topics:
9. For the existing window TVFs, GROUP BYs over the window start and end
are optimized to return only one result per group. With the proposal as
is, would it be possible to implement a replacement/copy of t
Hi Timo,
Thanks for the FLIP! Also, thanks for saying that this has been in your
head for a few years; there is a ton here.
1. For the pass through semantics, if the partition columns are already
listed in the pass through column, they are not duplicated right?
2. A number of places mention that
Hi everyone,
I'm super excited to start a discussion about FLIP-440: User-defined SQL
operators / ProcessTableFunction (PTF) [1].
This FLIP has been in my head for many years and Flink 2.0 is a good
time to open up Flink SQL to a new category of use cases. Following the
principle "Make simpl
22 matches
Mail list logo
