Thanks Timo for the FLIP! This is a great improvement to the FLINK sql
syntax around tables. I have two clarification questions:
1. For SEARCH_KEY
```
SELECT *
FROM
t_other,
LATERAL SEARCH_KEY(
input => t,
on_key => DESCRIPTOR(k),
lookup => t_other.name,
options => MAP[
'async', 'true',
'retry-predicate', 'lookup_miss',
'retry-strategy', 'fixed_delay',
'fixed-delay'='10s'
]
)
```
Table `t` needs to be an existing `LookupTableSource` [1], right? And we
will rewrite it to `StreamPhysicalLookupJoin` [2] or similar operator
during the physical optimization phase.
Also to support passing options, we need to extend `LookupContext` [3] to
have a `getOptions` or `getRuntimeOptions` method?
2. For FROM_CHANGELOG
```
SELECT * FROM FROM_CHANGELOG(s) AS t;
```
Do we need to introduce a `DataStream` resource in sql first?
Hao
[1]
https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java
[2]
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalLookupJoin.scala#L41
[3]
https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L82
On Fri, Mar 21, 2025 at 6:25 AM Timo Walther <twal...@apache.org> wrote:
> Hi everyone,
>
> I would like to start a discussion about FLIP-517: Better Handling of
> Dynamic Table Primitives with PTFs [1].
>
> In the past months, I have spent a significant amount of time with SQL
> semantics and the SQL standard around PTFs, when designing and
> implementing FLIP-440 [2]. For those of you that have not taken a look
> into the standard, the concept of Polymorphic Table Functions (PTF)
> enables syntax for implementing custom SQL operators. In my opinion,
> they are kind of a revolution in the SQL language. PTFs can take scalar
> values, tables, models (in Flink), and column lists as arguments. With
> these primitives, we can further evolve shortcomings in the Flink SQL
> language by leveraging syntax and semantics.
>
> I would like introduce a couple of built-in PTFs with the goal to make
> the handling of dynamic tables easier for users. Once users understand
> how a PTF works, they can easily select from a list of functions to
> approach a table for snapshots, changelogs, or searching.
>
> The FLIP proposes:
>
> SNAPSHOT()
> SEARCH_KEY()
> TO_CHANGELOG()
> FROM_CHANGELOG()
>
> I'm aware that this is a delicate topic, and might lead to controversial
> discussions. I hope with concise naming and syntax the benefit over the
> existing syntax becomes clear.
>
> There are more useful PTFs to come, but those are the ones that I
> currently see as the most fundamental ones to tell a round story around
> Flink SQL.
>
> Looking forward to your feedback.
>
> Thanks,
> Timo
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-517%3A+Better+Handling+of+Dynamic+Table+Primitives+with+PTFs
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=298781093
>
