Gustavo de Morais created FLINK-40896:
-----------------------------------------
Summary: Add built-in DEDUPLICATE_KEEP_FIRST PTF
Key: FLINK-40896
URL: https://issues.apache.org/jira/browse/FLINK-40896
Project: Flink
Issue Type: New Feature
Components: Table SQL / API, Table SQL / Runtime
Reporter: Gustavo de Morais
Assignee: Vas Shabu
Deduplication is a common Flink SQL need. Today users must express "keep first"
dedup via verbose {{ROW_NUMBER()}} windowing, which must match an exact pattern
to be planner-recognized and buries the intent in a windowed rank.
This ticket adds {{{}DEDUPLICATE_KEEP_FIRST{}}}, a built-in PTF with two modes:
* {*}Watermarkless{*}: first record per key wins, no watermark required.
* {*}Event-time{*}: with {{{}on_time{}}}, the smallest-event-time row per key
wins, finalized once the watermark passes it.
{code:java}
DEDUPLICATE_KEEP_FIRST(
input => TABLE(t) [PARTITION BY <cols>],
on_time => DESCRIPTOR(<rowtime>),
state_ttl => <interval>,
reset_ttl_on_duplicate => <boolean>,
uid => <string>
) {code}
Event-time mode requires insert-only input (updating input + {{on_time}}
rejected at planning time). Watermarkless mode accepts updating input,
absorbing later changes into the kept row.
See
[FLIP-614|https://cwiki.apache.org/confluence/spaces/FLINK/pages/451975182/FLIP-614+Add+built-in+DEDUPLICATE_KEEP_FIRST+PTF]
for full motivation, rejected alternatives, and compatibility plan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
