Hi all, I'd like to propose FLIP-622: Add built-in DEDUPLICATE and DEDUPLICATE_WINDOW PTFs [1] for discussion. It supersedes FLIP-614 [2]. While working on DEDUPLICATE_KEEP_LAST, we found a cleaner design: one function per output mode, with first or last as an argument. DEDUPLICATE_KEEP_FIRST is unreleased, so we want to raise this before anything ships.
Today deduplication is written as a ROW_NUMBER() over-window. That pattern is planned as deduplication only when ordered by a time attribute and becomes a general Top-N when ordered by a version column. This FLIP adds two built-in PTFs, each with keep => 'first' | 'last'. DEDUPLICATE keeps one row per key. Its output is updating, and it accepts insert-only and updating input: SELECT * FROM DEDUPLICATE( input => TABLE(clicks) PARTITION BY event_id, keep => 'first' ); DEDUPLICATE_WINDOW emits one final row per key and event-time window, and its output is insert-only: SELECT * FROM DEDUPLICATE_WINDOW( input => TABLE(clicks) PARTITION BY event_id, keep => 'first', duration => INTERVAL '5' MINUTES, on_time => DESCRIPTOR(ts) ); With this proposal, we aim to improve user experience by introducing a user-friendly feature to perform one of the most common tasks in Flink. Looking forward to your feedback and thoughts. Kind regards, Vas Shabu [1] https://cwiki.apache.org/confluence/spaces/FLINK/pages/462816464/FLIP-622+Add+built-in+DEDUPLICATE+and+DEDUPLICATE_WINDOW+PTFs [2] https://cwiki.apache.org/confluence/spaces/FLINK/pages/451975182/FLIP-614+Add+built-in+DEDUPLICATE_KEEP_FIRST+PTF
