Hi Feng, Thank you for your valuable comments. The reason for not including the scenarios above is as follows:
For <1>, the automatically inferred stateful operators are not easily expressible in SQL. This issue was discussed in FLIP-292, and besides ChangelogNormalize, SinkUpsertMateralizer also faces the same problem. For <2> and <3>, the challenge lies in internal implementation. During the default_rewrite phase, the row_number expression in LogicalProject is transformed into LogicalWindow by Calcite's CoreRules#PROJECT_TO_LOGICAL_PROJECT_AND_WINDOW. However, CalcRelSplitter does not pass the hints as an input argument when creating LogicalWindow, resulting in the loss of the hint at this step. To support this, we may need to rewrite some optimization rules in Calcite, which could be a follow-up work if required. Best, Jane On Tue, Oct 10, 2023 at 1:40 AM Feng Jin <jinfeng1...@gmail.com> wrote: > Hi Jane, > > Thank you for proposing this FLIP. > > I believe that this FLIP will greatly enhance the flexibility of setting > state, and by setting different operators' TTL, it can also increase job > stability, especially in regular join scenarios. > The parameter design is very concise, big +1 for this, and it is also > relatively easy to use for users. > > > I have a small question: in the FLIP, it only mentions join and group. > Should we also consider other scenarios? > > 1. the auto generated deduplicate operator[1]. > 2. the deduplicate query[2]. > 3. the topN query[3]. > > [1] > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-exec-source-cdc-events-duplicate > [2] > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/deduplication/ > [3] > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/topn/ > > > Best, > Feng > > On Sun, Oct 8, 2023 at 5:53 PM Jane Chan <qingyue....@gmail.com> wrote: > > > Hi devs, > > > > I'd like to initiate a discussion on FLIP-373: Support Configuring > > Different State TTLs using SQL Hint [1]. This proposal is on top of the > > FLIP-292 [2] to address typical scenarios with unambiguous semantics and > > hint propagation. > > > > I'm looking forward to your opinions! > > > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-373%3A+Support+Configuring+Different+State+TTLs+using+SQL+Hint > > [2] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration > > > > Best, > > Jane > > >
