Hi Jane, Thank you for providing this explanation.
Another small question, since there is no exception thrown when the STATE hint is set incorrectly, should we somehow show that the TTL setting has taken effect? For instance, exhibiting the TTL option within the operator's description? Best, Feng On Tue, Oct 10, 2023 at 7:51 PM Xuyang <xyzhong...@163.com> wrote: > Hi, Jane. > > > I think this syntax will be easier for users to set operator ttl. So big > +1. I left some minor comments here. > > > I notice that using STATE_TTL hints wrongly will not throw any exceptions. > But it seems that in the current join hint scenario, > if user uses an unknown table name as the chosen side, a validation > exception will be thrown. > Maybe we should distinguish which exceptions need to be thrown explicitly. > > > > > -- > > Best！ > Xuyang > > > > > > At 2023-10-10 18:23:55, "Jane Chan" <qingyue....@gmail.com> wrote: > >Hi Feng, > > > >Thank you for your valuable comments. The reason for not including the > >scenarios above is as follows: > > > >For <1>, the automatically inferred stateful operators are not easily > >expressible in SQL. This issue was discussed in FLIP-292, and besides > >ChangelogNormalize, SinkUpsertMateralizer also faces the same problem. > > > >For <2> and <3>, the challenge lies in internal implementation. During the > >default_rewrite phase, the row_number expression in LogicalProject is > >transformed into LogicalWindow by Calcite's > >CoreRules#PROJECT_TO_LOGICAL_PROJECT_AND_WINDOW. However, CalcRelSplitter > >does not pass the hints as an input argument when creating LogicalWindow, > >resulting in the loss of the hint at this step. To support this, we may > >need to rewrite some optimization rules in Calcite, which could be a > >follow-up work if required. > > > >Best, > >Jane > > > >On Tue, Oct 10, 2023 at 1:40 AM Feng Jin <jinfeng1...@gmail.com> wrote: > > > >> Hi Jane, > >> > >> Thank you for proposing this FLIP. > >> > >> I believe that this FLIP will greatly enhance the flexibility of setting > >> state, and by setting different operators' TTL, it can also increase job > >> stability, especially in regular join scenarios. > >> The parameter design is very concise, big +1 for this, and it is also > >> relatively easy to use for users. > >> > >> > >> I have a small question: in the FLIP, it only mentions join and group. > >> Should we also consider other scenarios? > >> > >> 1. the auto generated deduplicate operator[1]. > >> 2. the deduplicate query[2]. > >> 3. the topN query[3]. > >> > >> [1] > >> > >> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-exec-source-cdc-events-duplicate > >> [2] > >> > >> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/deduplication/ > >> [3] > >> > >> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/topn/ > >> > >> > >> Best, > >> Feng > >> > >> On Sun, Oct 8, 2023 at 5:53 PM Jane Chan <qingyue....@gmail.com> wrote: > >> > >> > Hi devs, > >> > > >> > I'd like to initiate a discussion on FLIP-373: Support Configuring > >> > Different State TTLs using SQL Hint [1]. This proposal is on top of > the > >> > FLIP-292 [2] to address typical scenarios with unambiguous semantics > and > >> > hint propagation. > >> > > >> > I'm looking forward to your opinions! > >> > > >> > > >> > [1] > >> > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-373%3A+Support+Configuring+Different+State+TTLs+using+SQL+Hint > >> > [2] > >> > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration > >> > > >> > Best, > >> > Jane > >> > > >> >
