I think showing the TTL for operators is a nice-to-have feature, not a must one in this FLIP. We can still get the information from the operator descriptions.
And I think we can continue the TTL showing work based on FLINK-33230 [1]. Last but not least, I prefer to throw exceptions if the TTL configuration is mistakenly used as it will affect the data correctness. [1] https://issues.apache.org/jira/browse/FLINK-33230 Best Yun Tang ________________________________ From: ConradJam <jam.gz...@gmail.com> Sent: Wednesday, October 11, 2023 20:30 To: dev@flink.apache.org <dev@flink.apache.org> Subject: Re: Re: [DISCUSS] FLIP-373: Support Configuring Different State TTLs using SQL Hint +1 TTL shows the state ttl for operators in Flink web ui can be know what operator state Zakelly Lan <zakelly....@gmail.com> 于2023年10月11日周三 19:14写道： > Hi Jane, > > The fine-grained TTL management is extremely useful for performance > tuning, so +1 for the idea. I have a minor suggestion: would it be > possible to provide a simple hint that allows the omission of the key? > For example, something like "SELECT /+ STATE_TTL('1h')/", which would > specify the TTL for all states in the 'SELECT' clause. > > And I also share the same concern as Feng. I am wondering if we could > show the state ttl for operators in Flink UI. > > > Best, > Zakelly > > On Wed, Oct 11, 2023 at 1:27 PM Feng Jin <jinfeng1...@gmail.com> wrote: > > > > Hi Jane, > > > > Thank you for providing this explanation. > > > > Another small question, since there is no exception thrown when the STATE > > hint is set incorrectly, > > should we somehow show that the TTL setting has taken effect? > > For instance, exhibiting the TTL option within the operator's > description? > > > > Best, > > Feng > > > > On Tue, Oct 10, 2023 at 7:51 PM Xuyang <xyzhong...@163.com> wrote: > > > > > Hi, Jane. > > > > > > > > > I think this syntax will be easier for users to set operator ttl. So > big > > > +1. I left some minor comments here. > > > > > > > > > I notice that using STATE_TTL hints wrongly will not throw any > exceptions. > > > But it seems that in the current join hint scenario, > > > if user uses an unknown table name as the chosen side, a validation > > > exception will be thrown. > > > Maybe we should distinguish which exceptions need to be thrown > explicitly. > > > > > > > > > > > > > > > -- > > > > > > Best！ > > > Xuyang > > > > > > > > > > > > > > > > > > At 2023-10-10 18:23:55, "Jane Chan" <qingyue....@gmail.com> wrote: > > > >Hi Feng, > > > > > > > >Thank you for your valuable comments. The reason for not including the > > > >scenarios above is as follows: > > > > > > > >For <1>, the automatically inferred stateful operators are not easily > > > >expressible in SQL. This issue was discussed in FLIP-292, and besides > > > >ChangelogNormalize, SinkUpsertMateralizer also faces the same problem. > > > > > > > >For <2> and <3>, the challenge lies in internal implementation. > During the > > > >default_rewrite phase, the row_number expression in LogicalProject is > > > >transformed into LogicalWindow by Calcite's > > > >CoreRules#PROJECT_TO_LOGICAL_PROJECT_AND_WINDOW. However, > CalcRelSplitter > > > >does not pass the hints as an input argument when creating > LogicalWindow, > > > >resulting in the loss of the hint at this step. To support this, we > may > > > >need to rewrite some optimization rules in Calcite, which could be a > > > >follow-up work if required. > > > > > > > >Best, > > > >Jane > > > > > > > >On Tue, Oct 10, 2023 at 1:40 AM Feng Jin <jinfeng1...@gmail.com> > wrote: > > > > > > > >> Hi Jane, > > > >> > > > >> Thank you for proposing this FLIP. > > > >> > > > >> I believe that this FLIP will greatly enhance the flexibility of > setting > > > >> state, and by setting different operators' TTL, it can also > increase job > > > >> stability, especially in regular join scenarios. > > > >> The parameter design is very concise, big +1 for this, and it is > also > > > >> relatively easy to use for users. > > > >> > > > >> > > > >> I have a small question: in the FLIP, it only mentions join and > group. > > > >> Should we also consider other scenarios? > > > >> > > > >> 1. the auto generated deduplicate operator[1]. > > > >> 2. the deduplicate query[2]. > > > >> 3. the topN query[3]. > > > >> > > > >> [1] > > > >> > > > >> > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-exec-source-cdc-events-duplicate > > > >> [2] > > > >> > > > >> > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/deduplication/ > > > >> [3] > > > >> > > > >> > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/topn/ > > > >> > > > >> > > > >> Best, > > > >> Feng > > > >> > > > >> On Sun, Oct 8, 2023 at 5:53 PM Jane Chan <qingyue....@gmail.com> > wrote: > > > >> > > > >> > Hi devs, > > > >> > > > > >> > I'd like to initiate a discussion on FLIP-373: Support Configuring > > > >> > Different State TTLs using SQL Hint [1]. This proposal is on top > of > > > the > > > >> > FLIP-292 [2] to address typical scenarios with unambiguous > semantics > > > and > > > >> > hint propagation. > > > >> > > > > >> > I'm looking forward to your opinions! > > > >> > > > > >> > > > > >> > [1] > > > >> > > > > >> > > > > >> > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-373%3A+Support+Configuring+Different+State+TTLs+using+SQL+Hint > > > >> > [2] > > > >> > > > > >> > > > > >> > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration > > > >> > > > > >> > Best, > > > >> > Jane > > > >> > > > > >> > > > > -- Best ConradJam
