+1 TTL shows the state ttl for operators in Flink web ui can be know what
operator state
Zakelly Lan <zakelly....@gmail.com> 于2023年10月11日周三 19:14写道：
> Hi Jane,
>
> The fine-grained TTL management is extremely useful for performance
> tuning, so +1 for the idea. I have a minor suggestion: would it be
> possible to provide a simple hint that allows the omission of the key?
> For example, something like "SELECT /+ STATE_TTL('1h')/", which would
> specify the TTL for all states in the 'SELECT' clause.
>
> And I also share the same concern as Feng. I am wondering if we could
> show the state ttl for operators in Flink UI.
>
>
> Best,
> Zakelly
>
> On Wed, Oct 11, 2023 at 1:27 PM Feng Jin <jinfeng1...@gmail.com> wrote:
> >
> > Hi Jane,
> >
> > Thank you for providing this explanation.
> >
> > Another small question, since there is no exception thrown when the STATE
> > hint is set incorrectly,
> > should we somehow show that the TTL setting has taken effect?
> > For instance, exhibiting the TTL option within the operator's
> description?
> >
> > Best,
> > Feng
> >
> > On Tue, Oct 10, 2023 at 7:51 PM Xuyang <xyzhong...@163.com> wrote:
> >
> > > Hi, Jane.
> > >
> > >
> > > I think this syntax will be easier for users to set operator ttl. So
> big
> > > +1. I left some minor comments here.
> > >
> > >
> > > I notice that using STATE_TTL hints wrongly will not throw any
> exceptions.
> > > But it seems that in the current join hint scenario,
> > > if user uses an unknown table name as the chosen side, a validation
> > > exception will be thrown.
> > > Maybe we should distinguish which exceptions need to be thrown
> explicitly.
> > >
> > >
> > >
> > >
> > > --
> > >
> > > Best！
> > > Xuyang
> > >
> > >
> > >
> > >
> > >
> > > At 2023-10-10 18:23:55, "Jane Chan" <qingyue....@gmail.com> wrote:
> > > >Hi Feng,
> > > >
> > > >Thank you for your valuable comments. The reason for not including the
> > > >scenarios above is as follows:
> > > >
> > > >For <1>, the automatically inferred stateful operators are not easily
> > > >expressible in SQL. This issue was discussed in FLIP-292, and besides
> > > >ChangelogNormalize, SinkUpsertMateralizer also faces the same problem.
> > > >
> > > >For <2> and <3>, the challenge lies in internal implementation.
> During the
> > > >default_rewrite phase, the row_number expression in LogicalProject is
> > > >transformed into LogicalWindow by Calcite's
> > > >CoreRules#PROJECT_TO_LOGICAL_PROJECT_AND_WINDOW. However,
> CalcRelSplitter
> > > >does not pass the hints as an input argument when creating
> LogicalWindow,
> > > >resulting in the loss of the hint at this step. To support this, we
> may
> > > >need to rewrite some optimization rules in Calcite, which could be a
> > > >follow-up work if required.
> > > >
> > > >Best,
> > > >Jane
> > > >
> > > >On Tue, Oct 10, 2023 at 1:40 AM Feng Jin <jinfeng1...@gmail.com>
> wrote:
> > > >
> > > >> Hi Jane,
> > > >>
> > > >> Thank you for proposing this FLIP.
> > > >>
> > > >> I believe that this FLIP will greatly enhance the flexibility of
> setting
> > > >> state, and by setting different operators' TTL, it can also
> increase job
> > > >> stability, especially in regular join scenarios.
> > > >> The parameter design is very concise, big +1 for this, and it is
> also
> > > >> relatively easy to use for users.
> > > >>
> > > >>
> > > >> I have a small question: in the FLIP, it only mentions join and
> group.
> > > >> Should we also consider other scenarios?
> > > >>
> > > >> 1. the auto generated deduplicate operator[1].
> > > >> 2. the deduplicate query[2].
> > > >> 3. the topN query[3].
> > > >>
> > > >> [1]
> > > >>
> > > >>
> > >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-exec-source-cdc-events-duplicate
> > > >> [2]
> > > >>
> > > >>
> > >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/deduplication/
> > > >> [3]
> > > >>
> > > >>
> > >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/topn/
> > > >>
> > > >>
> > > >> Best,
> > > >> Feng
> > > >>
> > > >> On Sun, Oct 8, 2023 at 5:53 PM Jane Chan <qingyue....@gmail.com>
> wrote:
> > > >>
> > > >> > Hi devs,
> > > >> >
> > > >> > I'd like to initiate a discussion on FLIP-373: Support Configuring
> > > >> > Different State TTLs using SQL Hint [1]. This proposal is on top
> of
> > > the
> > > >> > FLIP-292 [2] to address typical scenarios with unambiguous
> semantics
> > > and
> > > >> > hint propagation.
> > > >> >
> > > >> > I'm looking forward to your opinions!
> > > >> >
> > > >> >
> > > >> > [1]
> > > >> >
> > > >> >
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-373%3A+Support+Configuring+Different+State+TTLs+using+SQL+Hint
> > > >> > [2]
> > > >> >
> > > >> >
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration
> > > >> >
> > > >> > Best,
> > > >> > Jane
> > > >> >
> > > >>
> > >
>
--
Best
ConradJam
