Hi, Jane.
I think this syntax will be easier for users to set operator ttl. So big +1. I
left some minor comments here.
I notice that using STATE_TTL hints wrongly will not throw any exceptions. But
it seems that in the current join hint scenario,
if user uses an unknown table name as the chosen side, a validation exception
will be thrown.
Maybe we should distinguish which exceptions need to be thrown explicitly.
--
Best！
Xuyang
At 2023-10-10 18:23:55, "Jane Chan" <qingyue....@gmail.com> wrote:
>Hi Feng,
>
>Thank you for your valuable comments. The reason for not including the
>scenarios above is as follows:
>
>For <1>, the automatically inferred stateful operators are not easily
>expressible in SQL. This issue was discussed in FLIP-292, and besides
>ChangelogNormalize, SinkUpsertMateralizer also faces the same problem.
>
>For <2> and <3>, the challenge lies in internal implementation. During the
>default_rewrite phase, the row_number expression in LogicalProject is
>transformed into LogicalWindow by Calcite's
>CoreRules#PROJECT_TO_LOGICAL_PROJECT_AND_WINDOW. However, CalcRelSplitter
>does not pass the hints as an input argument when creating LogicalWindow,
>resulting in the loss of the hint at this step. To support this, we may
>need to rewrite some optimization rules in Calcite, which could be a
>follow-up work if required.
>
>Best,
>Jane
>
>On Tue, Oct 10, 2023 at 1:40 AM Feng Jin <jinfeng1...@gmail.com> wrote:
>
>> Hi Jane,
>>
>> Thank you for proposing this FLIP.
>>
>> I believe that this FLIP will greatly enhance the flexibility of setting
>> state, and by setting different operators' TTL, it can also increase job
>> stability, especially in regular join scenarios.
>> The parameter design is very concise, big +1 for this, and it is also
>> relatively easy to use for users.
>>
>>
>> I have a small question: in the FLIP, it only mentions join and group.
>> Should we also consider other scenarios?
>>
>> 1. the auto generated deduplicate operator[1].
>> 2. the deduplicate query[2].
>> 3. the topN query[3].
>>
>> [1]
>>
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-exec-source-cdc-events-duplicate
>> [2]
>>
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/deduplication/
>> [3]
>>
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/topn/
>>
>>
>> Best,
>> Feng
>>
>> On Sun, Oct 8, 2023 at 5:53 PM Jane Chan <qingyue....@gmail.com> wrote:
>>
>> > Hi devs,
>> >
>> > I'd like to initiate a discussion on FLIP-373: Support Configuring
>> > Different State TTLs using SQL Hint [1]. This proposal is on top of the
>> > FLIP-292 [2] to address typical scenarios with unambiguous semantics and
>> > hint propagation.
>> >
>> > I'm looking forward to your opinions!
>> >
>> >
>> > [1]
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-373%3A+Support+Configuring+Different+State+TTLs+using+SQL+Hint
>> > [2]
>> >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-292%3A+Enhance+COMPILED+PLAN+to+support+operator-level+state+TTL+configuration
>> >
>> > Best,
>> > Jane
>> >
>>
