Hi Jane,
Thanks for driving this, and your solution is absolutely creative. In my
company, there also exist some scenarios in which users want to config state
TTL at operator level, especially for window operators which regard TTL as
allow lateness.
To support this scenarios, we implemented a query hint like below:
SELECT /*+ STATE_TTL('1D') */
id,
max(num) as num
FROM source
GROUP BY id
For reasons to reject SQL Hints you mentioned in the FLIP, I have some
different opinions.
1. Hints may not only aim to inspire the planner. For example, we have dynamic
options for tables and users can config parallelism for source/sink operators.
State TTL is also kind of parameter which can be configured dynamically.
2. I agree that users need not to understand how SQL statements being
translated to operators. And exactly for this reason, json plan is too complex
for users to read. For example, if we enable
'table.optimizer.distinct-agg.split.enabled’ and
'table.optimizer.incremental-agg-enabled’, we get two Agg operators, I don’t
think all users know which operators are related to their queries and set TTL
correctly.
Therefore, I personally prefer to support this by query hints and users can
config TTL for their ‘group by’ query instead of several operators.
Best regards,
Yisha
> 2023年3月21日 19:51，Jane Chan <qingyue....@gmail.com> 写道：
>
> Hi devs,
>
> I'd like to start a discussion on FLIP-292: Support configuring state TTL
> at operator level for Table API & SQL programs [1].
>
> Currently, we only support job-level state TTL configuration via
> 'table.exec.state.ttl'. However, users may expect a fine-grained state TTL
> control to optimize state usage. Hence we propose to serialize/deserialize
> the state TTL as metadata of the operator's state to/from the compiled JSON
> plan, to achieve the goal that specifying different state TTL when
> transforming the exec node to stateful operators.
>
> Look forward to your opinions!
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240883951
>
> Best Regards,
> Jane Chan
