Correction: “users can set 'scan.startup.mode' for kafka connector” -> “users can set 'scan.startup.mode' for kafka connector by dynamic table option”
Shuo Cheng <njucs...@gmail.com>于2023年3月23日 周四21:50写道： > Hi Jane, > Thanks for driving this, operator level state ttl is absolutely a desired > feature. I would share my opinion as following: > > If the scope of this proposal is limited as an enhancement for compiled > json plan, it makes sense. I think it does not conflict with configuring > state ttl > in other ways, e.g., SQL HINT or something else, because they just work in > different level, SQL Hint works in the exact entrance of SQL API, while > compiled json plan is the intermediate results for SQL. > I think the final shape of state ttl configuring may like the that, users > can define operator state ttl using SQL HINT (assumption...), but it may > affects more than one stateful operators inside the same query block, then > users can further configure a specific one by modifying the compiled json > plan... > > In a word, this proposal is in good shape as an enhancement for compiled > json plan, and it's orthogonal with other ways like SQL Hint which works in > a higher level. > > > Nips: > > > "From the SQL semantic perspective, hints cannot intervene in the > calculation of data results." > I think it's more properly to say that hint does not affect the > equivalence of execution plans (hash agg vs sort agg), not the equivalence > of execution results, e.g., users can set 'scan.startup.mode' for kafka > connector, which also "intervene in the calculation of data results". > > Sincerely, > Shuo > > On Tue, Mar 21, 2023 at 7:52 PM Jane Chan <qingyue....@gmail.com> wrote: > >> Hi devs, >> >> I'd like to start a discussion on FLIP-292: Support configuring state TTL >> at operator level for Table API & SQL programs [1]. >> >> Currently, we only support job-level state TTL configuration via >> 'table.exec.state.ttl'. However, users may expect a fine-grained state TTL >> control to optimize state usage. Hence we propose to serialize/deserialize >> the state TTL as metadata of the operator's state to/from the compiled >> JSON >> plan, to achieve the goal that specifying different state TTL when >> transforming the exec node to stateful operators. >> >> Look forward to your opinions! >> >> [1] >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240883951 >> >> Best Regards, >> Jane Chan >> >
