I noticed that Yuxia had replied that "sink.paralleilsm" could help in some cases.
I think a better way is to integrate it with streamGraph or extend CompiledPlan just as FLIP-292 setting state TTL per operator [1] does. [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240883951 Best Yun Tang ________________________________ From: GREEN <1286649...@qq.com.INVALID> Sent: Tuesday, April 18, 2023 17:21 To: dev <dev@flink.apache.org> Subject: Re: [DISCUSS FLINKSQL PARALLELISM] During the process of generating streamgraph，I can modify the edge partitioner by configuring parameters. Just need to know in advance the structure of the streamgraph,This can be obtained by printing log. ---Original--- From: "liu ron"<ron9....@gmail.com> Date: Tue, Apr 18, 2023 09:37 AM To: "dev"<dev@flink.apache.org>; Subject: Re: [DISCUSS FLINKSQL PARALLELISM] Hi, Green Thanks for driving this discussion, in batch mode we have the Adaptive Batch Scheduler which automatically derives operator parallelism based on data volume at runtime, so we don't need to care about the parallelism. However, in stream mode, currently, Flink SQL can only set the parallelism of an operator globally, and many users would like to set the parallelism of an operator individually, which seems to be a pain point at the moment, and it would make sense to support set parallelism at operator granularity. Do you have any idea about the solution for this problem? Best, Ron GREEN <1286649...@qq.com.invalid> 于2023年4月14日周五 16:03写道： > Problem： > > > Currently, FlinkSQL can set a unified parallelism in the job,it > cannot set parallelism for each operator. > This can cause resource waste On the occasion of high > parallelism and small data volume.there may also be too many small > file for writing HDFS Scene. > > > Solution： > I can modify FlinkSQL to support operator parallelism.Is it meaningful to > do this？Let's discuss.
