Hi Eric, this sounds reasonable, there are definitely cases where you need to limit sink parallelism for example not to overload the storage or limit the number of output files
+1 Best, D. On Sun, Apr 23, 2023 at 1:09 PM Weihua Hu <huweihua....@gmail.com> wrote: > Hi, Eric > > Thanks for bringing this discussion. > I think it's reasonable to add ''setMaxParallelism" for DataStreamSink. > > +1 > > Best, > Weihua > > > On Sat, Apr 22, 2023 at 3:20 AM eric xiao <xiao.eric...@gmail.com> wrote: > > > Hi there devs, > > > > I would like to start a discussion thread for FLINK-31873[1]. > > > > We are in the processing of enabling Flink reactive mode as the default > > scheduling mode. While reading configuration docs [2] (I believe it was > > also mentioned during one of the training sessions during Flink Forward > > 2022), one can/should replace all setParallelism calls with > > setMaxParallelism when migrating to reactive mode. > > > > This currently isn't possible on a sink in a Flink pipeline as we do not > > expose a setMaxParallelism on the DataStreamSink class [3]. The > underlying > > Transformation class does have both a setMaxParallelism and > setParallelism > > function defined [4], but only setParallelism is offered in the > > DataStreamSink class. > > > > I believe adding setMaxParallelism would be beneficial for not just flink > > reactive mode, both modes of running of a flink pipeline (non reactive > > mode, flink auto scaling). > > > > Best, > > > > Eric Xiao > > > > [1] https://issues.apache.org/jira/browse/FLINK-31873 > > [2] > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/elastic_scaling/#configuration > > [3] > > > > > https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStreamSink.java > > [4] > > > > > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/dag/Transformation.java#L248-L285 > > >
