venkata91 commented on PR #24736: URL: https://github.com/apache/flink/pull/24736#issuecomment-2418177120
> Sorry for the delayed response due to the National Day holiday in China. > > Currently, there are three key fields related to parallelism: > > 1. defaultMaxParallelism: This comes from the config option values auto-parallelism.max-parallelism or default.parallelism. > 2. globalDefaultSourceParallelism: This comes from config option value default-source-parallelism or field defaultMaxParallelism. > 3. JobVertex max parallelism: This is either user-configured by calling setMaxParallelism method or fallback to defaultMaxParallelism. > > > I think the issue is, if source parallelism is not explicitly set and if `execution.batch.adaptive.auto-parallelism.max-parallelism` is set < `execution.batch.adaptive.auto-parallelism.default-source-parallelism` then the source vertex max parallelism is set to the value of `execution.batch.adaptive.auto-parallelism.max-parallelism`. > > The main issue here is that I think if a user has set the max parallelism of the source (called setMaxParallelism method), we must respect this max parallelism. However, the current implementation does not account for this and only considers globalDefaultSourceParallelism. I see. Basically you're saying if user sets max parallelism through `jobVertex`, then that shouldn't be overridden. I think I figured out how to handle this case and updated the changes. I am in the process of adding the unit/IT tests. Will update that shortly. Please take a look whenever you get a chance on the approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
