Hi devs, I would like to initiate a discussion about FLIP-478: Introduce Config Option Scope
Currently, Flink can be configured in various ways. Taking a session job as an example: 1. When the session cluster starts, the JobManager (JM) and TaskManager (TM) load the configuration files config.yaml from specified directory on their respective machines, merging them with command line parameters (-D) to form their component configuration. 2. When submitting a job, the client loads its local configuration file and merges it with configuration within the job code and command line parameters to obtain the job configuration. When this job is submitted to the session cluster, the values of cluster-level config options, such as the memory for JM and TM, will be defined by the JM/TM configuration. And the values of job-level config options, such as parallelism, will be defined by the job configuration. Additionally, some config options will fall back to the values defined in the configuration from JM/TM if their values are not defined in job configuration, such as restart strategies. However, it can be confusing for users to know where to set a config option. For example, we've received several bug reports from users saying that taskmanager.numberOfTaskSlots doesn’t take effect, which turned out to be because they mistakenly configured it in the job code. As a result, we intend to introduce the concept of Config Option Scope to help users understand the proper way to configure each config option. And the scopes should be displayed in Flink configuration documentation. For more details, please refer to FLIP-478 [1]. Looking forward to your feedback. Best, Junrui Lee [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-478+Introduce+Config+Option+Scope
