Thanks Junrui for starting this discussion! Prior to this, it takes some experience to correctly distinguish whether a configuration item is effective at CLUSTER or JOB level, and this proposal avoids such a burden. And the FLIP looks good to me.
Best, Yanquan Zhu Zhu <reed...@gmail.com> 于2024年10月9日周三 17:14写道： > Thanks Junrui for starting this discussion! > +1 for the proposal > > Clarifying the scopes of config options can be beneficial for users. > Even Flink developers occasionally encounter confusion regarding the > scopes of various configurations, such as attempting to retrieve > job-specific settings from `TaskExecutor#taskManagerConfiguration`. > Clarifying the options' scopes in documentation can be a good reference > and reminder. > > Thanks, > Zhu > > Junrui Lee <jrlee....@gmail.com> 于2024年10月9日周三 11:08写道： > > > Hi devs, > > > > I would like to initiate a discussion about FLIP-478: Introduce Config > > Option Scope > > > > Currently, Flink can be configured in various ways. Taking a session job > as > > an example: > > > > 1. When the session cluster starts, the JobManager (JM) and TaskManager > > (TM) load the configuration files config.yaml from specified directory on > > their respective machines, merging them with command line parameters (-D) > > to form their component configuration. > > > > 2. When submitting a job, the client loads its local configuration file > and > > merges it with configuration within the job code and command line > > parameters to obtain the job configuration. > > > > When this job is submitted to the session cluster, the values of > > cluster-level config options, such as the memory for JM and TM, will be > > defined by the JM/TM configuration. And the values of job-level config > > options, such as parallelism, will be defined by the job configuration. > > Additionally, some config options will fall back to the values defined in > > the configuration from JM/TM if their values are not defined in job > > configuration, such as restart strategies. > > > > However, it can be confusing for users to know where to set a config > > option. For example, we've received several bug reports from users saying > > that taskmanager.numberOfTaskSlots doesn’t take effect, which turned out > to > > be because they mistakenly configured it in the job code. > > > > As a result, we intend to introduce the concept of Config Option Scope to > > help users understand the proper way to configure each config option. And > > the scopes should be displayed in Flink configuration documentation. > > > > For more details, please refer to FLIP-478 [1]. Looking forward to your > > feedback. > > > > Best, > > > > Junrui Lee > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-478+Introduce+Config+Option+Scope > > >
