Thanks Junrui for driving this proposal! It's valuable for users to distinguish all flink options. I have 2 minor questions about this proposal:
1. Does this FLIP enrich the scope for all old flink options? 2. If the option is configured at the wrong level, could we output some warn log to remind users? For example, an option of cluster level is added at job configuration. Best, Rui On Thu, Oct 10, 2024 at 9:34 AM Yanquan Lv <decq12y...@gmail.com> wrote: > Thanks Junrui for starting this discussion! > Prior to this, it takes some experience to correctly distinguish whether a > configuration item is effective at CLUSTER or JOB level, and this proposal > avoids such a burden. > And the FLIP looks good to me. > > > Best, > Yanquan > > Zhu Zhu <reed...@gmail.com> 于2024年10月9日周三 17:14写道： > > > Thanks Junrui for starting this discussion! > > +1 for the proposal > > > > Clarifying the scopes of config options can be beneficial for users. > > Even Flink developers occasionally encounter confusion regarding the > > scopes of various configurations, such as attempting to retrieve > > job-specific settings from `TaskExecutor#taskManagerConfiguration`. > > Clarifying the options' scopes in documentation can be a good reference > > and reminder. > > > > Thanks, > > Zhu > > > > Junrui Lee <jrlee....@gmail.com> 于2024年10月9日周三 11:08写道： > > > > > Hi devs, > > > > > > I would like to initiate a discussion about FLIP-478: Introduce Config > > > Option Scope > > > > > > Currently, Flink can be configured in various ways. Taking a session > job > > as > > > an example: > > > > > > 1. When the session cluster starts, the JobManager (JM) and TaskManager > > > (TM) load the configuration files config.yaml from specified directory > on > > > their respective machines, merging them with command line parameters > (-D) > > > to form their component configuration. > > > > > > 2. When submitting a job, the client loads its local configuration file > > and > > > merges it with configuration within the job code and command line > > > parameters to obtain the job configuration. > > > > > > When this job is submitted to the session cluster, the values of > > > cluster-level config options, such as the memory for JM and TM, will be > > > defined by the JM/TM configuration. And the values of job-level config > > > options, such as parallelism, will be defined by the job configuration. > > > Additionally, some config options will fall back to the values defined > in > > > the configuration from JM/TM if their values are not defined in job > > > configuration, such as restart strategies. > > > > > > However, it can be confusing for users to know where to set a config > > > option. For example, we've received several bug reports from users > saying > > > that taskmanager.numberOfTaskSlots doesn’t take effect, which turned > out > > to > > > be because they mistakenly configured it in the job code. > > > > > > As a result, we intend to introduce the concept of Config Option Scope > to > > > help users understand the proper way to configure each config option. > And > > > the scopes should be displayed in Flink configuration documentation. > > > > > > For more details, please refer to FLIP-478 [1]. Looking forward to your > > > feedback. > > > > > > Best, > > > > > > Junrui Lee > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-478+Introduce+Config+Option+Scope > > > > > >
