Hi, devs After discussing offline with Lincoln Lee and Shengkai Fang, we realized that some SQL config options, like table.sql-dialect, may also server DDL in which case no Flink job will run. Including these options under the JOB scope is not accurate.
So we propose introducing a new SQL_EXECUTION scope, alongside the proposed CLUSTER, JOB, JOB_AND_CLUSTER, SQL_CLIENT, SQL_GATEWAY, and HISTORY_SERVER scopes. The SQL_EXECUTION scope includes config options related to Flink SQL/Table API that impact SQL or Table API execution, such as SQL translation, execution plan generation, and operator behavior. Users can set these options in a SQL/Table Program, or within the Sql-gateway or Sql-client. Config values specified in the code take precedence; if not present, values from the compiling service are used. And since SQL-related config options are already on a separate page with explanations on how to configure them, this FLIP will not detail the scope of SQL config options in the common configuration page[1]. So it will add little burden for users to understand this new scope. If there are no further comments, we will open the voting thread on Thursday, October 17, 2024. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/ Best, Junrui Rui Fan <1996fan...@gmail.com> 于2024年10月10日周四 11:33写道： > Thanks Junrui for the quick clarification! > > Sounds good to me, +1 for this FLIP. > > Best, > Rui > > On Thu, Oct 10, 2024 at 10:52 AM Junrui Lee <jrlee....@gmail.com> wrote: > > > Hi Rui, > > > > Thank you for your response. Here are my replies to your questions: > > > > Regarding question 1: We plan to label the config options on the Flink > > configuration page in the upcoming 2.0 version. In the long term, our > goal > > is to ensure that all config options visible to users are properly > labeled. > > > > Regarding question 2: That’s a good question, and we have indeed > considered > > similar approaches for pre-validating configuration items. However, due > to > > the current limitations of Flink, the ConfigOption acts as a format used > > when retrieving config values from the Configuration object. > Consequently, > > the relationship between ConfigOptions and the key-value pairs stored in > > the Configuration object is currently one-way; there is no way to reverse > > lookup a ConfigOption from a string key. > > > > This implies that we might need to construct a complete collection of > > ConfigOptions in advance, which is not an easy or straightforward task. > > Therefore, we prefer to focus this FLIP on introducing the concept of > > config option scope, while any potential functionality extensions can be > > addressed in future FLIPs. > > > > WDYT? > > > > Best, > > > > Junrui > > > > Rui Fan <1996fan...@gmail.com> 于2024年10月10日周四 10:18写道： > > > > > Thanks Junrui for driving this proposal! > > > > > > It's valuable for users to distinguish all flink options. > > > I have 2 minor questions about this proposal: > > > > > > 1. Does this FLIP enrich the scope for all old flink options? > > > 2. If the option is configured at the wrong level, could we > > > output some warn log to remind users? For example, > > > an option of cluster level is added at job configuration. > > > > > > Best, > > > Rui > > > > > > On Thu, Oct 10, 2024 at 9:34 AM Yanquan Lv <decq12y...@gmail.com> > wrote: > > > > > > > Thanks Junrui for starting this discussion! > > > > Prior to this, it takes some experience to correctly distinguish > > whether > > > a > > > > configuration item is effective at CLUSTER or JOB level, and this > > > proposal > > > > avoids such a burden. > > > > And the FLIP looks good to me. > > > > > > > > > > > > Best, > > > > Yanquan > > > > > > > > Zhu Zhu <reed...@gmail.com> 于2024年10月9日周三 17:14写道： > > > > > > > > > Thanks Junrui for starting this discussion! > > > > > +1 for the proposal > > > > > > > > > > Clarifying the scopes of config options can be beneficial for > users. > > > > > Even Flink developers occasionally encounter confusion regarding > the > > > > > scopes of various configurations, such as attempting to retrieve > > > > > job-specific settings from `TaskExecutor#taskManagerConfiguration`. > > > > > Clarifying the options' scopes in documentation can be a good > > reference > > > > > and reminder. > > > > > > > > > > Thanks, > > > > > Zhu > > > > > > > > > > Junrui Lee <jrlee....@gmail.com> 于2024年10月9日周三 11:08写道： > > > > > > > > > > > Hi devs, > > > > > > > > > > > > I would like to initiate a discussion about FLIP-478: Introduce > > > Config > > > > > > Option Scope > > > > > > > > > > > > Currently, Flink can be configured in various ways. Taking a > > session > > > > job > > > > > as > > > > > > an example: > > > > > > > > > > > > 1. When the session cluster starts, the JobManager (JM) and > > > TaskManager > > > > > > (TM) load the configuration files config.yaml from specified > > > directory > > > > on > > > > > > their respective machines, merging them with command line > > parameters > > > > (-D) > > > > > > to form their component configuration. > > > > > > > > > > > > 2. When submitting a job, the client loads its local > configuration > > > file > > > > > and > > > > > > merges it with configuration within the job code and command line > > > > > > parameters to obtain the job configuration. > > > > > > > > > > > > When this job is submitted to the session cluster, the values of > > > > > > cluster-level config options, such as the memory for JM and TM, > > will > > > be > > > > > > defined by the JM/TM configuration. And the values of job-level > > > config > > > > > > options, such as parallelism, will be defined by the job > > > configuration. > > > > > > Additionally, some config options will fall back to the values > > > defined > > > > in > > > > > > the configuration from JM/TM if their values are not defined in > job > > > > > > configuration, such as restart strategies. > > > > > > > > > > > > However, it can be confusing for users to know where to set a > > config > > > > > > option. For example, we've received several bug reports from > users > > > > saying > > > > > > that taskmanager.numberOfTaskSlots doesn’t take effect, which > > turned > > > > out > > > > > to > > > > > > be because they mistakenly configured it in the job code. > > > > > > > > > > > > As a result, we intend to introduce the concept of Config Option > > > Scope > > > > to > > > > > > help users understand the proper way to configure each config > > option. > > > > And > > > > > > the scopes should be displayed in Flink configuration > > documentation. > > > > > > > > > > > > For more details, please refer to FLIP-478 [1]. Looking forward > to > > > your > > > > > > feedback. > > > > > > > > > > > > Best, > > > > > > > > > > > > Junrui Lee > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-478+Introduce+Config+Option+Scope > > > > > > > > > > > > > > > > > > > > >
