Hi Shammon, Thanks a lot for your input!
I thought SQL Driver could act as a general-purpose default main class for Flink SQL. It could be used in Flink CLI submission, web submission, or SQL Client/Gateway submission. For SQL Client/Gateway submission, we use it implicitly if needed, and for the other approaches, explicitly. WRT to configurations, dynamic configurations are supported for most cases ATM. - For CLI submission, the dynamic configuration is supported via CLI, i.e. `-D` prefixed configurations. - For SQL Client/Gateway submission, we could also use `-D` dynamic configuration and SQL init file to customize the default session configuration. - For web submission, there’s no dynamic configuration as I’m aware of. Do you think it’s sufficient already or is it still necessary to implement it at the SQL driver level? Best, Paul Lam > 2023年5月31日 10:40，Shammon FY <zjur...@gmail.com> 写道： > > Thanks Paul for driving this proposal. > > I found the sql driver has no config related options. If I understand > correctly, the sql driver can be used to submit sql jobs in a 'job > submission service' such as sql-gateway. In general, in addition to the > default config for Flink cluster which includes k8s, ha and .etc, users may > also specify configurations for SQL jobs, including parallelism, number of > Task Managers, etc. > > For example, in sql-gateway users can `set` dynamic parameters before > submitting a sql statement, and for sql files users may put their > configurations in a `yaml` file. Should sql driver support dynamic > parameters and specify config file? > > Best, > Shammon FY > > On Tue, May 30, 2023 at 9:57 PM Weihua Hu <huweihua....@gmail.com> wrote: > >> Thanks Paul for the proposal. >> >> +1 for this. It is valuable in improving ease of use. >> >> I have a few questions. >> - Is SQLRunner the better name? We use this to run a SQL Job. (Not strong, >> the SQLDriver is fine for me) >> - Could we run SQL jobs using SQL in strings? Otherwise, we need to prepare >> a SQL file in an image for Kubernetes application mode, which may be a bit >> cumbersome. >> - I noticed that we don't specify the SQLDriver jar in the >> "run-application" >> command. Does that mean we need to perform automatic detection in Flink? >> >> >> Best, >> Weihua >> >> >> On Mon, May 29, 2023 at 7:24 PM Paul Lam <paullin3...@gmail.com> wrote: >> >>> Hi team, >>> >>> I’d like to start a discussion about FLIP-316 [1], which introduces a SQL >>> driver as the >>> default main class for Flink SQL jobs. >>> >>> Currently, Flink SQL could be executed out of the box either via SQL >>> Client/Gateway >>> or embedded in a Flink Java/Python program. >>> >>> However, each one has its drawback: >>> >>> - SQL Client/Gateway doesn’t support the application deployment mode [2] >>> - Flink Java/Python program requires extra work to write a non-SQL >> program >>> >>> Therefore, I propose adding a SQL driver to act as the default main class >>> for SQL jobs. >>> Please see the FLIP docs for details and feel free to comment. Thanks! >>> >>> [1] >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver >>> < >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver >>>> >>> [2] https://issues.apache.org/jira/browse/FLINK-26541 < >>> https://issues.apache.org/jira/browse/FLINK-26541> >>> >>> Best, >>> Paul Lam >>
