Thanks Shengkai for bringing up this discussion. I think it covers a lot of useful features which will dramatically improve the usability of our SQL Client. I have two questions regarding the FLIP.
1. Do you think we can let users set arbitrary configurations via the SET command? A connector may have its own configurations and we don't have a way to dynamically change such configurations in SQL Client. For example, users may want to be able to change hive conf when using hive connector [1]. 2. Any reason why we have to forbid queries in SQL files specified with the -f option? Hive supports a similar -f option but allows queries in the file. And a common use case is to run some query and redirect the results to a file. So I think maybe flink users would like to do the same, especially in batch scenarios. [1] https://issues.apache.org/jira/browse/FLINK-20590 On Fri, Jan 29, 2021 at 10:46 AM Sebastian Liu <liuyang0...@gmail.com> wrote: > Hi Shengkai, > > Glad to see this improvement. And I have some additional suggestions: > > #1. Unify the TableEnvironment in ExecutionContext to > StreamTableEnvironment for both streaming and batch sql. > #2. Improve the way of results retrieval: sql client collect the results > locally all at once using accumulators at present, > which may have memory issues in JM or Local for the big query result. > Accumulator is only suitable for testing purpose. > We may change to use SelectTableSink, which is based > on CollectSinkOperatorCoordinator. > #3. Do we need to consider Flink SQL gateway which is in FLIP-91. Seems > that this FLIP has not moved forward for a long time. > Provide a long running service out of the box to facilitate the sql > submission is necessary. > > What do you think of these? > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway > > > Shengkai Fang <fskm...@gmail.com> 于2021年1月28日周四 下午8:54写道： > > > Hi devs, > > > > Jark and I want to start a discussion about FLIP-163:SQL Client > > Improvements. > > > > Many users have complained about the problems of the sql client. For > > example, users can not register the table proposed by FLIP-95. > > > > The main changes in this FLIP: > > > > - use -i parameter to specify the sql file to initialize the table > > environment and deprecated YAML file; > > - add -f to submit sql file and deprecated '-u' parameter; > > - add more interactive commands, e.g ADD JAR; > > - support statement set syntax; > > > > > > For more detailed changes, please refer to FLIP-163[1]. > > > > Look forward to your feedback. > > > > > > Best, > > Shengkai > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-163%3A+SQL+Client+Improvements > > > > > -- > > *With kind regards > ------------------------------------------------------------ > Sebastian Liu 刘洋 > Institute of Computing Technology, Chinese Academy of Science > Mobile\WeChat: +86—15201613655 > E-mail: liuyang0...@gmail.com <liuyang0...@gmail.com> > QQ: 3239559* > -- Best regards! Rui Li
