Hi Yu, The PR looks good to me; the only thing I noticed is that the RemoveJarOperation may need to be added.
I asked some other questions; I don't think any of them are particularly large issues. Let me know when the next PR goes up and I'll take a look. Cheers, Jim On Sun, Nov 13, 2022 at 10:45 PM yu zelin <yuzelin....@gmail.com> wrote: > Hi Jim, > > It would be nice if you can take a look on > https://github.com/apache/flink/pull/21133 and give me some feedback. > > Best, > Yu Zelin > > > 2022年11月12日 00:44，Jim Hughes <jhug...@confluent.io.INVALID> 写道： > > > > Hi Shengkai, > > > > I think there is an additional case where a proxy is between the client > and > > gateway. In that case, being able to pass headers would allow for > > additional options / features. > > > > I see several PRs from Yu Zelin. Is there a first one to review? > > > > Cheers, > > > > Jim > > > > On Thu, Nov 10, 2022 at 9:42 PM Shengkai Fang <fskm...@gmail.com> wrote: > > > >> Hi, Jim. > >> > >>> how to pass additional headers when sending REST requests > >> > >> Could you share what headers do you want to send when using SQL > Client? I > >> think there are two cases we need to consider. Please correct me if I am > >> wrong. > >> > >> # Case 1 > >> > >> If users wants to connect to the SQL Gateway with its password, I think > the > >> users should type > >> ``` > >> ./sql-client.sh --user xxx --password xxx > >> ``` > >> in the terminal and the OpenSessionRequest should be enough. > >> > >> # Case 2 > >> > >> If users wants to modify the execution config, users should type > >> ``` > >> Flink SQL> SET `<config_name>` = `<config_value>`; > >> ``` > >> in the terminal. The Client can send ExecuteStatementRequest to the > >> Gateway. > >> > >>> As you have FLIPs or PRs, feel free to let me, Jamie, and Alexey know. > >> > >> It would be nice you can join us to finish the feature. I think the > >> modification about the SQL Gateway side is ready to review. > >> > >> Best, > >> Shengkai > >> > >> > >> Jim Hughes <jhug...@confluent.io.invalid> 于2022年11月11日周五 05:19写道： > >> > >>> Hi Yu Zelin, > >>> > >>> I have read through your draft and it looks good. I am new to Flink, > so > >> I > >>> haven't learned about everything which needs to be done yet. > >>> > >>> One of the considerations that I'm interested in understanding is how > to > >>> pass additional headers when sending REST requests. From looking at > the > >>> code, it looks like a custom `OutboundChannelHandlerFactory` could be > >>> created to read additional configuration and set headers. Does that > make > >>> sense? > >>> > >>> Thank you very much for sharing the proof of concept code and the > >>> document. As you have FLIPs or PRs, feel free to let me, Jamie, and > >> Alexey > >>> know. We'll be happy to review them. > >>> > >>> Cheers, > >>> > >>> Jim > >>> > >>> On Wed, Nov 9, 2022 at 11:43 PM yu zelin <yuzelin....@gmail.com> > wrote: > >>> > >>>> Hi, all > >>>> Sorry for late response. As Shengkai mentioned, Currently I’m working > >>> with > >>>> him on SQL Client, dedicating to implement the Remote Mode of SQL > >>> Client. I > >>>> have written a draft of implementation plan and will write a FLIP > about > >>> it > >>>> ASAP. If you are interested in, please take a look at the draft and > >> it’s > >>>> nice if you give me some feedback. > >>>> The doc is at: > >>>> > >>> > >> > https://docs.google.com/document/d/14cS4VBSamMUnlM_PZuK6QKLfriUuQU51iqET5oiYy_c/edit?usp=sharing > >>>> > >>>>> 2022年11月7日 11:19，Shengkai Fang <fskm...@gmail.com> 写道： > >>>>> > >>>>> Hi, all. Sorry for the late reply. > >>>>> > >>>>>> Is the gateway mode planned to be supported for SQL Client in 1.17? > >>>>>> Do you have anything you can already share so we can start with > >> your > >>>> work or just play around with it. > >>>>> > >>>>> Yes. @yzl is working on it and he will list the implementation plan > >>>> later and share the progress. I think the change is not very large and > >> I > >>>> think it's not a big problem to finish this in the release-1.17. I > will > >>>> join to develop this in the mid of November. > >>>>> > >>>>> Best, > >>>>> Shengkai > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> Jamie Grier <jgr...@apache.org <mailto:jgr...@apache.org>> > >>>> 于2022年11月5日周六 00:48写道： > >>>>>> Hi Shengkai, > >>>>>> > >>>>>> We're doing more and more Flink development at Confluent these days > >>> and > >>>> we're currently trying to bootstrap a prototype that relies on the SQL > >>>> Client and Gateway. We will be using the the components in some of > our > >>>> projects and would like to co-develop them with you and the rest of > the > >>>> Flink community. > >>>>>> > >>>>>> As of right now it's a pretty big blocker for our upcoming milestone > >>>> that the SQL Client has not yet been modified to talk to the SQL > >> Gateway > >>>> and we want to help with this effort ASAP! We would be even willing > to > >>>> take over the work if it's not yet started but I suspect it already > is. > >>>>>> > >>>>>> Anyway, rather than start working immediately on the SQL Client and > >>>> adding a the new Gateway mode ourselves we wanted to start a > >> conversation > >>>> with you and see where you're at with things and offer to help. > >>>>>> > >>>>>> Do you have anything you can already share so we can start with your > >>>> work or just play around with it. Like I said, we just want to get > >>> started > >>>> and are very able to help in this area. We see both the SQL Client > and > >>>> Gateway being very important for us and have a good team to help > >> develop > >>> it. > >>>>>> > >>>>>> Let me know if there is a branch you can share, etc. It would be > >> much > >>>> appreciated! > >>>>>> > >>>>>> -Jamie Grier > >>>>>> > >>>>>> > >>>>>> On 2022/10/28 06:06:49 Shengkai Fang wrote: > >>>>>>> Hi. > >>>>>>> > >>>>>>>> Is there a possibility for us to get engaged and at least > >>> introduce > >>>>>>> initial changes to support authentication/authorization? > >>>>>>> > >>>>>>> Yes. You can write a FLIP about the design and change. We can > >>> discuss > >>>> this > >>>>>>> in the dev mail. If the FLIP passes, we can develop it together. > >>>>>>> > >>>>>>>> Another question about persistent Gateway: did you have any > >>> specific > >>>>>>> thoughts about it or some draft design? > >>>>>>> > >>>>>>> We don't have any detailed plan about this. But I know Livy has a > >>>> similar > >>>>>>> feature. > >>>>>>> > >>>>>>> Best, > >>>>>>> Shengkai > >>>>>>> > >>>>>>> > >>>>>>> Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: > >>>> vendrov...@gmail.com>> 于2022年10月27日周四 15:12写道： > >>>>>>> > >>>>>>>> Apologies from the delayed response on my side. > >>>>>>>> > >>>>>>>> I think the authentication module is not part of our plan in > >> 1.17 > >>>> because > >>>>>>>>> of the busy work. I think we'll start the design at the end of > >>> the > >>>>>>>>> release-1.17. > >>>>>>>> > >>>>>>>> > >>>>>>>> Is there a possibility for us to get engaged and at least > >>> introduce > >>>>>>>> initial changes to support authentication/authorization? > >>>> Specifically, > >>>>>>>> changes in the API and in SQL Client. > >>>>>>>> > >>>>>>>> We expect the following authentication flow: > >>>>>>>> > >>>>>>>> On the SQL gateway we want to be able to use a delegation token. > >>>>>>>> SQL client should be able to supply an API key. > >>>>>>>> The SQL Gateway *would not *be submitting jobs on behalf of the > >>>> client. > >>>>>>>> > >>>>>>>> Ideally it would be nice to introduce some interfaces in the SQL > >>>> Gateway > >>>>>>>> that would allow implementing custom authentication and > >>>> authorization. > >>>>>>>> > >>>>>>>> Another question about persistent Gateway: did you have any > >>> specific > >>>>>>>> thoughts about it or some draft design? > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Alexey > >>>>>>>> > >>>>>>>> > >>>>>>>> On Fri, Oct 21, 2022 at 1:13 AM Shengkai Fang < > >> fskm...@gmail.com > >>>> <mailto:fskm...@gmail.com>> wrote: > >>>>>>>> > >>>>>>>>> Sorry for the late response. > >>>>>>>>> > >>>>>>>>> In the next version(Flink 1.17), we plan to support the SQL > >>> Client > >>>> to > >>>>>>>>> submit the statement to the Flink SQL Gateway. The FLINK-29486 > >>>>>>>>> <https://issues.apache.org/jira/browse/FLINK-29486> is the > >> first > >>>> step to > >>>>>>>>> remove the usage of the `Parser` in the client side, which > >> needs > >>>> to read > >>>>>>>>> the table schema during the converting sql node to operation. I > >>>> think the authentication > >>>>>>>>> module is not part of our plan in 1.17 because of the busy > >> work. > >>> I > >>>> think > >>>>>>>>> we'll start the design at the end of the release-1.17. > >>>>>>>>> But could you share more details about the requirements of the > >>>>>>>>> authentication? > >>>>>>>>> - Do you use the kerberos or delegation token or password to do > >>> the > >>>>>>>>> authentication? > >>>>>>>>> - After the authentication, do you need the sql gateway to > >> submit > >>>> the > >>>>>>>>> job on behalf of the client? > >>>>>>>>> - ... > >>>>>>>>> > >>>>>>>>> For detailed implementation, I think Hive and Presto are good > >>>> examples to > >>>>>>>>> dig in. If you have some thoughts about the authentication > >>> module, > >>>>>>>>> please let me know. > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> Shengkai > >>>>>>>>> > >>>>>>>>> Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: > >>>> vendrov...@gmail.com>> 于2022年10月19日周三 00:37写道： > >>>>>>>>> > >>>>>>>>>> Thank you for the response, Yuxia! > >>>>>>>>>> > >>>>>>>>>> Shengkai, I would like to learn more about nearest and a bit > >>> more > >>>>>>>>>> distant plans about development of the SQL Gateway and the SQL > >>>> Client. > >>>>>>>>>> Do you have a description of the work planned or maybe can > >> share > >>>> general > >>>>>>>>>> thoughts about the Authentication module, or Persistent > >> Gateway. > >>>>>>>>>> How can the authentication part be addressed on the SQL Client > >>>> side? > >>>>>>>>>> > >>>>>>>>>> Regards, > >>>>>>>>>> -Alexey > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Wed, Oct 12, 2022 at 11:24 PM yuxia < > >>>> luoyu...@alumni.sjtu.edu.cn <mailto:luoyu...@alumni.sjtu.edu.cn>> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>>> In what Flink’s release the connection from SQL Client to > >> the > >>>> Gateway > >>>>>>>>>>> is > >>>>>>>>>>> expected to be added? > >>>>>>>>>>> Flink 1.17 > >>>>>>>>>>> > >>>>>>>>>>>> “Authentication module” (2) and “Persistent Gateway” (4) as > >>>>>>>>>>> possible future work. Were there any recent discussions on > >>> these > >>>>>>>>>>> subjects? > >>>>>>>>>>> No recent discussions on these subjects, but I think it'll > >> come > >>>> in > >>>>>>>>>>> Flink 1.17 > >>>>>>>>>>> > >>>>>>>>>>>> Another related topic: are there ideas around making SQL > >>>> Gateway a > >>>>>>>>>>> multi-tenant > >>>>>>>>>>> component? > >>>>>>>>>>> Yes. > >>>>>>>>>>> > >>>>>>>>>>> Shengkaiis the maintainer of SQL Client and SQL gateway, > >> maybe > >>>> he can > >>>>>>>>>>> provide more information. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Best regards, > >>>>>>>>>>> Yuxia > >>>>>>>>>>> > >>>>>>>>>>> ----- 原始邮件 ----- > >>>>>>>>>>> 发件人: "Alexey Leonov-Vendrovskiy" <vendrov...@gmail.com > >>> <mailto: > >>>> vendrov...@gmail.com>> > >>>>>>>>>>> 收件人: "dev" <dev@flink.apache.org <mailto: > >> dev@flink.apache.org > >>>>> > >>>>>>>>>>> 发送时间: 星期四, 2022年 10 月 13日 下午 12:33:08 > >>>>>>>>>>> 主题: SQL Gateway and SQL Client > >>>>>>>>>>> > >>>>>>>>>>> Hi all, > >>>>>>>>>>> > >>>>>>>>>>> I’m Alexey from Confluent. This is my first email in this > >>>> discussion > >>>>>>>>>>> list. > >>>>>>>>>>> I’m rather new to Flink, and to local customs of > >> communication. > >>>> I want > >>>>>>>>>>> to > >>>>>>>>>>> dive deeper and hopefully get more involved over time. > >>>>>>>>>>> > >>>>>>>>>>> Currently I have a few questions around SQL Gateway and SQL > >>>> Client. > >>>>>>>>>>> Specifically I wanted to learn what is the vision around the > >>>> nearest > >>>>>>>>>>> future > >>>>>>>>>>> of these two components. > >>>>>>>>>>> > >>>>>>>>>>> In what Flink’s release the connection from SQL Client to the > >>>> Gateway is > >>>>>>>>>>> expected to be added? I was looking at > >>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-29486, and > >>> recently > >>>> it got > >>>>>>>>>>> renamed from “Enable SQL Client to Connect SQL Gateway in > >>> Remote > >>>> Mode” > >>>>>>>>>>> to > >>>>>>>>>>> “Introduce Client Parser to get statement type”. I did some > >>>> search, but > >>>>>>>>>>> didn’t find a good place where the client's work in this > >>>> direction is > >>>>>>>>>>> discussed or tracked. > >>>>>>>>>>> > >>>>>>>>>>> A couple questions about the SQL Gateway. The FLIP-91 > >>>>>>>>>>> < > >>>>>>>>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Gateway#FLIP91:SupportSQLGateway-Futurework > >>>>>>>>>>>> > >>>>>>>>>>> mentions “Authentication module” (2) and “Persistent Gateway” > >>>> (4) as > >>>>>>>>>>> possible future work. Were there any recent discussions on > >>> these > >>>>>>>>>>> subjects? > >>>>>>>>>>> Or maybe there are some ideas how to move these directions > >>>> forward? > >>>>>>>>>>> Another > >>>>>>>>>>> related topic: are there ideas around making SQL Gateway a > >>>> multi-tenant > >>>>>>>>>>> component? > >>>>>>>>>>> > >>>>>>>>>>> Thank you, > >>>>>>>>>>> > >>>>>>>>>>> Alexey > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>> > >>>> > >>> > >> > >
