Hi Yu, Shengkai, As a quick update, I've had a chance to try out Yu's POC and it is working for me. (Admittedly, I haven't tried it too extensively; I only tried basic operations.)
>From my experiments, I did leave a few comments on https://github.com/apache/flink/pull/20958. Overall, the PRs I see look pretty good. Are they going to be merged soon? Anything else I can do to help? Cheers, Jim On Mon, Nov 14, 2022 at 12:52 PM Jim Hughes <jhug...@confluent.io> wrote: > Hi Yu, > > The PR looks good to me; the only thing I noticed is that the > RemoveJarOperation may need to be added. > > I asked some other questions; I don't think any of them are particularly > large issues. Let me know when the next PR goes up and I'll take a look. > > Cheers, > > Jim > > On Sun, Nov 13, 2022 at 10:45 PM yu zelin <yuzelin....@gmail.com> wrote: > >> Hi Jim, >> >> It would be nice if you can take a look on >> https://github.com/apache/flink/pull/21133 and give me some feedback. >> >> Best, >> Yu Zelin >> >> > 2022年11月12日 00:44，Jim Hughes <jhug...@confluent.io.INVALID> 写道： >> > >> > Hi Shengkai, >> > >> > I think there is an additional case where a proxy is between the client >> and >> > gateway. In that case, being able to pass headers would allow for >> > additional options / features. >> > >> > I see several PRs from Yu Zelin. Is there a first one to review? >> > >> > Cheers, >> > >> > Jim >> > >> > On Thu, Nov 10, 2022 at 9:42 PM Shengkai Fang <fskm...@gmail.com> >> wrote: >> > >> >> Hi, Jim. >> >> >> >>> how to pass additional headers when sending REST requests >> >> >> >> Could you share what headers do you want to send when using SQL >> Client? I >> >> think there are two cases we need to consider. Please correct me if I >> am >> >> wrong. >> >> >> >> # Case 1 >> >> >> >> If users wants to connect to the SQL Gateway with its password, I >> think the >> >> users should type >> >> ``` >> >> ./sql-client.sh --user xxx --password xxx >> >> ``` >> >> in the terminal and the OpenSessionRequest should be enough. >> >> >> >> # Case 2 >> >> >> >> If users wants to modify the execution config, users should type >> >> ``` >> >> Flink SQL> SET `<config_name>` = `<config_value>`; >> >> ``` >> >> in the terminal. The Client can send ExecuteStatementRequest to the >> >> Gateway. >> >> >> >>> As you have FLIPs or PRs, feel free to let me, Jamie, and Alexey know. >> >> >> >> It would be nice you can join us to finish the feature. I think the >> >> modification about the SQL Gateway side is ready to review. >> >> >> >> Best, >> >> Shengkai >> >> >> >> >> >> Jim Hughes <jhug...@confluent.io.invalid> 于2022年11月11日周五 05:19写道： >> >> >> >>> Hi Yu Zelin, >> >>> >> >>> I have read through your draft and it looks good. I am new to Flink, >> so >> >> I >> >>> haven't learned about everything which needs to be done yet. >> >>> >> >>> One of the considerations that I'm interested in understanding is how >> to >> >>> pass additional headers when sending REST requests. From looking at >> the >> >>> code, it looks like a custom `OutboundChannelHandlerFactory` could be >> >>> created to read additional configuration and set headers. Does that >> make >> >>> sense? >> >>> >> >>> Thank you very much for sharing the proof of concept code and the >> >>> document. As you have FLIPs or PRs, feel free to let me, Jamie, and >> >> Alexey >> >>> know. We'll be happy to review them. >> >>> >> >>> Cheers, >> >>> >> >>> Jim >> >>> >> >>> On Wed, Nov 9, 2022 at 11:43 PM yu zelin <yuzelin....@gmail.com> >> wrote: >> >>> >> >>>> Hi, all >> >>>> Sorry for late response. As Shengkai mentioned, Currently I’m working >> >>> with >> >>>> him on SQL Client, dedicating to implement the Remote Mode of SQL >> >>> Client. I >> >>>> have written a draft of implementation plan and will write a FLIP >> about >> >>> it >> >>>> ASAP. If you are interested in, please take a look at the draft and >> >> it’s >> >>>> nice if you give me some feedback. >> >>>> The doc is at: >> >>>> >> >>> >> >> >> https://docs.google.com/document/d/14cS4VBSamMUnlM_PZuK6QKLfriUuQU51iqET5oiYy_c/edit?usp=sharing >> >>>> >> >>>>> 2022年11月7日 11:19，Shengkai Fang <fskm...@gmail.com> 写道： >> >>>>> >> >>>>> Hi, all. Sorry for the late reply. >> >>>>> >> >>>>>> Is the gateway mode planned to be supported for SQL Client in 1.17? >> >>>>>> Do you have anything you can already share so we can start with >> >> your >> >>>> work or just play around with it. >> >>>>> >> >>>>> Yes. @yzl is working on it and he will list the implementation plan >> >>>> later and share the progress. I think the change is not very large >> and >> >> I >> >>>> think it's not a big problem to finish this in the release-1.17. I >> will >> >>>> join to develop this in the mid of November. >> >>>>> >> >>>>> Best, >> >>>>> Shengkai >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> Jamie Grier <jgr...@apache.org <mailto:jgr...@apache.org>> >> >>>> 于2022年11月5日周六 00:48写道： >> >>>>>> Hi Shengkai, >> >>>>>> >> >>>>>> We're doing more and more Flink development at Confluent these days >> >>> and >> >>>> we're currently trying to bootstrap a prototype that relies on the >> SQL >> >>>> Client and Gateway. We will be using the the components in some of >> our >> >>>> projects and would like to co-develop them with you and the rest of >> the >> >>>> Flink community. >> >>>>>> >> >>>>>> As of right now it's a pretty big blocker for our upcoming >> milestone >> >>>> that the SQL Client has not yet been modified to talk to the SQL >> >> Gateway >> >>>> and we want to help with this effort ASAP! We would be even willing >> to >> >>>> take over the work if it's not yet started but I suspect it already >> is. >> >>>>>> >> >>>>>> Anyway, rather than start working immediately on the SQL Client and >> >>>> adding a the new Gateway mode ourselves we wanted to start a >> >> conversation >> >>>> with you and see where you're at with things and offer to help. >> >>>>>> >> >>>>>> Do you have anything you can already share so we can start with >> your >> >>>> work or just play around with it. Like I said, we just want to get >> >>> started >> >>>> and are very able to help in this area. We see both the SQL Client >> and >> >>>> Gateway being very important for us and have a good team to help >> >> develop >> >>> it. >> >>>>>> >> >>>>>> Let me know if there is a branch you can share, etc. It would be >> >> much >> >>>> appreciated! >> >>>>>> >> >>>>>> -Jamie Grier >> >>>>>> >> >>>>>> >> >>>>>> On 2022/10/28 06:06:49 Shengkai Fang wrote: >> >>>>>>> Hi. >> >>>>>>> >> >>>>>>>> Is there a possibility for us to get engaged and at least >> >>> introduce >> >>>>>>> initial changes to support authentication/authorization? >> >>>>>>> >> >>>>>>> Yes. You can write a FLIP about the design and change. We can >> >>> discuss >> >>>> this >> >>>>>>> in the dev mail. If the FLIP passes, we can develop it together. >> >>>>>>> >> >>>>>>>> Another question about persistent Gateway: did you have any >> >>> specific >> >>>>>>> thoughts about it or some draft design? >> >>>>>>> >> >>>>>>> We don't have any detailed plan about this. But I know Livy has a >> >>>> similar >> >>>>>>> feature. >> >>>>>>> >> >>>>>>> Best, >> >>>>>>> Shengkai >> >>>>>>> >> >>>>>>> >> >>>>>>> Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: >> >>>> vendrov...@gmail.com>> 于2022年10月27日周四 15:12写道： >> >>>>>>> >> >>>>>>>> Apologies from the delayed response on my side. >> >>>>>>>> >> >>>>>>>> I think the authentication module is not part of our plan in >> >> 1.17 >> >>>> because >> >>>>>>>>> of the busy work. I think we'll start the design at the end of >> >>> the >> >>>>>>>>> release-1.17. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> Is there a possibility for us to get engaged and at least >> >>> introduce >> >>>>>>>> initial changes to support authentication/authorization? >> >>>> Specifically, >> >>>>>>>> changes in the API and in SQL Client. >> >>>>>>>> >> >>>>>>>> We expect the following authentication flow: >> >>>>>>>> >> >>>>>>>> On the SQL gateway we want to be able to use a delegation token. >> >>>>>>>> SQL client should be able to supply an API key. >> >>>>>>>> The SQL Gateway *would not *be submitting jobs on behalf of the >> >>>> client. >> >>>>>>>> >> >>>>>>>> Ideally it would be nice to introduce some interfaces in the SQL >> >>>> Gateway >> >>>>>>>> that would allow implementing custom authentication and >> >>>> authorization. >> >>>>>>>> >> >>>>>>>> Another question about persistent Gateway: did you have any >> >>> specific >> >>>>>>>> thoughts about it or some draft design? >> >>>>>>>> >> >>>>>>>> Thanks, >> >>>>>>>> Alexey >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Fri, Oct 21, 2022 at 1:13 AM Shengkai Fang < >> >> fskm...@gmail.com >> >>>> <mailto:fskm...@gmail.com>> wrote: >> >>>>>>>> >> >>>>>>>>> Sorry for the late response. >> >>>>>>>>> >> >>>>>>>>> In the next version(Flink 1.17), we plan to support the SQL >> >>> Client >> >>>> to >> >>>>>>>>> submit the statement to the Flink SQL Gateway. The FLINK-29486 >> >>>>>>>>> <https://issues.apache.org/jira/browse/FLINK-29486> is the >> >> first >> >>>> step to >> >>>>>>>>> remove the usage of the `Parser` in the client side, which >> >> needs >> >>>> to read >> >>>>>>>>> the table schema during the converting sql node to operation. I >> >>>> think the authentication >> >>>>>>>>> module is not part of our plan in 1.17 because of the busy >> >> work. >> >>> I >> >>>> think >> >>>>>>>>> we'll start the design at the end of the release-1.17. >> >>>>>>>>> But could you share more details about the requirements of the >> >>>>>>>>> authentication? >> >>>>>>>>> - Do you use the kerberos or delegation token or password to do >> >>> the >> >>>>>>>>> authentication? >> >>>>>>>>> - After the authentication, do you need the sql gateway to >> >> submit >> >>>> the >> >>>>>>>>> job on behalf of the client? >> >>>>>>>>> - ... >> >>>>>>>>> >> >>>>>>>>> For detailed implementation, I think Hive and Presto are good >> >>>> examples to >> >>>>>>>>> dig in. If you have some thoughts about the authentication >> >>> module, >> >>>>>>>>> please let me know. >> >>>>>>>>> >> >>>>>>>>> Best, >> >>>>>>>>> Shengkai >> >>>>>>>>> >> >>>>>>>>> Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: >> >>>> vendrov...@gmail.com>> 于2022年10月19日周三 00:37写道： >> >>>>>>>>> >> >>>>>>>>>> Thank you for the response, Yuxia! >> >>>>>>>>>> >> >>>>>>>>>> Shengkai, I would like to learn more about nearest and a bit >> >>> more >> >>>>>>>>>> distant plans about development of the SQL Gateway and the SQL >> >>>> Client. >> >>>>>>>>>> Do you have a description of the work planned or maybe can >> >> share >> >>>> general >> >>>>>>>>>> thoughts about the Authentication module, or Persistent >> >> Gateway. >> >>>>>>>>>> How can the authentication part be addressed on the SQL Client >> >>>> side? >> >>>>>>>>>> >> >>>>>>>>>> Regards, >> >>>>>>>>>> -Alexey >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Wed, Oct 12, 2022 at 11:24 PM yuxia < >> >>>> luoyu...@alumni.sjtu.edu.cn <mailto:luoyu...@alumni.sjtu.edu.cn>> >> >>>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>>> In what Flink’s release the connection from SQL Client to >> >> the >> >>>> Gateway >> >>>>>>>>>>> is >> >>>>>>>>>>> expected to be added? >> >>>>>>>>>>> Flink 1.17 >> >>>>>>>>>>> >> >>>>>>>>>>>> “Authentication module” (2) and “Persistent Gateway” (4) as >> >>>>>>>>>>> possible future work. Were there any recent discussions on >> >>> these >> >>>>>>>>>>> subjects? >> >>>>>>>>>>> No recent discussions on these subjects, but I think it'll >> >> come >> >>>> in >> >>>>>>>>>>> Flink 1.17 >> >>>>>>>>>>> >> >>>>>>>>>>>> Another related topic: are there ideas around making SQL >> >>>> Gateway a >> >>>>>>>>>>> multi-tenant >> >>>>>>>>>>> component? >> >>>>>>>>>>> Yes. >> >>>>>>>>>>> >> >>>>>>>>>>> Shengkaiis the maintainer of SQL Client and SQL gateway, >> >> maybe >> >>>> he can >> >>>>>>>>>>> provide more information. >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> Best regards, >> >>>>>>>>>>> Yuxia >> >>>>>>>>>>> >> >>>>>>>>>>> ----- 原始邮件 ----- >> >>>>>>>>>>> 发件人: "Alexey Leonov-Vendrovskiy" <vendrov...@gmail.com >> >>> <mailto: >> >>>> vendrov...@gmail.com>> >> >>>>>>>>>>> 收件人: "dev" <dev@flink.apache.org <mailto: >> >> dev@flink.apache.org >> >>>>> >> >>>>>>>>>>> 发送时间: 星期四, 2022年 10 月 13日 下午 12:33:08 >> >>>>>>>>>>> 主题: SQL Gateway and SQL Client >> >>>>>>>>>>> >> >>>>>>>>>>> Hi all, >> >>>>>>>>>>> >> >>>>>>>>>>> I’m Alexey from Confluent. This is my first email in this >> >>>> discussion >> >>>>>>>>>>> list. >> >>>>>>>>>>> I’m rather new to Flink, and to local customs of >> >> communication. >> >>>> I want >> >>>>>>>>>>> to >> >>>>>>>>>>> dive deeper and hopefully get more involved over time. >> >>>>>>>>>>> >> >>>>>>>>>>> Currently I have a few questions around SQL Gateway and SQL >> >>>> Client. >> >>>>>>>>>>> Specifically I wanted to learn what is the vision around the >> >>>> nearest >> >>>>>>>>>>> future >> >>>>>>>>>>> of these two components. >> >>>>>>>>>>> >> >>>>>>>>>>> In what Flink’s release the connection from SQL Client to the >> >>>> Gateway is >> >>>>>>>>>>> expected to be added? I was looking at >> >>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-29486, and >> >>> recently >> >>>> it got >> >>>>>>>>>>> renamed from “Enable SQL Client to Connect SQL Gateway in >> >>> Remote >> >>>> Mode” >> >>>>>>>>>>> to >> >>>>>>>>>>> “Introduce Client Parser to get statement type”. I did some >> >>>> search, but >> >>>>>>>>>>> didn’t find a good place where the client's work in this >> >>>> direction is >> >>>>>>>>>>> discussed or tracked. >> >>>>>>>>>>> >> >>>>>>>>>>> A couple questions about the SQL Gateway. The FLIP-91 >> >>>>>>>>>>> < >> >>>>>>>>>>> >> >>>> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Gateway#FLIP91:SupportSQLGateway-Futurework >> >>>>>>>>>>>> >> >>>>>>>>>>> mentions “Authentication module” (2) and “Persistent Gateway” >> >>>> (4) as >> >>>>>>>>>>> possible future work. Were there any recent discussions on >> >>> these >> >>>>>>>>>>> subjects? >> >>>>>>>>>>> Or maybe there are some ideas how to move these directions >> >>>> forward? >> >>>>>>>>>>> Another >> >>>>>>>>>>> related topic: are there ideas around making SQL Gateway a >> >>>> multi-tenant >> >>>>>>>>>>> component? >> >>>>>>>>>>> >> >>>>>>>>>>> Thank you, >> >>>>>>>>>>> >> >>>>>>>>>>> Alexey >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>> >> >>>> >> >>>> >> >>> >> >> >> >>
