Hi Shengkai, I think there is an additional case where a proxy is between the client and gateway. In that case, being able to pass headers would allow for additional options / features.
I see several PRs from Yu Zelin. Is there a first one to review? Cheers, Jim On Thu, Nov 10, 2022 at 9:42 PM Shengkai Fang <fskm...@gmail.com> wrote: > Hi, Jim. > > > how to pass additional headers when sending REST requests > > Could you share what headers do you want to send when using SQL Client? I > think there are two cases we need to consider. Please correct me if I am > wrong. > > # Case 1 > > If users wants to connect to the SQL Gateway with its password, I think the > users should type > ``` > ./sql-client.sh --user xxx --password xxx > ``` > in the terminal and the OpenSessionRequest should be enough. > > # Case 2 > > If users wants to modify the execution config, users should type > ``` > Flink SQL> SET `<config_name>` = `<config_value>`; > ``` > in the terminal. The Client can send ExecuteStatementRequest to the > Gateway. > > > As you have FLIPs or PRs, feel free to let me, Jamie, and Alexey know. > > It would be nice you can join us to finish the feature. I think the > modification about the SQL Gateway side is ready to review. > > Best, > Shengkai > > > Jim Hughes <jhug...@confluent.io.invalid> 于2022年11月11日周五 05:19写道： > > > Hi Yu Zelin, > > > > I have read through your draft and it looks good. I am new to Flink, so > I > > haven't learned about everything which needs to be done yet. > > > > One of the considerations that I'm interested in understanding is how to > > pass additional headers when sending REST requests. From looking at the > > code, it looks like a custom `OutboundChannelHandlerFactory` could be > > created to read additional configuration and set headers. Does that make > > sense? > > > > Thank you very much for sharing the proof of concept code and the > > document. As you have FLIPs or PRs, feel free to let me, Jamie, and > Alexey > > know. We'll be happy to review them. > > > > Cheers, > > > > Jim > > > > On Wed, Nov 9, 2022 at 11:43 PM yu zelin <yuzelin....@gmail.com> wrote: > > > > > Hi, all > > > Sorry for late response. As Shengkai mentioned, Currently I’m working > > with > > > him on SQL Client, dedicating to implement the Remote Mode of SQL > > Client. I > > > have written a draft of implementation plan and will write a FLIP about > > it > > > ASAP. If you are interested in, please take a look at the draft and > it’s > > > nice if you give me some feedback. > > > The doc is at: > > > > > > https://docs.google.com/document/d/14cS4VBSamMUnlM_PZuK6QKLfriUuQU51iqET5oiYy_c/edit?usp=sharing > > > > > > > 2022年11月7日 11:19，Shengkai Fang <fskm...@gmail.com> 写道： > > > > > > > > Hi, all. Sorry for the late reply. > > > > > > > > > Is the gateway mode planned to be supported for SQL Client in 1.17? > > > > > Do you have anything you can already share so we can start with > your > > > work or just play around with it. > > > > > > > > Yes. @yzl is working on it and he will list the implementation plan > > > later and share the progress. I think the change is not very large and > I > > > think it's not a big problem to finish this in the release-1.17. I will > > > join to develop this in the mid of November. > > > > > > > > Best, > > > > Shengkai > > > > > > > > > > > > > > > > > > > > Jamie Grier <jgr...@apache.org <mailto:jgr...@apache.org>> > > > 于2022年11月5日周六 00:48写道： > > > >> Hi Shengkai, > > > >> > > > >> We're doing more and more Flink development at Confluent these days > > and > > > we're currently trying to bootstrap a prototype that relies on the SQL > > > Client and Gateway. We will be using the the components in some of our > > > projects and would like to co-develop them with you and the rest of the > > > Flink community. > > > >> > > > >> As of right now it's a pretty big blocker for our upcoming milestone > > > that the SQL Client has not yet been modified to talk to the SQL > Gateway > > > and we want to help with this effort ASAP! We would be even willing to > > > take over the work if it's not yet started but I suspect it already is. > > > >> > > > >> Anyway, rather than start working immediately on the SQL Client and > > > adding a the new Gateway mode ourselves we wanted to start a > conversation > > > with you and see where you're at with things and offer to help. > > > >> > > > >> Do you have anything you can already share so we can start with your > > > work or just play around with it. Like I said, we just want to get > > started > > > and are very able to help in this area. We see both the SQL Client and > > > Gateway being very important for us and have a good team to help > develop > > it. > > > >> > > > >> Let me know if there is a branch you can share, etc. It would be > much > > > appreciated! > > > >> > > > >> -Jamie Grier > > > >> > > > >> > > > >> On 2022/10/28 06:06:49 Shengkai Fang wrote: > > > >> > Hi. > > > >> > > > > >> > > Is there a possibility for us to get engaged and at least > > introduce > > > >> > initial changes to support authentication/authorization? > > > >> > > > > >> > Yes. You can write a FLIP about the design and change. We can > > discuss > > > this > > > >> > in the dev mail. If the FLIP passes, we can develop it together. > > > >> > > > > >> > > Another question about persistent Gateway: did you have any > > specific > > > >> > thoughts about it or some draft design? > > > >> > > > > >> > We don't have any detailed plan about this. But I know Livy has a > > > similar > > > >> > feature. > > > >> > > > > >> > Best, > > > >> > Shengkai > > > >> > > > > >> > > > > >> > Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: > > > vendrov...@gmail.com>> 于2022年10月27日周四 15:12写道： > > > >> > > > > >> > > Apologies from the delayed response on my side. > > > >> > > > > > >> > > I think the authentication module is not part of our plan in > 1.17 > > > because > > > >> > >> of the busy work. I think we'll start the design at the end of > > the > > > >> > >> release-1.17. > > > >> > > > > > >> > > > > > >> > > Is there a possibility for us to get engaged and at least > > introduce > > > >> > > initial changes to support authentication/authorization? > > > Specifically, > > > >> > > changes in the API and in SQL Client. > > > >> > > > > > >> > > We expect the following authentication flow: > > > >> > > > > > >> > > On the SQL gateway we want to be able to use a delegation token. > > > >> > > SQL client should be able to supply an API key. > > > >> > > The SQL Gateway *would not *be submitting jobs on behalf of the > > > client. > > > >> > > > > > >> > > Ideally it would be nice to introduce some interfaces in the SQL > > > Gateway > > > >> > > that would allow implementing custom authentication and > > > authorization. > > > >> > > > > > >> > > Another question about persistent Gateway: did you have any > > specific > > > >> > > thoughts about it or some draft design? > > > >> > > > > > >> > > Thanks, > > > >> > > Alexey > > > >> > > > > > >> > > > > > >> > > On Fri, Oct 21, 2022 at 1:13 AM Shengkai Fang < > fskm...@gmail.com > > > <mailto:fskm...@gmail.com>> wrote: > > > >> > > > > > >> > >> Sorry for the late response. > > > >> > >> > > > >> > >> In the next version(Flink 1.17), we plan to support the SQL > > Client > > > to > > > >> > >> submit the statement to the Flink SQL Gateway. The FLINK-29486 > > > >> > >> <https://issues.apache.org/jira/browse/FLINK-29486> is the > first > > > step to > > > >> > >> remove the usage of the `Parser` in the client side, which > needs > > > to read > > > >> > >> the table schema during the converting sql node to operation. I > > > think the authentication > > > >> > >> module is not part of our plan in 1.17 because of the busy > work. > > I > > > think > > > >> > >> we'll start the design at the end of the release-1.17. > > > >> > >> But could you share more details about the requirements of the > > > >> > >> authentication? > > > >> > >> - Do you use the kerberos or delegation token or password to do > > the > > > >> > >> authentication? > > > >> > >> - After the authentication, do you need the sql gateway to > submit > > > the > > > >> > >> job on behalf of the client? > > > >> > >> - ... > > > >> > >> > > > >> > >> For detailed implementation, I think Hive and Presto are good > > > examples to > > > >> > >> dig in. If you have some thoughts about the authentication > > module, > > > >> > >> please let me know. > > > >> > >> > > > >> > >> Best, > > > >> > >> Shengkai > > > >> > >> > > > >> > >> Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: > > > vendrov...@gmail.com>> 于2022年10月19日周三 00:37写道： > > > >> > >> > > > >> > >>> Thank you for the response, Yuxia! > > > >> > >>> > > > >> > >>> Shengkai, I would like to learn more about nearest and a bit > > more > > > >> > >>> distant plans about development of the SQL Gateway and the SQL > > > Client. > > > >> > >>> Do you have a description of the work planned or maybe can > share > > > general > > > >> > >>> thoughts about the Authentication module, or Persistent > Gateway. > > > >> > >>> How can the authentication part be addressed on the SQL Client > > > side? > > > >> > >>> > > > >> > >>> Regards, > > > >> > >>> -Alexey > > > >> > >>> > > > >> > >>> > > > >> > >>> On Wed, Oct 12, 2022 at 11:24 PM yuxia < > > > luoyu...@alumni.sjtu.edu.cn <mailto:luoyu...@alumni.sjtu.edu.cn>> > > > >> > >>> wrote: > > > >> > >>> > > > >> > >>>> > In what Flink’s release the connection from SQL Client to > the > > > Gateway > > > >> > >>>> is > > > >> > >>>> expected to be added? > > > >> > >>>> Flink 1.17 > > > >> > >>>> > > > >> > >>>> > “Authentication module” (2) and “Persistent Gateway” (4) as > > > >> > >>>> possible future work. Were there any recent discussions on > > these > > > >> > >>>> subjects? > > > >> > >>>> No recent discussions on these subjects, but I think it'll > come > > > in > > > >> > >>>> Flink 1.17 > > > >> > >>>> > > > >> > >>>> > Another related topic: are there ideas around making SQL > > > Gateway a > > > >> > >>>> multi-tenant > > > >> > >>>> component? > > > >> > >>>> Yes. > > > >> > >>>> > > > >> > >>>> Shengkaiis the maintainer of SQL Client and SQL gateway, > maybe > > > he can > > > >> > >>>> provide more information. > > > >> > >>>> > > > >> > >>>> > > > >> > >>>> > > > >> > >>>> Best regards, > > > >> > >>>> Yuxia > > > >> > >>>> > > > >> > >>>> ----- 原始邮件 ----- > > > >> > >>>> 发件人: "Alexey Leonov-Vendrovskiy" <vendrov...@gmail.com > > <mailto: > > > vendrov...@gmail.com>> > > > >> > >>>> 收件人: "dev" <dev@flink.apache.org <mailto: > dev@flink.apache.org > > >> > > > >> > >>>> 发送时间: 星期四, 2022年 10 月 13日 下午 12:33:08 > > > >> > >>>> 主题: SQL Gateway and SQL Client > > > >> > >>>> > > > >> > >>>> Hi all, > > > >> > >>>> > > > >> > >>>> I’m Alexey from Confluent. This is my first email in this > > > discussion > > > >> > >>>> list. > > > >> > >>>> I’m rather new to Flink, and to local customs of > communication. > > > I want > > > >> > >>>> to > > > >> > >>>> dive deeper and hopefully get more involved over time. > > > >> > >>>> > > > >> > >>>> Currently I have a few questions around SQL Gateway and SQL > > > Client. > > > >> > >>>> Specifically I wanted to learn what is the vision around the > > > nearest > > > >> > >>>> future > > > >> > >>>> of these two components. > > > >> > >>>> > > > >> > >>>> In what Flink’s release the connection from SQL Client to the > > > Gateway is > > > >> > >>>> expected to be added? I was looking at > > > >> > >>>> https://issues.apache.org/jira/browse/FLINK-29486, and > > recently > > > it got > > > >> > >>>> renamed from “Enable SQL Client to Connect SQL Gateway in > > Remote > > > Mode” > > > >> > >>>> to > > > >> > >>>> “Introduce Client Parser to get statement type”. I did some > > > search, but > > > >> > >>>> didn’t find a good place where the client's work in this > > > direction is > > > >> > >>>> discussed or tracked. > > > >> > >>>> > > > >> > >>>> A couple questions about the SQL Gateway. The FLIP-91 > > > >> > >>>> < > > > >> > >>>> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Gateway#FLIP91:SupportSQLGateway-Futurework > > > >> > >>>> > > > > >> > >>>> mentions “Authentication module” (2) and “Persistent Gateway” > > > (4) as > > > >> > >>>> possible future work. Were there any recent discussions on > > these > > > >> > >>>> subjects? > > > >> > >>>> Or maybe there are some ideas how to move these directions > > > forward? > > > >> > >>>> Another > > > >> > >>>> related topic: are there ideas around making SQL Gateway a > > > multi-tenant > > > >> > >>>> component? > > > >> > >>>> > > > >> > >>>> Thank you, > > > >> > >>>> > > > >> > >>>> Alexey > > > >> > >>>> > > > >> > >>> > > > >> > > > > > > > > > >
