Hi Jim, Thanks for your input. You can look here[1] on the server side. You can modify the return type of the handleRequest to Tuple2<Response, Headers> to get session-specific headers. RestClient has already exposed a method to send requests with headers for the client side.
[1] https://github.com/apache/flink/blob/master/flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/rest/handler/AbstractSqlGatewayRestHandler.java#L98 Jim Hughes <jhug...@confluent.io.invalid> 于2022年11月12日周六 00:45写道： > Hi Shengkai, > > I think there is an additional case where a proxy is between the client and > gateway. In that case, being able to pass headers would allow for > additional options / features. > > I see several PRs from Yu Zelin. Is there a first one to review? > > Cheers, > > Jim > > On Thu, Nov 10, 2022 at 9:42 PM Shengkai Fang <fskm...@gmail.com> wrote: > > > Hi, Jim. > > > > > how to pass additional headers when sending REST requests > > > > Could you share what headers do you want to send when using SQL Client? > I > > think there are two cases we need to consider. Please correct me if I am > > wrong. > > > > # Case 1 > > > > If users wants to connect to the SQL Gateway with its password, I think > the > > users should type > > ``` > > ./sql-client.sh --user xxx --password xxx > > ``` > > in the terminal and the OpenSessionRequest should be enough. > > > > # Case 2 > > > > If users wants to modify the execution config, users should type > > ``` > > Flink SQL> SET `<config_name>` = `<config_value>`; > > ``` > > in the terminal. The Client can send ExecuteStatementRequest to the > > Gateway. > > > > > As you have FLIPs or PRs, feel free to let me, Jamie, and Alexey know. > > > > It would be nice you can join us to finish the feature. I think the > > modification about the SQL Gateway side is ready to review. > > > > Best, > > Shengkai > > > > > > Jim Hughes <jhug...@confluent.io.invalid> 于2022年11月11日周五 05:19写道： > > > > > Hi Yu Zelin, > > > > > > I have read through your draft and it looks good. I am new to Flink, > so > > I > > > haven't learned about everything which needs to be done yet. > > > > > > One of the considerations that I'm interested in understanding is how > to > > > pass additional headers when sending REST requests. From looking at > the > > > code, it looks like a custom `OutboundChannelHandlerFactory` could be > > > created to read additional configuration and set headers. Does that > make > > > sense? > > > > > > Thank you very much for sharing the proof of concept code and the > > > document. As you have FLIPs or PRs, feel free to let me, Jamie, and > > Alexey > > > know. We'll be happy to review them. > > > > > > Cheers, > > > > > > Jim > > > > > > On Wed, Nov 9, 2022 at 11:43 PM yu zelin <yuzelin....@gmail.com> > wrote: > > > > > > > Hi, all > > > > Sorry for late response. As Shengkai mentioned, Currently I’m working > > > with > > > > him on SQL Client, dedicating to implement the Remote Mode of SQL > > > Client. I > > > > have written a draft of implementation plan and will write a FLIP > about > > > it > > > > ASAP. If you are interested in, please take a look at the draft and > > it’s > > > > nice if you give me some feedback. > > > > The doc is at: > > > > > > > > > > https://docs.google.com/document/d/14cS4VBSamMUnlM_PZuK6QKLfriUuQU51iqET5oiYy_c/edit?usp=sharing > > > > > > > > > 2022年11月7日 11:19，Shengkai Fang <fskm...@gmail.com> 写道： > > > > > > > > > > Hi, all. Sorry for the late reply. > > > > > > > > > > > Is the gateway mode planned to be supported for SQL Client in > 1.17? > > > > > > Do you have anything you can already share so we can start with > > your > > > > work or just play around with it. > > > > > > > > > > Yes. @yzl is working on it and he will list the implementation plan > > > > later and share the progress. I think the change is not very large > and > > I > > > > think it's not a big problem to finish this in the release-1.17. I > will > > > > join to develop this in the mid of November. > > > > > > > > > > Best, > > > > > Shengkai > > > > > > > > > > > > > > > > > > > > > > > > > Jamie Grier <jgr...@apache.org <mailto:jgr...@apache.org>> > > > > 于2022年11月5日周六 00:48写道： > > > > >> Hi Shengkai, > > > > >> > > > > >> We're doing more and more Flink development at Confluent these > days > > > and > > > > we're currently trying to bootstrap a prototype that relies on the > SQL > > > > Client and Gateway. We will be using the the components in some of > our > > > > projects and would like to co-develop them with you and the rest of > the > > > > Flink community. > > > > >> > > > > >> As of right now it's a pretty big blocker for our upcoming > milestone > > > > that the SQL Client has not yet been modified to talk to the SQL > > Gateway > > > > and we want to help with this effort ASAP! We would be even willing > to > > > > take over the work if it's not yet started but I suspect it already > is. > > > > >> > > > > >> Anyway, rather than start working immediately on the SQL Client > and > > > > adding a the new Gateway mode ourselves we wanted to start a > > conversation > > > > with you and see where you're at with things and offer to help. > > > > >> > > > > >> Do you have anything you can already share so we can start with > your > > > > work or just play around with it. Like I said, we just want to get > > > started > > > > and are very able to help in this area. We see both the SQL Client > and > > > > Gateway being very important for us and have a good team to help > > develop > > > it. > > > > >> > > > > >> Let me know if there is a branch you can share, etc. It would be > > much > > > > appreciated! > > > > >> > > > > >> -Jamie Grier > > > > >> > > > > >> > > > > >> On 2022/10/28 06:06:49 Shengkai Fang wrote: > > > > >> > Hi. > > > > >> > > > > > >> > > Is there a possibility for us to get engaged and at least > > > introduce > > > > >> > initial changes to support authentication/authorization? > > > > >> > > > > > >> > Yes. You can write a FLIP about the design and change. We can > > > discuss > > > > this > > > > >> > in the dev mail. If the FLIP passes, we can develop it together. > > > > >> > > > > > >> > > Another question about persistent Gateway: did you have any > > > specific > > > > >> > thoughts about it or some draft design? > > > > >> > > > > > >> > We don't have any detailed plan about this. But I know Livy has > a > > > > similar > > > > >> > feature. > > > > >> > > > > > >> > Best, > > > > >> > Shengkai > > > > >> > > > > > >> > > > > > >> > Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: > > > > vendrov...@gmail.com>> 于2022年10月27日周四 15:12写道： > > > > >> > > > > > >> > > Apologies from the delayed response on my side. > > > > >> > > > > > > >> > > I think the authentication module is not part of our plan in > > 1.17 > > > > because > > > > >> > >> of the busy work. I think we'll start the design at the end > of > > > the > > > > >> > >> release-1.17. > > > > >> > > > > > > >> > > > > > > >> > > Is there a possibility for us to get engaged and at least > > > introduce > > > > >> > > initial changes to support authentication/authorization? > > > > Specifically, > > > > >> > > changes in the API and in SQL Client. > > > > >> > > > > > > >> > > We expect the following authentication flow: > > > > >> > > > > > > >> > > On the SQL gateway we want to be able to use a delegation > token. > > > > >> > > SQL client should be able to supply an API key. > > > > >> > > The SQL Gateway *would not *be submitting jobs on behalf of > the > > > > client. > > > > >> > > > > > > >> > > Ideally it would be nice to introduce some interfaces in the > SQL > > > > Gateway > > > > >> > > that would allow implementing custom authentication and > > > > authorization. > > > > >> > > > > > > >> > > Another question about persistent Gateway: did you have any > > > specific > > > > >> > > thoughts about it or some draft design? > > > > >> > > > > > > >> > > Thanks, > > > > >> > > Alexey > > > > >> > > > > > > >> > > > > > > >> > > On Fri, Oct 21, 2022 at 1:13 AM Shengkai Fang < > > fskm...@gmail.com > > > > <mailto:fskm...@gmail.com>> wrote: > > > > >> > > > > > > >> > >> Sorry for the late response. > > > > >> > >> > > > > >> > >> In the next version(Flink 1.17), we plan to support the SQL > > > Client > > > > to > > > > >> > >> submit the statement to the Flink SQL Gateway. The > FLINK-29486 > > > > >> > >> <https://issues.apache.org/jira/browse/FLINK-29486> is the > > first > > > > step to > > > > >> > >> remove the usage of the `Parser` in the client side, which > > needs > > > > to read > > > > >> > >> the table schema during the converting sql node to > operation. I > > > > think the authentication > > > > >> > >> module is not part of our plan in 1.17 because of the busy > > work. > > > I > > > > think > > > > >> > >> we'll start the design at the end of the release-1.17. > > > > >> > >> But could you share more details about the requirements of > the > > > > >> > >> authentication? > > > > >> > >> - Do you use the kerberos or delegation token or password to > do > > > the > > > > >> > >> authentication? > > > > >> > >> - After the authentication, do you need the sql gateway to > > submit > > > > the > > > > >> > >> job on behalf of the client? > > > > >> > >> - ... > > > > >> > >> > > > > >> > >> For detailed implementation, I think Hive and Presto are good > > > > examples to > > > > >> > >> dig in. If you have some thoughts about the authentication > > > module, > > > > >> > >> please let me know. > > > > >> > >> > > > > >> > >> Best, > > > > >> > >> Shengkai > > > > >> > >> > > > > >> > >> Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: > > > > vendrov...@gmail.com>> 于2022年10月19日周三 00:37写道： > > > > >> > >> > > > > >> > >>> Thank you for the response, Yuxia! > > > > >> > >>> > > > > >> > >>> Shengkai, I would like to learn more about nearest and a bit > > > more > > > > >> > >>> distant plans about development of the SQL Gateway and the > SQL > > > > Client. > > > > >> > >>> Do you have a description of the work planned or maybe can > > share > > > > general > > > > >> > >>> thoughts about the Authentication module, or Persistent > > Gateway. > > > > >> > >>> How can the authentication part be addressed on the SQL > Client > > > > side? > > > > >> > >>> > > > > >> > >>> Regards, > > > > >> > >>> -Alexey > > > > >> > >>> > > > > >> > >>> > > > > >> > >>> On Wed, Oct 12, 2022 at 11:24 PM yuxia < > > > > luoyu...@alumni.sjtu.edu.cn <mailto:luoyu...@alumni.sjtu.edu.cn>> > > > > >> > >>> wrote: > > > > >> > >>> > > > > >> > >>>> > In what Flink’s release the connection from SQL Client to > > the > > > > Gateway > > > > >> > >>>> is > > > > >> > >>>> expected to be added? > > > > >> > >>>> Flink 1.17 > > > > >> > >>>> > > > > >> > >>>> > “Authentication module” (2) and “Persistent Gateway” (4) > as > > > > >> > >>>> possible future work. Were there any recent discussions on > > > these > > > > >> > >>>> subjects? > > > > >> > >>>> No recent discussions on these subjects, but I think it'll > > come > > > > in > > > > >> > >>>> Flink 1.17 > > > > >> > >>>> > > > > >> > >>>> > Another related topic: are there ideas around making SQL > > > > Gateway a > > > > >> > >>>> multi-tenant > > > > >> > >>>> component? > > > > >> > >>>> Yes. > > > > >> > >>>> > > > > >> > >>>> Shengkaiis the maintainer of SQL Client and SQL gateway, > > maybe > > > > he can > > > > >> > >>>> provide more information. > > > > >> > >>>> > > > > >> > >>>> > > > > >> > >>>> > > > > >> > >>>> Best regards, > > > > >> > >>>> Yuxia > > > > >> > >>>> > > > > >> > >>>> ----- 原始邮件 ----- > > > > >> > >>>> 发件人: "Alexey Leonov-Vendrovskiy" <vendrov...@gmail.com > > > <mailto: > > > > vendrov...@gmail.com>> > > > > >> > >>>> 收件人: "dev" <dev@flink.apache.org <mailto: > > dev@flink.apache.org > > > >> > > > > >> > >>>> 发送时间: 星期四, 2022年 10 月 13日 下午 12:33:08 > > > > >> > >>>> 主题: SQL Gateway and SQL Client > > > > >> > >>>> > > > > >> > >>>> Hi all, > > > > >> > >>>> > > > > >> > >>>> I’m Alexey from Confluent. This is my first email in this > > > > discussion > > > > >> > >>>> list. > > > > >> > >>>> I’m rather new to Flink, and to local customs of > > communication. > > > > I want > > > > >> > >>>> to > > > > >> > >>>> dive deeper and hopefully get more involved over time. > > > > >> > >>>> > > > > >> > >>>> Currently I have a few questions around SQL Gateway and SQL > > > > Client. > > > > >> > >>>> Specifically I wanted to learn what is the vision around > the > > > > nearest > > > > >> > >>>> future > > > > >> > >>>> of these two components. > > > > >> > >>>> > > > > >> > >>>> In what Flink’s release the connection from SQL Client to > the > > > > Gateway is > > > > >> > >>>> expected to be added? I was looking at > > > > >> > >>>> https://issues.apache.org/jira/browse/FLINK-29486, and > > > recently > > > > it got > > > > >> > >>>> renamed from “Enable SQL Client to Connect SQL Gateway in > > > Remote > > > > Mode” > > > > >> > >>>> to > > > > >> > >>>> “Introduce Client Parser to get statement type”. I did > some > > > > search, but > > > > >> > >>>> didn’t find a good place where the client's work in this > > > > direction is > > > > >> > >>>> discussed or tracked. > > > > >> > >>>> > > > > >> > >>>> A couple questions about the SQL Gateway. The FLIP-91 > > > > >> > >>>> < > > > > >> > >>>> > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Gateway#FLIP91:SupportSQLGateway-Futurework > > > > >> > >>>> > > > > > >> > >>>> mentions “Authentication module” (2) and “Persistent > Gateway” > > > > (4) as > > > > >> > >>>> possible future work. Were there any recent discussions on > > > these > > > > >> > >>>> subjects? > > > > >> > >>>> Or maybe there are some ideas how to move these directions > > > > forward? > > > > >> > >>>> Another > > > > >> > >>>> related topic: are there ideas around making SQL Gateway a > > > > multi-tenant > > > > >> > >>>> component? > > > > >> > >>>> > > > > >> > >>>> Thank you, > > > > >> > >>>> > > > > >> > >>>> Alexey > > > > >> > >>>> > > > > >> > >>> > > > > >> > > > > > > > > > > > > > > >
