Hi Yu Zelin, I have read through your draft and it looks good. I am new to Flink, so I haven't learned about everything which needs to be done yet.
One of the considerations that I'm interested in understanding is how to pass additional headers when sending REST requests. From looking at the code, it looks like a custom `OutboundChannelHandlerFactory` could be created to read additional configuration and set headers. Does that make sense? Thank you very much for sharing the proof of concept code and the document. As you have FLIPs or PRs, feel free to let me, Jamie, and Alexey know. We'll be happy to review them. Cheers, Jim On Wed, Nov 9, 2022 at 11:43 PM yu zelin <yuzelin....@gmail.com> wrote: > Hi, all > Sorry for late response. As Shengkai mentioned, Currently I’m working with > him on SQL Client, dedicating to implement the Remote Mode of SQL Client. I > have written a draft of implementation plan and will write a FLIP about it > ASAP. If you are interested in, please take a look at the draft and it’s > nice if you give me some feedback. > The doc is at: > https://docs.google.com/document/d/14cS4VBSamMUnlM_PZuK6QKLfriUuQU51iqET5oiYy_c/edit?usp=sharing > > > 2022年11月7日 11:19，Shengkai Fang <fskm...@gmail.com> 写道： > > > > Hi, all. Sorry for the late reply. > > > > > Is the gateway mode planned to be supported for SQL Client in 1.17? > > > Do you have anything you can already share so we can start with your > work or just play around with it. > > > > Yes. @yzl is working on it and he will list the implementation plan > later and share the progress. I think the change is not very large and I > think it's not a big problem to finish this in the release-1.17. I will > join to develop this in the mid of November. > > > > Best, > > Shengkai > > > > > > > > > > Jamie Grier <jgr...@apache.org <mailto:jgr...@apache.org>> > 于2022年11月5日周六 00:48写道： > >> Hi Shengkai, > >> > >> We're doing more and more Flink development at Confluent these days and > we're currently trying to bootstrap a prototype that relies on the SQL > Client and Gateway. We will be using the the components in some of our > projects and would like to co-develop them with you and the rest of the > Flink community. > >> > >> As of right now it's a pretty big blocker for our upcoming milestone > that the SQL Client has not yet been modified to talk to the SQL Gateway > and we want to help with this effort ASAP! We would be even willing to > take over the work if it's not yet started but I suspect it already is. > >> > >> Anyway, rather than start working immediately on the SQL Client and > adding a the new Gateway mode ourselves we wanted to start a conversation > with you and see where you're at with things and offer to help. > >> > >> Do you have anything you can already share so we can start with your > work or just play around with it. Like I said, we just want to get started > and are very able to help in this area. We see both the SQL Client and > Gateway being very important for us and have a good team to help develop it. > >> > >> Let me know if there is a branch you can share, etc. It would be much > appreciated! > >> > >> -Jamie Grier > >> > >> > >> On 2022/10/28 06:06:49 Shengkai Fang wrote: > >> > Hi. > >> > > >> > > Is there a possibility for us to get engaged and at least introduce > >> > initial changes to support authentication/authorization? > >> > > >> > Yes. You can write a FLIP about the design and change. We can discuss > this > >> > in the dev mail. If the FLIP passes, we can develop it together. > >> > > >> > > Another question about persistent Gateway: did you have any specific > >> > thoughts about it or some draft design? > >> > > >> > We don't have any detailed plan about this. But I know Livy has a > similar > >> > feature. > >> > > >> > Best, > >> > Shengkai > >> > > >> > > >> > Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: > vendrov...@gmail.com>> 于2022年10月27日周四 15:12写道： > >> > > >> > > Apologies from the delayed response on my side. > >> > > > >> > > I think the authentication module is not part of our plan in 1.17 > because > >> > >> of the busy work. I think we'll start the design at the end of the > >> > >> release-1.17. > >> > > > >> > > > >> > > Is there a possibility for us to get engaged and at least introduce > >> > > initial changes to support authentication/authorization? > Specifically, > >> > > changes in the API and in SQL Client. > >> > > > >> > > We expect the following authentication flow: > >> > > > >> > > On the SQL gateway we want to be able to use a delegation token. > >> > > SQL client should be able to supply an API key. > >> > > The SQL Gateway *would not *be submitting jobs on behalf of the > client. > >> > > > >> > > Ideally it would be nice to introduce some interfaces in the SQL > Gateway > >> > > that would allow implementing custom authentication and > authorization. > >> > > > >> > > Another question about persistent Gateway: did you have any specific > >> > > thoughts about it or some draft design? > >> > > > >> > > Thanks, > >> > > Alexey > >> > > > >> > > > >> > > On Fri, Oct 21, 2022 at 1:13 AM Shengkai Fang <fskm...@gmail.com > <mailto:fskm...@gmail.com>> wrote: > >> > > > >> > >> Sorry for the late response. > >> > >> > >> > >> In the next version(Flink 1.17), we plan to support the SQL Client > to > >> > >> submit the statement to the Flink SQL Gateway. The FLINK-29486 > >> > >> <https://issues.apache.org/jira/browse/FLINK-29486> is the first > step to > >> > >> remove the usage of the `Parser` in the client side, which needs > to read > >> > >> the table schema during the converting sql node to operation. I > think the authentication > >> > >> module is not part of our plan in 1.17 because of the busy work. I > think > >> > >> we'll start the design at the end of the release-1.17. > >> > >> But could you share more details about the requirements of the > >> > >> authentication? > >> > >> - Do you use the kerberos or delegation token or password to do the > >> > >> authentication? > >> > >> - After the authentication, do you need the sql gateway to submit > the > >> > >> job on behalf of the client? > >> > >> - ... > >> > >> > >> > >> For detailed implementation, I think Hive and Presto are good > examples to > >> > >> dig in. If you have some thoughts about the authentication module, > >> > >> please let me know. > >> > >> > >> > >> Best, > >> > >> Shengkai > >> > >> > >> > >> Alexey Leonov-Vendrovskiy <vendrov...@gmail.com <mailto: > vendrov...@gmail.com>> 于2022年10月19日周三 00:37写道： > >> > >> > >> > >>> Thank you for the response, Yuxia! > >> > >>> > >> > >>> Shengkai, I would like to learn more about nearest and a bit more > >> > >>> distant plans about development of the SQL Gateway and the SQL > Client. > >> > >>> Do you have a description of the work planned or maybe can share > general > >> > >>> thoughts about the Authentication module, or Persistent Gateway. > >> > >>> How can the authentication part be addressed on the SQL Client > side? > >> > >>> > >> > >>> Regards, > >> > >>> -Alexey > >> > >>> > >> > >>> > >> > >>> On Wed, Oct 12, 2022 at 11:24 PM yuxia < > luoyu...@alumni.sjtu.edu.cn <mailto:luoyu...@alumni.sjtu.edu.cn>> > >> > >>> wrote: > >> > >>> > >> > >>>> > In what Flink’s release the connection from SQL Client to the > Gateway > >> > >>>> is > >> > >>>> expected to be added? > >> > >>>> Flink 1.17 > >> > >>>> > >> > >>>> > “Authentication module” (2) and “Persistent Gateway” (4) as > >> > >>>> possible future work. Were there any recent discussions on these > >> > >>>> subjects? > >> > >>>> No recent discussions on these subjects, but I think it'll come > in > >> > >>>> Flink 1.17 > >> > >>>> > >> > >>>> > Another related topic: are there ideas around making SQL > Gateway a > >> > >>>> multi-tenant > >> > >>>> component? > >> > >>>> Yes. > >> > >>>> > >> > >>>> Shengkaiis the maintainer of SQL Client and SQL gateway, maybe > he can > >> > >>>> provide more information. > >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> Best regards, > >> > >>>> Yuxia > >> > >>>> > >> > >>>> ----- 原始邮件 ----- > >> > >>>> 发件人: "Alexey Leonov-Vendrovskiy" <vendrov...@gmail.com <mailto: > vendrov...@gmail.com>> > >> > >>>> 收件人: "dev" <dev@flink.apache.org <mailto:dev@flink.apache.org>> > >> > >>>> 发送时间: 星期四, 2022年 10 月 13日 下午 12:33:08 > >> > >>>> 主题: SQL Gateway and SQL Client > >> > >>>> > >> > >>>> Hi all, > >> > >>>> > >> > >>>> I’m Alexey from Confluent. This is my first email in this > discussion > >> > >>>> list. > >> > >>>> I’m rather new to Flink, and to local customs of communication. > I want > >> > >>>> to > >> > >>>> dive deeper and hopefully get more involved over time. > >> > >>>> > >> > >>>> Currently I have a few questions around SQL Gateway and SQL > Client. > >> > >>>> Specifically I wanted to learn what is the vision around the > nearest > >> > >>>> future > >> > >>>> of these two components. > >> > >>>> > >> > >>>> In what Flink’s release the connection from SQL Client to the > Gateway is > >> > >>>> expected to be added? I was looking at > >> > >>>> https://issues.apache.org/jira/browse/FLINK-29486, and recently > it got > >> > >>>> renamed from “Enable SQL Client to Connect SQL Gateway in Remote > Mode” > >> > >>>> to > >> > >>>> “Introduce Client Parser to get statement type”. I did some > search, but > >> > >>>> didn’t find a good place where the client's work in this > direction is > >> > >>>> discussed or tracked. > >> > >>>> > >> > >>>> A couple questions about the SQL Gateway. The FLIP-91 > >> > >>>> < > >> > >>>> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Gateway#FLIP91:SupportSQLGateway-Futurework > >> > >>>> > > >> > >>>> mentions “Authentication module” (2) and “Persistent Gateway” > (4) as > >> > >>>> possible future work. Were there any recent discussions on these > >> > >>>> subjects? > >> > >>>> Or maybe there are some ideas how to move these directions > forward? > >> > >>>> Another > >> > >>>> related topic: are there ideas around making SQL Gateway a > multi-tenant > >> > >>>> component? > >> > >>>> > >> > >>>> Thank you, > >> > >>>> > >> > >>>> Alexey > >> > >>>> > >> > >>> > >> > > >
