Hi, all Sorry for late response. As Shengkai mentioned, Currently I’m working with him on SQL Client, dedicating to implement the Remote Mode of SQL Client. I have written a draft of implementation plan and will write a FLIP about it ASAP. If you are interested in, please take a look at the draft and it’s nice if you give me some feedback. The doc is at: https://docs.google.com/document/d/14cS4VBSamMUnlM_PZuK6QKLfriUuQU51iqET5oiYy_c/edit?usp=sharing
> 2022年11月7日 11:19，Shengkai Fang <fskm...@gmail.com> 写道： > > Hi, all. Sorry for the late reply. > > > Is the gateway mode planned to be supported for SQL Client in 1.17? > > Do you have anything you can already share so we can start with your work > > or just play around with it. > > Yes. @yzl is working on it and he will list the implementation plan later and > share the progress. I think the change is not very large and I think it's not > a big problem to finish this in the release-1.17. I will join to develop this > in the mid of November. > > Best, > Shengkai > > > > > Jamie Grier <jgr...@apache.org <mailto:jgr...@apache.org>> 于2022年11月5日周六 > 00:48写道： >> Hi Shengkai, >> >> We're doing more and more Flink development at Confluent these days and >> we're currently trying to bootstrap a prototype that relies on the SQL >> Client and Gateway. We will be using the the components in some of our >> projects and would like to co-develop them with you and the rest of the >> Flink community. >> >> As of right now it's a pretty big blocker for our upcoming milestone that >> the SQL Client has not yet been modified to talk to the SQL Gateway and we >> want to help with this effort ASAP! We would be even willing to take over >> the work if it's not yet started but I suspect it already is. >> >> Anyway, rather than start working immediately on the SQL Client and adding a >> the new Gateway mode ourselves we wanted to start a conversation with you >> and see where you're at with things and offer to help. >> >> Do you have anything you can already share so we can start with your work or >> just play around with it. Like I said, we just want to get started and are >> very able to help in this area. We see both the SQL Client and Gateway >> being very important for us and have a good team to help develop it. >> >> Let me know if there is a branch you can share, etc. It would be much >> appreciated! >> >> -Jamie Grier >> >> >> On 2022/10/28 06:06:49 Shengkai Fang wrote: >> > Hi. >> > >> > > Is there a possibility for us to get engaged and at least introduce >> > initial changes to support authentication/authorization? >> > >> > Yes. You can write a FLIP about the design and change. We can discuss this >> > in the dev mail. If the FLIP passes, we can develop it together. >> > >> > > Another question about persistent Gateway: did you have any specific >> > thoughts about it or some draft design? >> > >> > We don't have any detailed plan about this. But I know Livy has a similar >> > feature. >> > >> > Best, >> > Shengkai >> > >> > >> > Alexey Leonov-Vendrovskiy <vendrov...@gmail.com >> > <mailto:vendrov...@gmail.com>> 于2022年10月27日周四 15:12写道： >> > >> > > Apologies from the delayed response on my side. >> > > >> > > I think the authentication module is not part of our plan in 1.17 >> > > because >> > >> of the busy work. I think we'll start the design at the end of the >> > >> release-1.17. >> > > >> > > >> > > Is there a possibility for us to get engaged and at least introduce >> > > initial changes to support authentication/authorization? Specifically, >> > > changes in the API and in SQL Client. >> > > >> > > We expect the following authentication flow: >> > > >> > > On the SQL gateway we want to be able to use a delegation token. >> > > SQL client should be able to supply an API key. >> > > The SQL Gateway *would not *be submitting jobs on behalf of the client. >> > > >> > > Ideally it would be nice to introduce some interfaces in the SQL Gateway >> > > that would allow implementing custom authentication and authorization. >> > > >> > > Another question about persistent Gateway: did you have any specific >> > > thoughts about it or some draft design? >> > > >> > > Thanks, >> > > Alexey >> > > >> > > >> > > On Fri, Oct 21, 2022 at 1:13 AM Shengkai Fang <fskm...@gmail.com >> > > <mailto:fskm...@gmail.com>> wrote: >> > > >> > >> Sorry for the late response. >> > >> >> > >> In the next version(Flink 1.17), we plan to support the SQL Client to >> > >> submit the statement to the Flink SQL Gateway. The FLINK-29486 >> > >> <https://issues.apache.org/jira/browse/FLINK-29486> is the first step to >> > >> remove the usage of the `Parser` in the client side, which needs to read >> > >> the table schema during the converting sql node to operation. I think >> > >> the authentication >> > >> module is not part of our plan in 1.17 because of the busy work. I think >> > >> we'll start the design at the end of the release-1.17. >> > >> But could you share more details about the requirements of the >> > >> authentication? >> > >> - Do you use the kerberos or delegation token or password to do the >> > >> authentication? >> > >> - After the authentication, do you need the sql gateway to submit the >> > >> job on behalf of the client? >> > >> - ... >> > >> >> > >> For detailed implementation, I think Hive and Presto are good examples >> > >> to >> > >> dig in. If you have some thoughts about the authentication module, >> > >> please let me know. >> > >> >> > >> Best, >> > >> Shengkai >> > >> >> > >> Alexey Leonov-Vendrovskiy <vendrov...@gmail.com >> > >> <mailto:vendrov...@gmail.com>> 于2022年10月19日周三 00:37写道： >> > >> >> > >>> Thank you for the response, Yuxia! >> > >>> >> > >>> Shengkai, I would like to learn more about nearest and a bit more >> > >>> distant plans about development of the SQL Gateway and the SQL Client. >> > >>> Do you have a description of the work planned or maybe can share >> > >>> general >> > >>> thoughts about the Authentication module, or Persistent Gateway. >> > >>> How can the authentication part be addressed on the SQL Client side? >> > >>> >> > >>> Regards, >> > >>> -Alexey >> > >>> >> > >>> >> > >>> On Wed, Oct 12, 2022 at 11:24 PM yuxia <luoyu...@alumni.sjtu.edu.cn >> > >>> <mailto:luoyu...@alumni.sjtu.edu.cn>> >> > >>> wrote: >> > >>> >> > >>>> > In what Flink’s release the connection from SQL Client to the >> > >>>> > Gateway >> > >>>> is >> > >>>> expected to be added? >> > >>>> Flink 1.17 >> > >>>> >> > >>>> > “Authentication module” (2) and “Persistent Gateway” (4) as >> > >>>> possible future work. Were there any recent discussions on these >> > >>>> subjects? >> > >>>> No recent discussions on these subjects, but I think it'll come in >> > >>>> Flink 1.17 >> > >>>> >> > >>>> > Another related topic: are there ideas around making SQL Gateway a >> > >>>> multi-tenant >> > >>>> component? >> > >>>> Yes. >> > >>>> >> > >>>> Shengkaiis the maintainer of SQL Client and SQL gateway, maybe he can >> > >>>> provide more information. >> > >>>> >> > >>>> >> > >>>> >> > >>>> Best regards, >> > >>>> Yuxia >> > >>>> >> > >>>> ----- 原始邮件 ----- >> > >>>> 发件人: "Alexey Leonov-Vendrovskiy" <vendrov...@gmail.com >> > >>>> <mailto:vendrov...@gmail.com>> >> > >>>> 收件人: "dev" <dev@flink.apache.org <mailto:dev@flink.apache.org>> >> > >>>> 发送时间: 星期四, 2022年 10 月 13日 下午 12:33:08 >> > >>>> 主题: SQL Gateway and SQL Client >> > >>>> >> > >>>> Hi all, >> > >>>> >> > >>>> I’m Alexey from Confluent. This is my first email in this discussion >> > >>>> list. >> > >>>> I’m rather new to Flink, and to local customs of communication. I want >> > >>>> to >> > >>>> dive deeper and hopefully get more involved over time. >> > >>>> >> > >>>> Currently I have a few questions around SQL Gateway and SQL Client. >> > >>>> Specifically I wanted to learn what is the vision around the nearest >> > >>>> future >> > >>>> of these two components. >> > >>>> >> > >>>> In what Flink’s release the connection from SQL Client to the Gateway >> > >>>> is >> > >>>> expected to be added? I was looking at >> > >>>> https://issues.apache.org/jira/browse/FLINK-29486, and recently it got >> > >>>> renamed from “Enable SQL Client to Connect SQL Gateway in Remote Mode” >> > >>>> to >> > >>>> “Introduce Client Parser to get statement type”. I did some search, >> > >>>> but >> > >>>> didn’t find a good place where the client's work in this direction is >> > >>>> discussed or tracked. >> > >>>> >> > >>>> A couple questions about the SQL Gateway. The FLIP-91 >> > >>>> < >> > >>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Gateway#FLIP91:SupportSQLGateway-Futurework >> > >>>> > >> > >>>> mentions “Authentication module” (2) and “Persistent Gateway” (4) as >> > >>>> possible future work. Were there any recent discussions on these >> > >>>> subjects? >> > >>>> Or maybe there are some ideas how to move these directions forward? >> > >>>> Another >> > >>>> related topic: are there ideas around making SQL Gateway a >> > >>>> multi-tenant >> > >>>> component? >> > >>>> >> > >>>> Thank you, >> > >>>> >> > >>>> Alexey >> > >>>> >> > >>> >> >
