Hi, all. Sorry for the late reply. > Is the gateway mode planned to be supported for SQL Client in 1.17? > Do you have anything you can already share so we can start with your work or just play around with it.
Yes. @yzl is working on it and he will list the implementation plan later and share the progress. I think the change is not very large and I think it's not a big problem to finish this in the release-1.17. I will join to develop this in the mid of November. Best, Shengkai Jamie Grier <jgr...@apache.org> 于2022年11月5日周六 00:48写道： > Hi Shengkai, > > We're doing more and more Flink development at Confluent these days and > we're currently trying to bootstrap a prototype that relies on the SQL > Client and Gateway. We will be using the the components in some of our > projects and would like to co-develop them with you and the rest of the > Flink community. > > As of right now it's a pretty big blocker for our upcoming milestone that > the SQL Client has not yet been modified to talk to the SQL Gateway and we > want to help with this effort ASAP! We would be even willing to take over > the work if it's not yet started but I suspect it already is. > > Anyway, rather than start working immediately on the SQL Client and adding > a the new Gateway mode ourselves we wanted to start a conversation with you > and see where you're at with things and offer to help. > > Do you have anything you can already share so we can start with your work > or just play around with it. Like I said, we just want to get started and > are very able to help in this area. We see both the SQL Client and Gateway > being very important for us and have a good team to help develop it. > > Let me know if there is a branch you can share, etc. It would be much > appreciated! > > -Jamie Grier > > > On 2022/10/28 06:06:49 Shengkai Fang wrote: > > Hi. > > > > > Is there a possibility for us to get engaged and at least introduce > > initial changes to support authentication/authorization? > > > > Yes. You can write a FLIP about the design and change. We can discuss > this > > in the dev mail. If the FLIP passes, we can develop it together. > > > > > Another question about persistent Gateway: did you have any specific > > thoughts about it or some draft design? > > > > We don't have any detailed plan about this. But I know Livy has a similar > > feature. > > > > Best, > > Shengkai > > > > > > Alexey Leonov-Vendrovskiy <vendrov...@gmail.com> 于2022年10月27日周四 15:12写道： > > > > > Apologies from the delayed response on my side. > > > > > > I think the authentication module is not part of our plan in 1.17 > because > > >> of the busy work. I think we'll start the design at the end of the > > >> release-1.17. > > > > > > > > > Is there a possibility for us to get engaged and at least introduce > > > initial changes to support authentication/authorization? Specifically, > > > changes in the API and in SQL Client. > > > > > > We expect the following authentication flow: > > > > > > On the SQL gateway we want to be able to use a delegation token. > > > SQL client should be able to supply an API key. > > > The SQL Gateway *would not *be submitting jobs on behalf of the client. > > > > > > Ideally it would be nice to introduce some interfaces in the SQL > Gateway > > > that would allow implementing custom authentication and authorization. > > > > > > Another question about persistent Gateway: did you have any specific > > > thoughts about it or some draft design? > > > > > > Thanks, > > > Alexey > > > > > > > > > On Fri, Oct 21, 2022 at 1:13 AM Shengkai Fang <fskm...@gmail.com> > wrote: > > > > > >> Sorry for the late response. > > >> > > >> In the next version(Flink 1.17), we plan to support the SQL Client to > > >> submit the statement to the Flink SQL Gateway. The FLINK-29486 > > >> <https://issues.apache.org/jira/browse/FLINK-29486> is the first > step to > > >> remove the usage of the `Parser` in the client side, which needs to > read > > >> the table schema during the converting sql node to operation. I think > the authentication > > >> module is not part of our plan in 1.17 because of the busy work. I > think > > >> we'll start the design at the end of the release-1.17. > > >> But could you share more details about the requirements of the > > >> authentication? > > >> - Do you use the kerberos or delegation token or password to do the > > >> authentication? > > >> - After the authentication, do you need the sql gateway to submit the > > >> job on behalf of the client? > > >> - ... > > >> > > >> For detailed implementation, I think Hive and Presto are good > examples to > > >> dig in. If you have some thoughts about the authentication module, > > >> please let me know. > > >> > > >> Best, > > >> Shengkai > > >> > > >> Alexey Leonov-Vendrovskiy <vendrov...@gmail.com> 于2022年10月19日周三 > 00:37写道： > > >> > > >>> Thank you for the response, Yuxia! > > >>> > > >>> Shengkai, I would like to learn more about nearest and a bit more > > >>> distant plans about development of the SQL Gateway and the SQL > Client. > > >>> Do you have a description of the work planned or maybe can share > general > > >>> thoughts about the Authentication module, or Persistent Gateway. > > >>> How can the authentication part be addressed on the SQL Client side? > > >>> > > >>> Regards, > > >>> -Alexey > > >>> > > >>> > > >>> On Wed, Oct 12, 2022 at 11:24 PM yuxia <luoyu...@alumni.sjtu.edu.cn> > > >>> wrote: > > >>> > > >>>> > In what Flink’s release the connection from SQL Client to the > Gateway > > >>>> is > > >>>> expected to be added? > > >>>> Flink 1.17 > > >>>> > > >>>> > “Authentication module” (2) and “Persistent Gateway” (4) as > > >>>> possible future work. Were there any recent discussions on these > > >>>> subjects? > > >>>> No recent discussions on these subjects, but I think it'll come in > > >>>> Flink 1.17 > > >>>> > > >>>> > Another related topic: are there ideas around making SQL Gateway a > > >>>> multi-tenant > > >>>> component? > > >>>> Yes. > > >>>> > > >>>> Shengkaiis the maintainer of SQL Client and SQL gateway, maybe he > can > > >>>> provide more information. > > >>>> > > >>>> > > >>>> > > >>>> Best regards, > > >>>> Yuxia > > >>>> > > >>>> ----- 原始邮件 ----- > > >>>> 发件人: "Alexey Leonov-Vendrovskiy" <vendrov...@gmail.com> > > >>>> 收件人: "dev" <dev@flink.apache.org> > > >>>> 发送时间: 星期四, 2022年 10 月 13日 下午 12:33:08 > > >>>> 主题: SQL Gateway and SQL Client > > >>>> > > >>>> Hi all, > > >>>> > > >>>> I’m Alexey from Confluent. This is my first email in this discussion > > >>>> list. > > >>>> I’m rather new to Flink, and to local customs of communication. I > want > > >>>> to > > >>>> dive deeper and hopefully get more involved over time. > > >>>> > > >>>> Currently I have a few questions around SQL Gateway and SQL Client. > > >>>> Specifically I wanted to learn what is the vision around the nearest > > >>>> future > > >>>> of these two components. > > >>>> > > >>>> In what Flink’s release the connection from SQL Client to the > Gateway is > > >>>> expected to be added? I was looking at > > >>>> https://issues.apache.org/jira/browse/FLINK-29486, and recently it > got > > >>>> renamed from “Enable SQL Client to Connect SQL Gateway in Remote > Mode” > > >>>> to > > >>>> “Introduce Client Parser to get statement type”. I did some > search, but > > >>>> didn’t find a good place where the client's work in this direction > is > > >>>> discussed or tracked. > > >>>> > > >>>> A couple questions about the SQL Gateway. The FLIP-91 > > >>>> < > > >>>> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Gateway#FLIP91:SupportSQLGateway-Futurework > > >>>> > > > >>>> mentions “Authentication module” (2) and “Persistent Gateway” (4) as > > >>>> possible future work. Were there any recent discussions on these > > >>>> subjects? > > >>>> Or maybe there are some ideas how to move these directions forward? > > >>>> Another > > >>>> related topic: are there ideas around making SQL Gateway a > multi-tenant > > >>>> component? > > >>>> > > >>>> Thank you, > > >>>> > > >>>> Alexey > > >>>> > > >>> > > >
