Small correction: I am *not *saying we should not do this, perhaps this is the best solution to finding a good compromise here, but I am trying to discover + acknowledge the full implications of this proposal so they can be discussed.
Sorry :) On Wed, Jun 30, 2021 at 9:53 AM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hi Gabor, > > Thanks for your answers. I appreciate the explanations. Please see my > responses + further questions below. > > > * What stability semantics do you envision for this API? >>> >> As I foresee the API will be as stable as Netty API. Since there is >> guarantee on no breaking changes between minor versions we can give the >> same guarantee. >> If for whatever reason we need to break it we can do it in major version >> like every other open source project does. >> > > * Does Flink expose dependencies’ APIs in other places? Since this exposes >>> the Netty API, will this make it difficult to upgrade Netty? >>> >> I don't expect breaking changes between minor versions so such cases >> there will be no issues. If there is a breaking change in major version >> we need to wait Flink major version too. >> > > To clarify, you are proposing this new API to have the same stability > guarantees as @Public currently does? Where we will not introduce breaking > changes unless absolutely necessary (and requiring a FLIP, etc.)? > > If this is the case, I think this puts the community in a tough position > where we are forced to maintain compatibility with something that we do not > have control over. It also locks Flink into the current major version of > Netty (and the Netty framework itself) for the foreseeable future. > > I am saying we should not do this, perhaps this is the best solution to > finding a good compromise here, but I am trying to discover + acknowledge > the full implications of this proposal so they can be discussed. > > What do you think about marking this API as @Experimental and not > guaranteeing stability between versions? Then, if we do decide we need to > upgrade Netty (or move away from it), we can do so. > > * I share Till's concern about multiple factories – other HTTP middleware >>> frameworks commonly support chaining middlewares. Since the proposed API >>> does not include these features/guarantee ordering, do you see any reason >>> to allow more than one factory? >>> >> I personally can't come up with a use-case where ordering is a must. I'm >> not telling that this is not a valid use-case but adding a feature w/o >> business rationale would include the maintenance cost (though I'm open to >> add). >> As I've seen Till also can't give example for that (please see the doc >> comments). If you have anything in mind please share it and we can add >> priority to the API. >> There is another option too, namely we can be defensive and we can add >> the priority right now. I would do this only if everybody states in mail >> that it would be the best option, >> otherwise I would stick to the original plan. >> > > Let me try to come up with a use case: > * Someone creates an authentication module for integrating with Google's > OAuth and publishes it to flink-packages > * Another person in another org wants to use Google OAuth and then add > internal authorization based on the user > * In this scenario, *Google OAuth must come before the internal > authorization* > * They place their module and the Google OAuth module to be picked up by > the service loader > * What happens? > > I do not think that the current proposal has a way to handle this, besides > having the implementor of the internal authorization module bundle > everything into one, as you have suggested. Since this is the only way to > achieve order, why not restrict the service loader to only allow one? This > way the API is explicit in what it supports. > > > Let me know what you think, > Austin > > > On Wed, Jun 30, 2021 at 5:24 AM Gabor Somogyi <gabor.g.somo...@gmail.com> > wrote: > >> Hi Austin, >> >> Please see my answers embedded down below. >> >> BR, >> G >> >> >> >> On Tue, Jun 29, 2021 at 9:59 PM Austin Cawley-Edwards < >> austin.caw...@gmail.com> wrote: >> >>> Hi all, >>> >>> Thanks for the updated proposal. I have a few questions about the API, >>> please see below. >>> >>> * What stability semantics do you envision for this API? >>> >> As I foresee the API will be as stable as Netty API. Since there is >> guarantee on no breaking changes between minor versions we can give the >> same guarantee. >> If for whatever reason we need to break it we can do it in major version >> like every other open source project does. >> >> >>> * Does Flink expose dependencies’ APIs in other places? Since this >>> exposes the Netty API, will this make it difficult to upgrade Netty? >>> >> I don't expect breaking changes between minor versions so such cases >> there will be no issues. If there is a breaking change in major version >> we need to wait Flink major version too. >> >> >>> * I share Till's concern about multiple factories – other HTTP >>> middleware frameworks commonly support chaining middlewares. Since the >>> proposed API does not include these features/guarantee ordering, do you see >>> any reason to allow more than one factory? >>> >> I personally can't come up with a use-case where ordering is a must. I'm >> not telling that this is not a valid use-case but adding a feature w/o >> business rationale would include the maintenance cost (though I'm open to >> add). >> As I've seen Till also can't give example for that (please see the doc >> comments). If you have anything in mind please share it and we can add >> priority to the API. >> There is another option too, namely we can be defensive and we can add >> the priority right now. I would do this only if everybody states in mail >> that it would be the best option, >> otherwise I would stick to the original plan. >> >> >>> >>> Best, >>> Austin >>> >>> On Tue, Jun 29, 2021 at 8:55 AM Márton Balassi <balassi.mar...@gmail.com> >>> wrote: >>> >>>> Hi all, >>>> >>>> I commend Konstantin and Till when it comes to standing up for the >>>> community values. >>>> >>>> Based on your feedback we are withdrawing the original proposal and >>>> attaching a more general custom netty handler API proposal [1] written >>>> by >>>> G. The change necessary to the Flink repository is approximately 500 >>>> lines >>>> of code. [2] >>>> >>>> Please let us focus on discussing the details of this API and whether it >>>> covers the necessary use cases. >>>> >>>> [1] >>>> >>>> https://docs.google.com/document/d/1Idnw8YauMK1x_14iv0rVF0Hqm58J6Dg-hi-hEuL6hwM/edit#heading=h.ijcbce3c5gip >>>> [2] >>>> >>>> https://github.com/gaborgsomogyi/flink/commit/942f23679ac21428bb87fc85557b9b443fcaf310 >>>> >>>> Thanks, >>>> Marton >>>> >>>> On Wed, Jun 23, 2021 at 9:36 PM Austin Cawley-Edwards < >>>> austin.caw...@gmail.com> wrote: >>>> >>>> > Hi all, >>>> > >>>> > Thanks, Konstantin and Till, for guiding the discussion. >>>> > >>>> > I was not aware of the results of the call with Konstantin and was >>>> > attempting to resolve the unanswered questions before more, >>>> potentially >>>> > fruitless, work was done. >>>> > >>>> > I am also looking forward to the coming proposal, as well as >>>> increasing my >>>> > understanding of this specific use case + its limitations! >>>> > >>>> > Best, >>>> > Austin >>>> > >>>> > On Tue, Jun 22, 2021 at 6:32 AM Till Rohrmann <trohrm...@apache.org> >>>> > wrote: >>>> > >>>> > > Hi everyone, >>>> > > >>>> > > I do like the idea of keeping the actual change outside of Flink >>>> but to >>>> > > enable Flink to support such a use case (different authentication >>>> > > mechanisms). I think this is a good compromise for the community >>>> that >>>> > > combines long-term maintainability with support for new use-cases. >>>> I am >>>> > > looking forward to your proposal. >>>> > > >>>> > > I also want to second Konstantin here that the tone of your last >>>> email, >>>> > > Marton, does not reflect the values and manners of the Flink >>>> community >>>> > and >>>> > > is not representative of how we conduct discussions. Especially, >>>> the more >>>> > > senior community members should know this and act accordingly in >>>> order to >>>> > > be good role models for others in the community. Technical >>>> discussions >>>> > > should not be decided by who wields presumably the greatest >>>> authority but >>>> > > by the soundness of arguments and by what is the best solution for a >>>> > > problem. >>>> > > >>>> > > Let us now try to find the best solution for the problem at hand! >>>> > > >>>> > > Cheers, >>>> > > Till >>>> > > >>>> > > On Tue, Jun 22, 2021 at 11:24 AM Konstantin Knauf < >>>> kna...@apache.org> >>>> > > wrote: >>>> > > >>>> > > > Hi everyone, >>>> > > > >>>> > > > First, Marton and I had a brief conversation yesterday offline and >>>> > > > discussed exploring the approach of exposing the authentication >>>> > > > functionality via an API. So, I am looking forward to your >>>> proposal in >>>> > > that >>>> > > > direction. The benefit of such a solution would be that it is >>>> > extensible >>>> > > > for others and it does add a smaller maintenance (in particular >>>> > testing) >>>> > > > footprint to Apache Flink itself. If we end up going down this >>>> route, >>>> > > > flink-packages.org would be a great way to promote these third >>>> party >>>> > > > "authentication modules". >>>> > > > >>>> > > > Second, Marton, I understand your frustration about the long >>>> discussion >>>> > > on >>>> > > > this "simple matter", but the condescending tone of your last mail >>>> > feels >>>> > > > uncalled for to me. Austin expressed a valid opinion on the topic, >>>> > which >>>> > > is >>>> > > > based on his experience from other Open Source frameworks (CNCF >>>> > mostly). >>>> > > I >>>> > > > am sure you agree that it is important for Apache Flink to stay >>>> open >>>> > and >>>> > > to >>>> > > > consider different approaches and ideas and I don't think it >>>> helps the >>>> > > > culture of discussion to shoot it down like this ("This is where >>>> this >>>> > > > discussion stops."). >>>> > > > >>>> > > > Let's continue to move this discussion forward and I am sure we'll >>>> > find a >>>> > > > consensus based on product and technological considerations. >>>> > > > >>>> > > > Thanks, >>>> > > > >>>> > > > Konstantin >>>> > > > >>>> > > > On Tue, Jun 22, 2021 at 9:31 AM Márton Balassi < >>>> > balassi.mar...@gmail.com >>>> > > > >>>> > > > wrote: >>>> > > > >>>> > > > > Hi Austin, >>>> > > > > >>>> > > > > Thank you for your thoughts. This is where this discussion >>>> stops. >>>> > This >>>> > > > > email thread already contains more characters than the >>>> implementation >>>> > > and >>>> > > > > what is needed for the next 20 years of maintenance. >>>> > > > > >>>> > > > > It is great that you have a view on modern solutions and thank >>>> you >>>> > for >>>> > > > > offering your help with brainstorming solutions. I am >>>> responsible for >>>> > > > Flink >>>> > > > > at Cloudera and we do need an implementation like this and it >>>> is in >>>> > > fact >>>> > > > > already in production at dozens of customers. We are open to >>>> adapting >>>> > > > that >>>> > > > > to expose a more generic API (and keeping Kerberos to our >>>> fork), to >>>> > > > > contribute this to the community as others have asked for it >>>> and to >>>> > > > protect >>>> > > > > ourselves from occasionally having to update this critical >>>> > > implementation >>>> > > > > path based on changes in the Apache codebase. I have worked with >>>> > close >>>> > > > to a >>>> > > > > hundred Big Data customers as a consultant and an engineering >>>> manager >>>> > > and >>>> > > > > committed hundreds of changes to Apache Flink over the past >>>> decade, >>>> > > > please >>>> > > > > trust my judgement on a simple matter like this. >>>> > > > > >>>> > > > > Please forgive me for referencing authority, this discussion was >>>> > > getting >>>> > > > > out of hand. Please keep vigilant. >>>> > > > > >>>> > > > > Best, >>>> > > > > Marton >>>> > > > > >>>> > > > > On Mon, Jun 21, 2021 at 10:50 PM Austin Cawley-Edwards < >>>> > > > > austin.caw...@gmail.com> wrote: >>>> > > > > >>>> > > > > > Hi Gabor + Marton, >>>> > > > > > >>>> > > > > > I don't believe that the issue with this proposal is the >>>> specific >>>> > > > > mechanism >>>> > > > > > proposed (Kerberos), but rather that it is not the level to >>>> > implement >>>> > > > it >>>> > > > > at >>>> > > > > > (Flink). I'm just one voice, so please take this with a grain >>>> of >>>> > > salt. >>>> > > > > > >>>> > > > > > In the other solutions previously noted there is no need to >>>> > > instrument >>>> > > > > > Flink which, in addition to reducing the maintenance burden, >>>> > > provides a >>>> > > > > > better, decoupled end result. >>>> > > > > > >>>> > > > > > IMO we should not add any new API in Flink for this use case. >>>> I >>>> > think >>>> > > > it >>>> > > > > is >>>> > > > > > unfortunate and sympathize with the work that has already >>>> been done >>>> > > on >>>> > > > > this >>>> > > > > > feature – perhaps we could brainstorm ways to run this >>>> alongside >>>> > > Flink >>>> > > > in >>>> > > > > > your setup. Again, I don't think the proposed solution of an >>>> > agnostic >>>> > > > API >>>> > > > > > would not work, nor is it a bad idea, but is not one that >>>> will make >>>> > > > Flink >>>> > > > > > more compatible with the modern solutions to this problem. >>>> > > > > > >>>> > > > > > Best, >>>> > > > > > Austin >>>> > > > > > >>>> > > > > > On Mon, Jun 21, 2021 at 2:18 PM Márton Balassi < >>>> > > > balassi.mar...@gmail.com >>>> > > > > > >>>> > > > > > wrote: >>>> > > > > > >>>> > > > > > > Hi team, >>>> > > > > > > >>>> > > > > > > Thank you for your input. Based on this discussion I agree >>>> with G >>>> > > > that >>>> > > > > > > selecting and standardizing on a specific strong >>>> authentication >>>> > > > > mechanism >>>> > > > > > > is more challenging than the whole rest of the scope of this >>>> > > > > > authentication >>>> > > > > > > story. :-) I suggest that G and I go back to the drawing >>>> board >>>> > and >>>> > > > come >>>> > > > > > up >>>> > > > > > > with an API that can support multiple authentication >>>> mechanisms, >>>> > > and >>>> > > > we >>>> > > > > > > would only merge said API to Flink. Specific >>>> implementations of >>>> > it >>>> > > > can >>>> > > > > be >>>> > > > > > > maintained outside of the project. This way we tackle the >>>> main >>>> > > > > challenge >>>> > > > > > in >>>> > > > > > > a truly minimal way. >>>> > > > > > > >>>> > > > > > > Best, >>>> > > > > > > Marton >>>> > > > > > > >>>> > > > > > > On Mon, Jun 21, 2021 at 4:18 PM Gabor Somogyi < >>>> > > > > gabor.g.somo...@gmail.com >>>> > > > > > > >>>> > > > > > > wrote: >>>> > > > > > > >>>> > > > > > > > Hi All, >>>> > > > > > > > >>>> > > > > > > > We see that adding any kind of specific authentication >>>> raises >>>> > > more >>>> > > > > > > > questions than answers. >>>> > > > > > > > What would be if a generic API would be added without any >>>> real >>>> > > > > > > > authentication logic? >>>> > > > > > > > That way every provider can add its own protocol >>>> implementation >>>> > > as >>>> > > > > > > > additional jar. >>>> > > > > > > > >>>> > > > > > > > BR, >>>> > > > > > > > G >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > > On Thu, Jun 17, 2021 at 7:53 PM Austin Cawley-Edwards < >>>> > > > > > > > austin.caw...@gmail.com> wrote: >>>> > > > > > > > >>>> > > > > > > >> Hi all, >>>> > > > > > > >> >>>> > > > > > > >> Sorry to be joining the conversation late. I'm also on >>>> the >>>> > side >>>> > > of >>>> > > > > > > >> Konstantin, generally, in that this seems to not be a >>>> core >>>> > goal >>>> > > of >>>> > > > > > Flink >>>> > > > > > > >> as >>>> > > > > > > >> a project and adds a maintenance burden. >>>> > > > > > > >> >>>> > > > > > > >> Would another con of Kerberos be that is likely a fading >>>> > project >>>> > > > in >>>> > > > > > > terms >>>> > > > > > > >> of network security? (serious question, please correct >>>> me if >>>> > > there >>>> > > > > is >>>> > > > > > > >> reason to believe it is gaining adoption) >>>> > > > > > > >> >>>> > > > > > > >> The point about Kerberos being independent of >>>> infrastructure >>>> > is >>>> > > a >>>> > > > > good >>>> > > > > > > one >>>> > > > > > > >> but is something that is also solved by modern sidecar >>>> > proxies + >>>> > > > > > service >>>> > > > > > > >> meshes that can run across Kubernetes and bare-metal. >>>> These >>>> > > > > solutions >>>> > > > > > > also >>>> > > > > > > >> handle certificate provisioning, rotation, etc. in >>>> addition to >>>> > > > > > > >> higher-level >>>> > > > > > > >> authorization policies. Some examples of projects with >>>> this >>>> > > > > "universal >>>> > > > > > > >> infrastructure support" are Kuma[1] (CNCF Sandbox, I'm a >>>> > > > maintainer) >>>> > > > > > and >>>> > > > > > > >> Istio[2] (Google). >>>> > > > > > > >> >>>> > > > > > > >> Wondering out loud: has anyone tried to run Flink on top >>>> of >>>> > > > > cilium[3], >>>> > > > > > > >> which also provides zero-trust networking at the kernel >>>> level >>>> > > > > without >>>> > > > > > > >> needing to instrument applications? This currently only >>>> runs >>>> > on >>>> > > > > > > Kubernetes >>>> > > > > > > >> on Linux, so that's a major limitation, but solves many >>>> of the >>>> > > > > request >>>> > > > > > > >> forging concerns at all levels. >>>> > > > > > > >> >>>> > > > > > > >> Thanks, >>>> > > > > > > >> Austin >>>> > > > > > > >> >>>> > > > > > > >> [1]: https://kuma.io/docs/1.1.6/quickstart/universal/ >>>> > > > > > > >> [2]: >>>> > > https://istio.io/latest/docs/setup/install/virtual-machine/ >>>> > > > > > > >> [3]: https://cilium.io/ >>>> > > > > > > >> >>>> > > > > > > >> On Thu, Jun 17, 2021 at 1:50 PM Till Rohrmann < >>>> > > > trohrm...@apache.org >>>> > > > > > >>>> > > > > > > >> wrote: >>>> > > > > > > >> >>>> > > > > > > >> > I left some comments in the Google document. It would >>>> be >>>> > great >>>> > > > if >>>> > > > > > > >> > someone from the community with security experience >>>> could >>>> > also >>>> > > > > take >>>> > > > > > a >>>> > > > > > > >> look >>>> > > > > > > >> > at it. Maybe Eron you have an opinion on the topic. >>>> > > > > > > >> > >>>> > > > > > > >> > Cheers, >>>> > > > > > > >> > Till >>>> > > > > > > >> > >>>> > > > > > > >> > On Thu, Jun 17, 2021 at 6:57 PM Till Rohrmann < >>>> > > > > trohrm...@apache.org >>>> > > > > > > >>>> > > > > > > >> > wrote: >>>> > > > > > > >> > >>>> > > > > > > >> > > Hi Gabor, >>>> > > > > > > >> > > >>>> > > > > > > >> > > I haven't found time to look into the updated FLIP >>>> yet. >>>> > I'll >>>> > > > try >>>> > > > > > to >>>> > > > > > > >> do it >>>> > > > > > > >> > > asap. >>>> > > > > > > >> > > >>>> > > > > > > >> > > Cheers, >>>> > > > > > > >> > > Till >>>> > > > > > > >> > > >>>> > > > > > > >> > > On Wed, Jun 16, 2021 at 9:35 PM Konstantin Knauf < >>>> > > > > > kna...@apache.org >>>> > > > > > > > >>>> > > > > > > >> > > wrote: >>>> > > > > > > >> > > >>>> > > > > > > >> > >> Hi Gabor, >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> > However representing Kerberos as completely new >>>> feature >>>> > > is >>>> > > > > not >>>> > > > > > > true >>>> > > > > > > >> > >> because >>>> > > > > > > >> > >> it's already in since Flink makes authentication at >>>> least >>>> > > > with >>>> > > > > > HDFS >>>> > > > > > > >> and >>>> > > > > > > >> > >> Hbase through Kerberos. >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> True, that is one way to look at it, but there are >>>> > > > differences, >>>> > > > > > > too: >>>> > > > > > > >> > >> Control Plane vs Data Plane, Core vs Connectors. >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> > Adding OIDC or OAuth2 has the exact same concerns >>>> what >>>> > > > you've >>>> > > > > > > guys >>>> > > > > > > >> > just >>>> > > > > > > >> > >> raised. Why exactly these? If you think this would >>>> be >>>> > > > > beneficial >>>> > > > > > we >>>> > > > > > > >> can >>>> > > > > > > >> > >> discuss it in detail >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> That's exactly my point. Once we start adding authx >>>> > > support, >>>> > > > we >>>> > > > > > > will >>>> > > > > > > >> > >> sooner or later discuss other options besides >>>> Kerberos, >>>> > > too. >>>> > > > A >>>> > > > > > user >>>> > > > > > > >> who >>>> > > > > > > >> > >> would like to use OAuth can not easily use Kerberos, >>>> > right? >>>> > > > > > > >> > >> That is one of the reasons I am skeptical about >>>> adding >>>> > > > initial >>>> > > > > > > authx >>>> > > > > > > >> > >> support. >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> > Related authorization you've mentioned it can be >>>> > > > complicated >>>> > > > > > over >>>> > > > > > > >> > time. >>>> > > > > > > >> > >> Can >>>> > > > > > > >> > >> you show us an example? We've knowledge with couple >>>> of >>>> > open >>>> > > > > > source >>>> > > > > > > >> > >> components >>>> > > > > > > >> > >> but authorization was never a horror complex story. >>>> I >>>> > > > > personally >>>> > > > > > > have >>>> > > > > > > >> > the >>>> > > > > > > >> > >> most experience with Spark which I think is quite >>>> simple >>>> > > and >>>> > > > > > > stable. >>>> > > > > > > >> > Users >>>> > > > > > > >> > >> can be viewers/admins >>>> > > > > > > >> > >> and jobs started by others can't be modified. If >>>> you can >>>> > > > share >>>> > > > > an >>>> > > > > > > >> > example >>>> > > > > > > >> > >> over-complication we can discuss on facts. >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> Authorization is a new aspect that needs to be >>>> considered >>>> > > for >>>> > > > > > every >>>> > > > > > > >> > >> addition to the REST API. In the future users might >>>> ask >>>> > for >>>> > > > > > > >> additional >>>> > > > > > > >> > >> roles (e.g. an editor), user-defined roles and >>>> you've >>>> > > already >>>> > > > > > > >> mentioned >>>> > > > > > > >> > >> job-level permissions yourself. And keep in mind >>>> that >>>> > there >>>> > > > > might >>>> > > > > > > >> also >>>> > > > > > > >> > be >>>> > > > > > > >> > >> larger additions in the future like the >>>> > flink-sql-gateway. >>>> > > > > > > >> Contributions >>>> > > > > > > >> > >> like this become more expensive the more aspects we >>>> need >>>> > to >>>> > > > > > > consider. >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> In general, I believe, it is important that the >>>> community >>>> > > > > focuses >>>> > > > > > > its >>>> > > > > > > >> > >> efforts where we can generate the most value to the >>>> user >>>> > > and >>>> > > > - >>>> > > > > > > >> > personally - >>>> > > > > > > >> > >> I don't think there is much to gain by extending >>>> Flink's >>>> > > > scope >>>> > > > > in >>>> > > > > > > >> that >>>> > > > > > > >> > >> direction. Of course, this is not black and white >>>> and >>>> > there >>>> > > > are >>>> > > > > > > other >>>> > > > > > > >> > valid >>>> > > > > > > >> > >> opinions. >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> Thanks, >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> Konstantin >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> On Wed, Jun 16, 2021 at 7:38 PM Gabor Somogyi < >>>> > > > > > > >> > gabor.g.somo...@gmail.com> >>>> > > > > > > >> > >> wrote: >>>> > > > > > > >> > >> >>>> > > > > > > >> > >>> Hi Konstantin, >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >>> Thanks for the response. Related new feature >>>> > introduction >>>> > > in >>>> > > > > > case >>>> > > > > > > of >>>> > > > > > > >> > >>> Basic >>>> > > > > > > >> > >>> auth I tend to agree, anything else can be chosen. >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >>> However representing Kerberos as completely new >>>> feature >>>> > is >>>> > > > not >>>> > > > > > > true >>>> > > > > > > >> > >>> because >>>> > > > > > > >> > >>> it's already in since Flink makes authentication at >>>> > least >>>> > > > with >>>> > > > > > > HDFS >>>> > > > > > > >> and >>>> > > > > > > >> > >>> Hbase through Kerberos. >>>> > > > > > > >> > >>> The main problem with the actual Kerberos >>>> implementation >>>> > > is >>>> > > > > that >>>> > > > > > > it >>>> > > > > > > >> > >>> contains several bugs and only partially >>>> implemented. >>>> > > > > Following >>>> > > > > > > your >>>> > > > > > > >> > >>> suggestion can we agree that we >>>> > > > > > > >> > >>> skip the Basic auth implementation and finish an >>>> already >>>> > > > > started >>>> > > > > > > >> > Kerberos >>>> > > > > > > >> > >>> story by adding History Server and Job Dashboard >>>> > > > > authentication? >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >>> Adding OIDC or OAuth2 has the exact same concerns >>>> what >>>> > > > you've >>>> > > > > > guys >>>> > > > > > > >> just >>>> > > > > > > >> > >>> raised. Why exactly these? If you think this would >>>> be >>>> > > > > beneficial >>>> > > > > > > we >>>> > > > > > > >> can >>>> > > > > > > >> > >>> discuss it in detail >>>> > > > > > > >> > >>> but as a side story it would be good to finish a >>>> halfway >>>> > > > done >>>> > > > > > > >> Kerberos >>>> > > > > > > >> > >>> story. >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >>> Related authorization you've mentioned it can be >>>> > > complicated >>>> > > > > > over >>>> > > > > > > >> time. >>>> > > > > > > >> > >>> Can >>>> > > > > > > >> > >>> you show us an example? We've knowledge with >>>> couple of >>>> > > open >>>> > > > > > source >>>> > > > > > > >> > >>> components >>>> > > > > > > >> > >>> but authorization was never a horror complex >>>> story. I >>>> > > > > personally >>>> > > > > > > >> have >>>> > > > > > > >> > the >>>> > > > > > > >> > >>> most experience with Spark which I think is quite >>>> simple >>>> > > and >>>> > > > > > > stable. >>>> > > > > > > >> > >>> Users >>>> > > > > > > >> > >>> can be viewers/admins >>>> > > > > > > >> > >>> and jobs started by others can't be modified. If >>>> you can >>>> > > > share >>>> > > > > > an >>>> > > > > > > >> > example >>>> > > > > > > >> > >>> over-complication we can discuss on facts. >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >>> Thank you in advance! >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >>> BR, >>>> > > > > > > >> > >>> G >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >>> On Wed, Jun 16, 2021 at 5:42 PM Konstantin Knauf < >>>> > > > > > > kna...@apache.org >>>> > > > > > > >> > >>>> > > > > > > >> > >>> wrote: >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >>> > Hi everyone, >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > sorry for joining late and thanks for the >>>> insightful >>>> > > > > > discussion. >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > In general, I'd personally prefer not to >>>> increase the >>>> > > > > surface >>>> > > > > > > >> area of >>>> > > > > > > >> > >>> > Apache Flink unless there is a good reason. It >>>> seems >>>> > we >>>> > > > all >>>> > > > > > > agree >>>> > > > > > > >> > that >>>> > > > > > > >> > >>> > authx is not part of the core value proposition >>>> of >>>> > > Apache >>>> > > > > > Flink, >>>> > > > > > > >> so >>>> > > > > > > >> > if >>>> > > > > > > >> > >>> we >>>> > > > > > > >> > >>> > can delegate this problem to a more specialized >>>> tool, >>>> > I >>>> > > am >>>> > > > > in >>>> > > > > > > >> favor >>>> > > > > > > >> > of >>>> > > > > > > >> > >>> > that. Apache Flink is already huge and a lot of >>>> work >>>> > > goes >>>> > > > > into >>>> > > > > > > >> > >>> maintenance, >>>> > > > > > > >> > >>> > so I personally have become more sensitive to >>>> this >>>> > > aspect >>>> > > > > over >>>> > > > > > > >> time. >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > If we add support for Basic Auth and Kerberos >>>> now, >>>> > users >>>> > > > > will >>>> > > > > > > >> sooner >>>> > > > > > > >> > or >>>> > > > > > > >> > >>> > later ask for OIDC, LDAP, SAML,... I acknowledge >>>> that >>>> > > > > Kerberos >>>> > > > > > > is >>>> > > > > > > >> > >>> widely >>>> > > > > > > >> > >>> > used in the corporate, on-premises context, but >>>> isn't >>>> > > the >>>> > > > > > focus >>>> > > > > > > >> > moving >>>> > > > > > > >> > >>> more >>>> > > > > > > >> > >>> > towards more web-friendly standards like >>>> OIDC/OAuth >>>> > 2.0? >>>> > > > If >>>> > > > > we >>>> > > > > > > >> only >>>> > > > > > > >> > >>> want to >>>> > > > > > > >> > >>> > support a single protocol, there is an argument >>>> to be >>>> > > made >>>> > > > > > that >>>> > > > > > > it >>>> > > > > > > >> > >>> should >>>> > > > > > > >> > >>> > be OIDC and Dex [1,2] as a bridge to everything >>>> else. >>>> > > Have >>>> > > > > > OIDC >>>> > > > > > > or >>>> > > > > > > >> > >>> OAuth2 >>>> > > > > > > >> > >>> > been considered instead of Kerberos? How do you >>>> see >>>> > the >>>> > > > > market >>>> > > > > > > >> > moving? >>>> > > > > > > >> > >>> But >>>> > > > > > > >> > >>> > as I said before, in my opinion we can generate >>>> more >>>> > > value >>>> > > > > by >>>> > > > > > > >> > investing >>>> > > > > > > >> > >>> > into other areas of Apache Flink. >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > Authorization also has the potential to become >>>> more >>>> > > > > > fine-grained >>>> > > > > > > >> and >>>> > > > > > > >> > >>> > complex over time: you already mentioned >>>> restricting >>>> > the >>>> > > > > > actions >>>> > > > > > > >> > that a >>>> > > > > > > >> > >>> > specific user can do in a cluster. >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > Cheers, >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > Konstantin >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > [1] https://github.com/dexidp/dex >>>> > > > > > > >> > >>> > [2] https://github.com/dexidp/dex/issues/1903 >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > On Wed, Jun 16, 2021 at 11:44 AM Gabor Somogyi < >>>> > > > > > > >> > >>> gabor.g.somo...@gmail.com> >>>> > > > > > > >> > >>> > wrote: >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> >> Hi Till, >>>> > > > > > > >> > >>> >> >>>> > > > > > > >> > >>> >> Did you have the chance to take a look at the >>>> doc? >>>> > Not >>>> > > > yet >>>> > > > > > seen >>>> > > > > > > >> any >>>> > > > > > > >> > >>> >> update. >>>> > > > > > > >> > >>> >> >>>> > > > > > > >> > >>> >> BR, >>>> > > > > > > >> > >>> >> G >>>> > > > > > > >> > >>> >> >>>> > > > > > > >> > >>> >> >>>> > > > > > > >> > >>> >> On Wed, Jun 9, 2021 at 1:43 PM Till Rohrmann < >>>> > > > > > > >> trohrm...@apache.org> >>>> > > > > > > >> > >>> >> wrote: >>>> > > > > > > >> > >>> >> >>>> > > > > > > >> > >>> >> > Thanks for the update Gabor. I'll take a look >>>> and >>>> > > > respond >>>> > > > > > in >>>> > > > > > > >> the >>>> > > > > > > >> > >>> >> document. >>>> > > > > > > >> > >>> >> > >>>> > > > > > > >> > >>> >> > Cheers, >>>> > > > > > > >> > >>> >> > Till >>>> > > > > > > >> > >>> >> > >>>> > > > > > > >> > >>> >> > On Wed, Jun 9, 2021 at 12:59 PM Gabor Somogyi >>>> < >>>> > > > > > > >> > >>> >> gabor.g.somo...@gmail.com> >>>> > > > > > > >> > >>> >> > wrote: >>>> > > > > > > >> > >>> >> > >>>> > > > > > > >> > >>> >> >> Hi Till, >>>> > > > > > > >> > >>> >> >> >>>> > > > > > > >> > >>> >> >> Your proxy suggestion has been considered >>>> in-depth >>>> > > and >>>> > > > > > > updated >>>> > > > > > > >> > the >>>> > > > > > > >> > >>> FLIP >>>> > > > > > > >> > >>> >> >> accordingly. >>>> > > > > > > >> > >>> >> >> We've considered 2 proxy implementation >>>> (Nginx and >>>> > > > > Squid) >>>> > > > > > > but >>>> > > > > > > >> > >>> according >>>> > > > > > > >> > >>> >> >> to our analysis and testing it's not >>>> suitable for >>>> > > the >>>> > > > > > > >> mentioned >>>> > > > > > > >> > >>> >> use-cases. >>>> > > > > > > >> > >>> >> >> Please take a look at the rejected >>>> alternatives >>>> > for >>>> > > > > > detailed >>>> > > > > > > >> > >>> >> explanation. >>>> > > > > > > >> > >>> >> >> >>>> > > > > > > >> > >>> >> >> Thanks for your time in advance! >>>> > > > > > > >> > >>> >> >> >>>> > > > > > > >> > >>> >> >> BR, >>>> > > > > > > >> > >>> >> >> G >>>> > > > > > > >> > >>> >> >> >>>> > > > > > > >> > >>> >> >> >>>> > > > > > > >> > >>> >> >> On Fri, Jun 4, 2021 at 3:31 PM Till Rohrmann >>>> < >>>> > > > > > > >> > trohrm...@apache.org >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> >> >> wrote: >>>> > > > > > > >> > >>> >> >> >>>> > > > > > > >> > >>> >> >>> As I've said I am not a security expert and >>>> > that's >>>> > > > why >>>> > > > > I >>>> > > > > > > >> have to >>>> > > > > > > >> > >>> ask >>>> > > > > > > >> > >>> >> for >>>> > > > > > > >> > >>> >> >>> clarification, Gabor. You are saying that >>>> if we >>>> > > > > > configure a >>>> > > > > > > >> > >>> >> truststore for >>>> > > > > > > >> > >>> >> >>> the REST endpoint with a single trusted >>>> > certificate >>>> > > > > which >>>> > > > > > > has >>>> > > > > > > >> > been >>>> > > > > > > >> > >>> >> >>> generated by the operator of the Flink >>>> cluster, >>>> > > then >>>> > > > > the >>>> > > > > > > >> > attacker >>>> > > > > > > >> > >>> can >>>> > > > > > > >> > >>> >> >>> generate a new certificate, sign it and >>>> then talk >>>> > > to >>>> > > > > the >>>> > > > > > > >> Flink >>>> > > > > > > >> > >>> >> cluster if >>>> > > > > > > >> > >>> >> >>> he has access to the node on which the REST >>>> > > endpoint >>>> > > > > > runs? >>>> > > > > > > My >>>> > > > > > > >> > >>> >> understanding >>>> > > > > > > >> > >>> >> >>> was that you need the corresponding private >>>> key >>>> > > which >>>> > > > > in >>>> > > > > > my >>>> > > > > > > >> > >>> proposed >>>> > > > > > > >> > >>> >> setup >>>> > > > > > > >> > >>> >> >>> would be under the control of the operator >>>> as >>>> > well >>>> > > > > (e.g. >>>> > > > > > > >> stored >>>> > > > > > > >> > >>> in a >>>> > > > > > > >> > >>> >> >>> keystore on the same machine but guarded by >>>> some >>>> > > > > secret). >>>> > > > > > > >> That >>>> > > > > > > >> > way >>>> > > > > > > >> > >>> >> (if I am >>>> > > > > > > >> > >>> >> >>> not mistaken), only the entity which has >>>> access >>>> > to >>>> > > > the >>>> > > > > > > >> keystore >>>> > > > > > > >> > is >>>> > > > > > > >> > >>> >> able to >>>> > > > > > > >> > >>> >> >>> talk to the Flink cluster. >>>> > > > > > > >> > >>> >> >>> >>>> > > > > > > >> > >>> >> >>> Maybe we are also getting our wires crossed >>>> here >>>> > > and >>>> > > > > are >>>> > > > > > > >> talking >>>> > > > > > > >> > >>> about >>>> > > > > > > >> > >>> >> >>> different things. >>>> > > > > > > >> > >>> >> >>> >>>> > > > > > > >> > >>> >> >>> Thanks for listing the pros and cons of >>>> Kerberos. >>>> > > > > > > Concerning >>>> > > > > > > >> > what >>>> > > > > > > >> > >>> >> other >>>> > > > > > > >> > >>> >> >>> authentication mechanisms are used in the >>>> > > industry, I >>>> > > > > am >>>> > > > > > > not >>>> > > > > > > >> > 100% >>>> > > > > > > >> > >>> >> sure. >>>> > > > > > > >> > >>> >> >>> >>>> > > > > > > >> > >>> >> >>> Cheers, >>>> > > > > > > >> > >>> >> >>> Till >>>> > > > > > > >> > >>> >> >>> >>>> > > > > > > >> > >>> >> >>> On Fri, Jun 4, 2021 at 11:09 AM Gabor >>>> Somogyi < >>>> > > > > > > >> > >>> >> gabor.g.somo...@gmail.com> >>>> > > > > > > >> > >>> >> >>> wrote: >>>> > > > > > > >> > >>> >> >>> >>>> > > > > > > >> > >>> >> >>>> > I did not mean for the user to sign its >>>> own >>>> > > > > > certificates >>>> > > > > > > >> but >>>> > > > > > > >> > >>> for >>>> > > > > > > >> > >>> >> the >>>> > > > > > > >> > >>> >> >>>> operator of the cluster. Once the user >>>> request >>>> > > hits >>>> > > > > the >>>> > > > > > > >> proxy, >>>> > > > > > > >> > it >>>> > > > > > > >> > >>> >> should no >>>> > > > > > > >> > >>> >> >>>> longer be under his control. I think I do >>>> not >>>> > > fully >>>> > > > > > > >> understand >>>> > > > > > > >> > >>> yet >>>> > > > > > > >> > >>> >> why this >>>> > > > > > > >> > >>> >> >>>> would not work. >>>> > > > > > > >> > >>> >> >>>> I said it's not solving the authentication >>>> > problem >>>> > > > > over >>>> > > > > > > any >>>> > > > > > > >> > >>> proxy. >>>> > > > > > > >> > >>> >> Even >>>> > > > > > > >> > >>> >> >>>> if the operator is signing the certificate >>>> one >>>> > can >>>> > > > > have >>>> > > > > > > >> access >>>> > > > > > > >> > >>> to an >>>> > > > > > > >> > >>> >> >>>> internal node. >>>> > > > > > > >> > >>> >> >>>> Such case anybody can craft certificates >>>> which >>>> > is >>>> > > > > > accepted >>>> > > > > > > >> by >>>> > > > > > > >> > the >>>> > > > > > > >> > >>> >> >>>> server. When it's accepted a bad guy can >>>> cancel >>>> > > jobs >>>> > > > > > > causing >>>> > > > > > > >> > huge >>>> > > > > > > >> > >>> >> impacts. >>>> > > > > > > >> > >>> >> >>>> >>>> > > > > > > >> > >>> >> >>>> > Also, I am missing a bit the comparison >>>> of >>>> > > > Kerberos >>>> > > > > to >>>> > > > > > > >> other >>>> > > > > > > >> > >>> >> >>>> authentication mechanisms and why they were >>>> > > rejected >>>> > > > > in >>>> > > > > > > >> favour >>>> > > > > > > >> > of >>>> > > > > > > >> > >>> >> Kerberos. >>>> > > > > > > >> > >>> >> >>>> PROS: >>>> > > > > > > >> > >>> >> >>>> * Since it's not depending on cloud >>>> provider >>>> > > and/or >>>> > > > > k8s >>>> > > > > > or >>>> > > > > > > >> > >>> bare-metal >>>> > > > > > > >> > >>> >> >>>> etc. deployment it's the biggest plus >>>> > > > > > > >> > >>> >> >>>> * Centralized with tools and no need to >>>> write >>>> > tons >>>> > > > of >>>> > > > > > > tools >>>> > > > > > > >> > >>> around >>>> > > > > > > >> > >>> >> >>>> * There are clients/tools on almost all >>>> OS-es >>>> > and >>>> > > > > > several >>>> > > > > > > >> > >>> languages >>>> > > > > > > >> > >>> >> >>>> * Super huge users are using it for years >>>> in >>>> > > > > production >>>> > > > > > > w/o >>>> > > > > > > >> > huge >>>> > > > > > > >> > >>> >> issues >>>> > > > > > > >> > >>> >> >>>> * Provides cross-realm trust possibility >>>> amongst >>>> > > > other >>>> > > > > > > >> features >>>> > > > > > > >> > >>> >> >>>> * Several open source components using it >>>> which >>>> > > > could >>>> > > > > > > >> increase >>>> > > > > > > >> > >>> >> >>>> compatibility >>>> > > > > > > >> > >>> >> >>>> >>>> > > > > > > >> > >>> >> >>>> CONS: >>>> > > > > > > >> > >>> >> >>>> * Not everybody using kerberos >>>> > > > > > > >> > >>> >> >>>> * It would increase the code footprint but >>>> this >>>> > is >>>> > > > > true >>>> > > > > > > for >>>> > > > > > > >> > many >>>> > > > > > > >> > >>> >> >>>> features (as a side note I'm here to >>>> maintain >>>> > it) >>>> > > > > > > >> > >>> >> >>>> >>>> > > > > > > >> > >>> >> >>>> Feel free to add your points because it >>>> only >>>> > > > > represents >>>> > > > > > a >>>> > > > > > > >> > single >>>> > > > > > > >> > >>> >> >>>> viewpoint. >>>> > > > > > > >> > >>> >> >>>> Also if you have any better option for >>>> strong >>>> > > > > > > authentication >>>> > > > > > > >> > >>> please >>>> > > > > > > >> > >>> >> >>>> share it and we can consider the pros/cons >>>> here. >>>> > > > > > > >> > >>> >> >>>> >>>> > > > > > > >> > >>> >> >>>> BR, >>>> > > > > > > >> > >>> >> >>>> G >>>> > > > > > > >> > >>> >> >>>> >>>> > > > > > > >> > >>> >> >>>> >>>> > > > > > > >> > >>> >> >>>> On Fri, Jun 4, 2021 at 10:32 AM Till >>>> Rohrmann < >>>> > > > > > > >> > >>> trohrm...@apache.org> >>>> > > > > > > >> > >>> >> >>>> wrote: >>>> > > > > > > >> > >>> >> >>>> >>>> > > > > > > >> > >>> >> >>>>> I did not mean for the user to sign its >>>> own >>>> > > > > > certificates >>>> > > > > > > >> but >>>> > > > > > > >> > >>> for the >>>> > > > > > > >> > >>> >> >>>>> operator of the cluster. Once the user >>>> request >>>> > > hits >>>> > > > > the >>>> > > > > > > >> proxy, >>>> > > > > > > >> > >>> it >>>> > > > > > > >> > >>> >> should no >>>> > > > > > > >> > >>> >> >>>>> longer be under his control. I think I do >>>> not >>>> > > fully >>>> > > > > > > >> understand >>>> > > > > > > >> > >>> yet >>>> > > > > > > >> > >>> >> why this >>>> > > > > > > >> > >>> >> >>>>> would not work. >>>> > > > > > > >> > >>> >> >>>>> >>>> > > > > > > >> > >>> >> >>>>> What I would like to avoid is to add more >>>> > > > complexity >>>> > > > > > into >>>> > > > > > > >> > Flink >>>> > > > > > > >> > >>> if >>>> > > > > > > >> > >>> >> >>>>> there is an easy solution which fulfills >>>> the >>>> > > > > > > requirements. >>>> > > > > > > >> > >>> That's >>>> > > > > > > >> > >>> >> why I >>>> > > > > > > >> > >>> >> >>>>> would like to exercise thoroughly through >>>> the >>>> > > > > different >>>> > > > > > > >> > >>> >> alternatives. Also, >>>> > > > > > > >> > >>> >> >>>>> I am missing a bit the comparison of >>>> Kerberos >>>> > to >>>> > > > > other >>>> > > > > > > >> > >>> >> authentication >>>> > > > > > > >> > >>> >> >>>>> mechanisms and why they were rejected in >>>> favour >>>> > > of >>>> > > > > > > >> Kerberos. >>>> > > > > > > >> > >>> >> >>>>> >>>> > > > > > > >> > >>> >> >>>>> Cheers, >>>> > > > > > > >> > >>> >> >>>>> Till >>>> > > > > > > >> > >>> >> >>>>> >>>> > > > > > > >> > >>> >> >>>>> On Fri, Jun 4, 2021 at 10:26 AM Gyula >>>> Fóra < >>>> > > > > > > >> gyf...@apache.org >>>> > > > > > > >> > > >>>> > > > > > > >> > >>> >> wrote: >>>> > > > > > > >> > >>> >> >>>>> >>>> > > > > > > >> > >>> >> >>>>>> Hi! >>>> > > > > > > >> > >>> >> >>>>>> >>>> > > > > > > >> > >>> >> >>>>>> I think there might be possible >>>> alternatives >>>> > but >>>> > > > it >>>> > > > > > > seems >>>> > > > > > > >> > >>> Kerberos >>>> > > > > > > >> > >>> >> on >>>> > > > > > > >> > >>> >> >>>>>> the rest endpoint ticks all the right >>>> boxes >>>> > and >>>> > > > > > > provides a >>>> > > > > > > >> > >>> super >>>> > > > > > > >> > >>> >> clean and >>>> > > > > > > >> > >>> >> >>>>>> simple solution for strong >>>> authentication. >>>> > > > > > > >> > >>> >> >>>>>> >>>> > > > > > > >> > >>> >> >>>>>> I wouldn’t even consider sidecar proxies >>>> etc >>>> > if >>>> > > we >>>> > > > > can >>>> > > > > > > >> solve >>>> > > > > > > >> > >>> it in >>>> > > > > > > >> > >>> >> >>>>>> such a simple way as proposed by G. >>>> > > > > > > >> > >>> >> >>>>>> >>>> > > > > > > >> > >>> >> >>>>>> Cheers >>>> > > > > > > >> > >>> >> >>>>>> Gyula >>>> > > > > > > >> > >>> >> >>>>>> >>>> > > > > > > >> > >>> >> >>>>>> On Fri, 4 Jun 2021 at 10:03, Till >>>> Rohrmann < >>>> > > > > > > >> > >>> trohrm...@apache.org> >>>> > > > > > > >> > >>> >> >>>>>> wrote: >>>> > > > > > > >> > >>> >> >>>>>> >>>> > > > > > > >> > >>> >> >>>>>>> I am not saying that we shouldn't add a >>>> > strong >>>> > > > > > > >> > authentication >>>> > > > > > > >> > >>> >> >>>>>>> mechanism if there are good reasons for >>>> it. I >>>> > > > > > primarily >>>> > > > > > > >> > would >>>> > > > > > > >> > >>> >> like to >>>> > > > > > > >> > >>> >> >>>>>>> understand the context a bit better in >>>> order >>>> > to >>>> > > > > give >>>> > > > > > > >> > qualified >>>> > > > > > > >> > >>> >> feedback and >>>> > > > > > > >> > >>> >> >>>>>>> come to a good decision. In order to do >>>> > this, I >>>> > > > > have >>>> > > > > > > the >>>> > > > > > > >> > >>> feeling >>>> > > > > > > >> > >>> >> that we >>>> > > > > > > >> > >>> >> >>>>>>> haven't fully considered all available >>>> > options >>>> > > > > which >>>> > > > > > > are >>>> > > > > > > >> on >>>> > > > > > > >> > >>> the >>>> > > > > > > >> > >>> >> table, tbh. >>>> > > > > > > >> > >>> >> >>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>> Does the problem of certificate expiry >>>> also >>>> > > apply >>>> > > > > for >>>> > > > > > > >> > >>> self-signed >>>> > > > > > > >> > >>> >> >>>>>>> certificates? If yes, then this should >>>> then >>>> > > also >>>> > > > > be a >>>> > > > > > > >> > problem >>>> > > > > > > >> > >>> for >>>> > > > > > > >> > >>> >> the >>>> > > > > > > >> > >>> >> >>>>>>> internal encryption of Flink's >>>> communication. >>>> > > If >>>> > > > > not, >>>> > > > > > > >> then >>>> > > > > > > >> > one >>>> > > > > > > >> > >>> >> could use >>>> > > > > > > >> > >>> >> >>>>>>> self-signed certificates with a longer >>>> > validity >>>> > > > to >>>> > > > > > > solve >>>> > > > > > > >> the >>>> > > > > > > >> > >>> >> mentioned >>>> > > > > > > >> > >>> >> >>>>>>> issue. >>>> > > > > > > >> > >>> >> >>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>> I think you can set up Flink in such a >>>> way >>>> > that >>>> > > > you >>>> > > > > > > don't >>>> > > > > > > >> > >>> have to >>>> > > > > > > >> > >>> >> >>>>>>> handle all the different certificates. >>>> For >>>> > > > example, >>>> > > > > > you >>>> > > > > > > >> > could >>>> > > > > > > >> > >>> >> deploy Flink >>>> > > > > > > >> > >>> >> >>>>>>> with a "sidecar proxy" which is >>>> responsible >>>> > for >>>> > > > the >>>> > > > > > > >> > >>> >> authentication using an >>>> > > > > > > >> > >>> >> >>>>>>> arbitrary method (e.g. Kerberos) and >>>> then >>>> > bind >>>> > > > the >>>> > > > > > REST >>>> > > > > > > >> > >>> endpoint >>>> > > > > > > >> > >>> >> to a local >>>> > > > > > > >> > >>> >> >>>>>>> network interface. That way, the REST >>>> > endpoint >>>> > > > > would >>>> > > > > > > >> only be >>>> > > > > > > >> > >>> >> available >>>> > > > > > > >> > >>> >> >>>>>>> through the sidecar proxy. >>>> Additionally, one >>>> > > > could >>>> > > > > > > enable >>>> > > > > > > >> > SSL >>>> > > > > > > >> > >>> for >>>> > > > > > > >> > >>> >> this >>>> > > > > > > >> > >>> >> >>>>>>> communication. Would this be a solution >>>> for >>>> > the >>>> > > > > > > problem? >>>> > > > > > > >> > >>> >> >>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>> Cheers, >>>> > > > > > > >> > >>> >> >>>>>>> Till >>>> > > > > > > >> > >>> >> >>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>> On Thu, Jun 3, 2021 at 10:46 PM Márton >>>> > Balassi >>>> > > < >>>> > > > > > > >> > >>> >> >>>>>>> balassi.mar...@gmail.com> wrote: >>>> > > > > > > >> > >>> >> >>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>> That is an interesting idea, Till. >>>> > > > > > > >> > >>> >> >>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>> The main issue with it is that TLS >>>> > > certificates >>>> > > > > have >>>> > > > > > > an >>>> > > > > > > >> > >>> >> expiration >>>> > > > > > > >> > >>> >> >>>>>>>> time, usually they get approved for a >>>> couple >>>> > > > > years. >>>> > > > > > > >> Forcing >>>> > > > > > > >> > >>> our >>>> > > > > > > >> > >>> >> users to >>>> > > > > > > >> > >>> >> >>>>>>>> restart jobs to reprovision TLS >>>> certificates >>>> > > > would >>>> > > > > > be >>>> > > > > > > >> weird >>>> > > > > > > >> > >>> when >>>> > > > > > > >> > >>> >> we could >>>> > > > > > > >> > >>> >> >>>>>>>> just implement a single proper strong >>>> > > > > authentication >>>> > > > > > > >> > >>> mechanism >>>> > > > > > > >> > >>> >> instead in a >>>> > > > > > > >> > >>> >> >>>>>>>> couple hundred lines of code. :-) >>>> > > > > > > >> > >>> >> >>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>> In many cases it is also impractical >>>> to go >>>> > the >>>> > > > TLS >>>> > > > > > > >> mutual >>>> > > > > > > >> > >>> route, >>>> > > > > > > >> > >>> >> >>>>>>>> because the Flink Dashboard can end up >>>> on >>>> > any >>>> > > > node >>>> > > > > > in >>>> > > > > > > >> the >>>> > > > > > > >> > >>> >> k8s/Yarn cluster >>>> > > > > > > >> > >>> >> >>>>>>>> which means that we need a certificate >>>> per >>>> > > node >>>> > > > > (due >>>> > > > > > > to >>>> > > > > > > >> the >>>> > > > > > > >> > >>> >> mutual auth), >>>> > > > > > > >> > >>> >> >>>>>>>> but if we also want to protect the >>>> private >>>> > key >>>> > > > of >>>> > > > > > > these >>>> > > > > > > >> > from >>>> > > > > > > >> > >>> >> users >>>> > > > > > > >> > >>> >> >>>>>>>> accidentally or intentionally leaking >>>> them >>>> > > then >>>> > > > we >>>> > > > > > > need >>>> > > > > > > >> > this >>>> > > > > > > >> > >>> per >>>> > > > > > > >> > >>> >> user. As >>>> > > > > > > >> > >>> >> >>>>>>>> in we end up managing user*machine >>>> number >>>> > > > > > certificates >>>> > > > > > > >> and >>>> > > > > > > >> > >>> >> having to renew >>>> > > > > > > >> > >>> >> >>>>>>>> them periodically, which albeit >>>> automatable >>>> > is >>>> > > > > > > >> > unfortunately >>>> > > > > > > >> > >>> not >>>> > > > > > > >> > >>> >> yet >>>> > > > > > > >> > >>> >> >>>>>>>> automated in all large organizations. >>>> > > > > > > >> > >>> >> >>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>> I fully agree that TLS certificate >>>> mutual >>>> > > > > > > authentication >>>> > > > > > > >> > has >>>> > > > > > > >> > >>> its >>>> > > > > > > >> > >>> >> >>>>>>>> nice properties, especially at very >>>> large >>>> > > > > (multiple >>>> > > > > > > >> > thousand >>>> > > > > > > >> > >>> >> node) clusters >>>> > > > > > > >> > >>> >> >>>>>>>> - but it has its own challenges too. >>>> Thanks >>>> > > for >>>> > > > > > > >> bringing it >>>> > > > > > > >> > >>> up. >>>> > > > > > > >> > >>> >> >>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>> Happy to have this added to the >>>> rejected >>>> > > > > alternative >>>> > > > > > > >> list >>>> > > > > > > >> > so >>>> > > > > > > >> > >>> that >>>> > > > > > > >> > >>> >> >>>>>>>> we have the full picture documented. >>>> > > > > > > >> > >>> >> >>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>> On Thu, Jun 3, 2021 at 5:52 PM Till >>>> > Rohrmann < >>>> > > > > > > >> > >>> >> trohrm...@apache.org> >>>> > > > > > > >> > >>> >> >>>>>>>> wrote: >>>> > > > > > > >> > >>> >> >>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>> I guess the idea would then be to let >>>> the >>>> > > proxy >>>> > > > > do >>>> > > > > > > the >>>> > > > > > > >> > >>> >> >>>>>>>>> authentication job and only forward >>>> the >>>> > > request >>>> > > > > via >>>> > > > > > > an >>>> > > > > > > >> SSL >>>> > > > > > > >> > >>> >> mutually >>>> > > > > > > >> > >>> >> >>>>>>>>> encrypted connection to the Flink >>>> cluster. >>>> > > > Would >>>> > > > > > this >>>> > > > > > > >> be >>>> > > > > > > >> > >>> >> possible? The >>>> > > > > > > >> > >>> >> >>>>>>>>> beauty of this setup is in my opinion >>>> that >>>> > > this >>>> > > > > > setup >>>> > > > > > > >> > should >>>> > > > > > > >> > >>> >> work with all >>>> > > > > > > >> > >>> >> >>>>>>>>> kinds of authentication mechanisms. >>>> > > > > > > >> > >>> >> >>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>> Cheers, >>>> > > > > > > >> > >>> >> >>>>>>>>> Till >>>> > > > > > > >> > >>> >> >>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>> On Thu, Jun 3, 2021 at 3:12 PM Gabor >>>> > Somogyi >>>> > > < >>>> > > > > > > >> > >>> >> >>>>>>>>> gabor.g.somo...@gmail.com> wrote: >>>> > > > > > > >> > >>> >> >>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>> Thanks for giving options to fulfil >>>> the >>>> > > need. >>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>> Users are looking for a solution >>>> where >>>> > users >>>> > > > can >>>> > > > > > be >>>> > > > > > > >> > >>> identified >>>> > > > > > > >> > >>> >> on >>>> > > > > > > >> > >>> >> >>>>>>>>>> the whole cluster and restrict >>>> access to >>>> > > > > > > >> > resources/actions. >>>> > > > > > > >> > >>> >> >>>>>>>>>> A good example for such an action is >>>> > > > cancelling >>>> > > > > > > other >>>> > > > > > > >> > users >>>> > > > > > > >> > >>> >> >>>>>>>>>> running jobs. >>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>> * SSL does provide mutual >>>> authentication >>>> > but >>>> > > > > when >>>> > > > > > > >> > >>> >> authentication >>>> > > > > > > >> > >>> >> >>>>>>>>>> passed there is no user based on >>>> > > restrictions >>>> > > > > can >>>> > > > > > be >>>> > > > > > > >> > made. >>>> > > > > > > >> > >>> >> >>>>>>>>>> * The less problematic part is that >>>> > > > > > > >> > generating/maintaining >>>> > > > > > > >> > >>> >> short >>>> > > > > > > >> > >>> >> >>>>>>>>>> time valid certificates would be a >>>> hard >>>> > > > (that's >>>> > > > > > the >>>> > > > > > > >> > reason >>>> > > > > > > >> > >>> KDC >>>> > > > > > > >> > >>> >> like servers >>>> > > > > > > >> > >>> >> >>>>>>>>>> exist). >>>> > > > > > > >> > >>> >> >>>>>>>>>> Having long time valid certificates >>>> would >>>> > > > widen >>>> > > > > > the >>>> > > > > > > >> > attack >>>> > > > > > > >> > >>> >> >>>>>>>>>> surface but since the first concern >>>> is >>>> > there >>>> > > > > this >>>> > > > > > is >>>> > > > > > > >> > just a >>>> > > > > > > >> > >>> >> cosmetic issue. >>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>> All in all using TLS certificates is >>>> not >>>> > > > > > sufficient >>>> > > > > > > in >>>> > > > > > > >> > >>> these >>>> > > > > > > >> > >>> >> >>>>>>>>>> environments unfortunately. >>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>> BR, >>>> > > > > > > >> > >>> >> >>>>>>>>>> G >>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>> On Thu, Jun 3, 2021 at 12:49 PM Till >>>> > > Rohrmann >>>> > > > < >>>> > > > > > > >> > >>> >> >>>>>>>>>> trohrm...@apache.org> wrote: >>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>> Thanks for the information Gabor. >>>> If it >>>> > is >>>> > > > > about >>>> > > > > > > >> > securing >>>> > > > > > > >> > >>> the >>>> > > > > > > >> > >>> >> >>>>>>>>>>> communication between the REST >>>> client and >>>> > > the >>>> > > > > > REST >>>> > > > > > > >> > server, >>>> > > > > > > >> > >>> >> then Flink >>>> > > > > > > >> > >>> >> >>>>>>>>>>> already supports enabling mutual SSL >>>> > > > > > authentication >>>> > > > > > > >> [1]. >>>> > > > > > > >> > >>> >> Would this be >>>> > > > > > > >> > >>> >> >>>>>>>>>>> enough to secure the communication >>>> and to >>>> > > > pass >>>> > > > > an >>>> > > > > > > >> audit? >>>> > > > > > > >> > >>> >> >>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>> [1] >>>> > > > > > > >> > >>> >> >>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >>>> > > > > > > >> >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/security/security-ssl/#external--rest-connectivity >>>> > > > > > > >> > >>> >> >>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>> Cheers, >>>> > > > > > > >> > >>> >> >>>>>>>>>>> Till >>>> > > > > > > >> > >>> >> >>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>> On Thu, Jun 3, 2021 at 10:33 AM >>>> Gabor >>>> > > > Somogyi < >>>> > > > > > > >> > >>> >> >>>>>>>>>>> gabor.g.somo...@gmail.com> wrote: >>>> > > > > > > >> > >>> >> >>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Hi Till, >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Since I'm working in security area >>>> 10+ >>>> > > years >>>> > > > > let >>>> > > > > > > me >>>> > > > > > > >> > >>> share my >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> thought. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> I would like to emphasise there are >>>> > > experts >>>> > > > > > better >>>> > > > > > > >> than >>>> > > > > > > >> > >>> me >>>> > > > > > > >> > >>> >> but >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> I have some >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> basics. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> The discussion is open and not >>>> trying to >>>> > > > tell >>>> > > > > > > alone >>>> > > > > > > >> > >>> things... >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > I mean if an attacker can get >>>> access >>>> > to >>>> > > > one >>>> > > > > of >>>> > > > > > > the >>>> > > > > > > >> > >>> >> machines, >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> then it >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> should also be possible to obtain >>>> the >>>> > > right >>>> > > > > > > Kerberos >>>> > > > > > > >> > >>> token. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Not necessarily. For example if >>>> one gets >>>> > > > > access >>>> > > > > > > to a >>>> > > > > > > >> > >>> specific >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> user's >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> credentials then it's not possible >>>> to >>>> > > > > compromise >>>> > > > > > > >> other >>>> > > > > > > >> > >>> user's >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> jobs, data, >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> etc... >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Security is like an onion, the more >>>> > layers >>>> > > > has >>>> > > > > > > been >>>> > > > > > > >> > >>> added the >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> more time an >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> attacker needs to proceed. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> At the end of the day if one is >>>> in, then >>>> > > > most >>>> > > > > > > >> probably >>>> > > > > > > >> > >>> can >>>> > > > > > > >> > >>> >> find >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> the way but >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> this time is normally enough to >>>> > sysadmins >>>> > > or >>>> > > > > > > >> security >>>> > > > > > > >> > >>> >> experts to >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> close down the system and minimize >>>> the >>>> > > > damage. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> The other thing is that all tokens >>>> has a >>>> > > > > timeout >>>> > > > > > > >> and if >>>> > > > > > > >> > >>> the >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> token is >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> invalid then the attacker can't >>>> proceed >>>> > > > > further. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Is Kerberos also the standard >>>> > > > authentication >>>> > > > > > > >> protocol >>>> > > > > > > >> > >>> for >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Kubernetes >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> deployments? >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Kerberos is an industry standard >>>> which >>>> > is >>>> > > > > > > >> > >>> cloud/deployment >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> agnostic and it >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> can be used in any deployments >>>> including >>>> > > > k8s. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> The main intention is to use >>>> kerberos in >>>> > > k8s >>>> > > > > > > >> > deployments >>>> > > > > > > >> > >>> too >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> since we're >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> going this direction as well. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Please see how Spark does this: >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >>>> > > > > > > >> >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://spark.apache.org/docs/latest/security.html#secure-interaction-with-kubernetes >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Last but not least the most >>>> important >>>> > > reason >>>> > > > > to >>>> > > > > > > add >>>> > > > > > > >> at >>>> > > > > > > >> > >>> least >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> one strong >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> authentication is that we have >>>> users who >>>> > > has >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> hard requirements on this. They're >>>> doing >>>> > > > > > security >>>> > > > > > > >> > audits >>>> > > > > > > >> > >>> and >>>> > > > > > > >> > >>> >> if >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> they fail >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> then it's deal breaking. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> That is why we have added kerberos >>>> at >>>> > the >>>> > > > > first >>>> > > > > > > >> place. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Unfortunately we >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> can't name them in this public >>>> list, >>>> > > however >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> the customers who specifically >>>> asked for >>>> > > > this >>>> > > > > > were >>>> > > > > > > >> > >>> mainly in >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> the banking >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> and telco sector. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> BR, >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> G >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> On Thu, Jun 3, 2021 at 9:20 AM Till >>>> > > > Rohrmann < >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> trohrm...@apache.org> wrote: >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Thanks for updating the document >>>> > Márton. >>>> > > > Why >>>> > > > > > is >>>> > > > > > > it >>>> > > > > > > >> > that >>>> > > > > > > >> > >>> >> banks >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> will >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > consider it more secure if Flink >>>> comes >>>> > > > with >>>> > > > > > > >> Kerberos >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> authentication >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > (assuming a properly secured >>>> setup)? I >>>> > > > mean >>>> > > > > if >>>> > > > > > > an >>>> > > > > > > >> > >>> attacker >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> can get access >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > to one of the machines, then it >>>> should >>>> > > > also >>>> > > > > be >>>> > > > > > > >> > >>> possible to >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> obtain the right >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Kerberos token. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > I am not an authentication >>>> expert and >>>> > > > that's >>>> > > > > > > why I >>>> > > > > > > >> > >>> wanted >>>> > > > > > > >> > >>> >> to >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> ask what are >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > other authentication protocols >>>> other >>>> > > than >>>> > > > > > > >> Kerberos? >>>> > > > > > > >> > >>> Why did >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> we select >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Kerberos and not any other >>>> > > authentication >>>> > > > > > > >> protocol? >>>> > > > > > > >> > >>> Maybe >>>> > > > > > > >> > >>> >> you >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> can list the >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > pros and cons for the different >>>> > > protocols. >>>> > > > > Is >>>> > > > > > > >> > Kerberos >>>> > > > > > > >> > >>> also >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> the standard >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > authentication protocol for >>>> Kubernetes >>>> > > > > > > >> deployments? >>>> > > > > > > >> > If >>>> > > > > > > >> > >>> not, >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> what would be >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > the answer when deploying on K8s? >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Cheers, >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Till >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > On Wed, Jun 2, 2021 at 12:07 PM >>>> Gabor >>>> > > > > Somogyi >>>> > > > > > < >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> gabor.g.somo...@gmail.com> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > wrote: >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> Hi team, >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> Happy to be here and hope I can >>>> > provide >>>> > > > > > quality >>>> > > > > > > >> > >>> additions >>>> > > > > > > >> > >>> >> in >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> the future. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> Thank you all for helpful the >>>> > > > suggestions! >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> Considering them the FLIP has >>>> been >>>> > > > modified >>>> > > > > > and >>>> > > > > > > >> the >>>> > > > > > > >> > >>> work >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> continues on the >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> already existing Jira. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> BR, >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> G >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> On Wed, Jun 2, 2021 at 11:23 AM >>>> > Márton >>>> > > > > > Balassi >>>> > > > > > > < >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> balassi.mar...@gmail.com> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> wrote: >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Thanks, Chesney - I totally >>>> missed >>>> > > that. >>>> > > > > > > >> Answered >>>> > > > > > > >> > on >>>> > > > > > > >> > >>> the >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> ticket too, let >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> us continue there then. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Till, I agree that we should >>>> keep >>>> > this >>>> > > > > > > codepath >>>> > > > > > > >> as >>>> > > > > > > >> > >>> slim >>>> > > > > > > >> > >>> >> as >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> possible. It >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> is an important design >>>> decision that >>>> > > we >>>> > > > > aim >>>> > > > > > to >>>> > > > > > > >> keep >>>> > > > > > > >> > >>> the >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> list of >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> authentication protocols to a >>>> > minimum. >>>> > > > We >>>> > > > > > > >> believe >>>> > > > > > > >> > >>> that >>>> > > > > > > >> > >>> >> this >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> should not be a >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> primary concern of Flink and a >>>> > trusted >>>> > > > > proxy >>>> > > > > > > >> > service >>>> > > > > > > >> > >>> (for >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> example Apache >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Knox) should be used to enable >>>> a >>>> > > > multitude >>>> > > > > > of >>>> > > > > > > >> > enduser >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> authentication >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> mechanisms. The bare minimum of >>>> > > > > > authentication >>>> > > > > > > >> > >>> mechanisms >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> to support >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> consequently consist of a >>>> single >>>> > > strong >>>> > > > > > > >> > >>> authentication >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> protocol for which >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Kerberos is the enterprise >>>> solution >>>> > > and >>>> > > > > HTTP >>>> > > > > > > >> Basic >>>> > > > > > > >> > >>> >> primary >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> for development >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> and light-weight scenarios. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Added the above wording to G's >>>> doc. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >>>> > > > > > > >> >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> On Tue, Jun 1, 2021 at 11:47 AM >>>> > > Chesnay >>>> > > > > > > >> Schepler < >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> ches...@apache.org> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> wrote: >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> There's a related effort: >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>>> > > > > > > >> https://issues.apache.org/jira/browse/FLINK-21108 >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> On 6/1/2021 10:14 AM, Till >>>> Rohrmann >>>> > > > > wrote: >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Hi Gabor, welcome to the >>>> Flink >>>> > > > > community! >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Thanks for sharing this >>>> proposal >>>> > > with >>>> > > > > the >>>> > > > > > > >> > >>> community >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Márton. In >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> general, I >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > agree that authentication is >>>> > > missing >>>> > > > > and >>>> > > > > > > that >>>> > > > > > > >> > >>> this is >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> required for >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> using >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Flink within an enterprise. >>>> The >>>> > > > thing I >>>> > > > > > am >>>> > > > > > > >> > >>> wondering >>>> > > > > > > >> > >>> >> is >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> whether this >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > feature strictly needs to be >>>> > > > > implemented >>>> > > > > > > >> inside >>>> > > > > > > >> > of >>>> > > > > > > >> > >>> >> Flink >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> or whether a >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> proxy >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > setup could do the job? >>>> Have you >>>> > > > > > considered >>>> > > > > > > >> this >>>> > > > > > > >> > >>> >> option? >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> If yes, then >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> it >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > would be good to list it >>>> under >>>> > the >>>> > > > > point >>>> > > > > > of >>>> > > > > > > >> > >>> rejected >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> alternatives. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > I do see the benefit of >>>> > > implementing >>>> > > > > this >>>> > > > > > > >> > feature >>>> > > > > > > >> > >>> >> inside >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> of Flink if >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> many >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > users need it. If not, then >>>> it >>>> > > might >>>> > > > be >>>> > > > > > > >> easier >>>> > > > > > > >> > >>> for the >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> project to not >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > increase the surface area >>>> since >>>> > it >>>> > > > > makes >>>> > > > > > > the >>>> > > > > > > >> > >>> overall >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> maintenance >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> harder. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Cheers, >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Till >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > On Mon, May 31, 2021 at >>>> 4:57 PM >>>> > > > Márton >>>> > > > > > > >> Balassi < >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> mbala...@apache.org> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> wrote: >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Hi team, >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Firstly I would like to >>>> > introduce >>>> > > > > Gabor >>>> > > > > > > or G >>>> > > > > > > >> > [1] >>>> > > > > > > >> > >>> for >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> short to the >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> community, he is a Spark >>>> > committer >>>> > > > who >>>> > > > > > has >>>> > > > > > > >> > >>> recently >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> transitioned to >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> the >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Flink Engineering team at >>>> > Cloudera >>>> > > > and >>>> > > > > > is >>>> > > > > > > >> > looking >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> forward to >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> contributing >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> to Apache Flink. >>>> Previously G >>>> > > > > primarily >>>> > > > > > > >> focused >>>> > > > > > > >> > >>> on >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Spark Streaming >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> and >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> security. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Based on requests from our >>>> > > > customers G >>>> > > > > > has >>>> > > > > > > >> > >>> >> implemented >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Kerberos and >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> HTTP >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Basic Authentication for >>>> the >>>> > Flink >>>> > > > > > > Dashboard >>>> > > > > > > >> > and >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> HistoryServer. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> Previously >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> lacked an authentication >>>> story. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> We are looking to >>>> contribute >>>> > this >>>> > > > > > > >> functionality >>>> > > > > > > >> > >>> back >>>> > > > > > > >> > >>> >> to >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> the >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> community, we >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> believe that given Flink's >>>> > > maturity >>>> > > > > > there >>>> > > > > > > >> > should >>>> > > > > > > >> > >>> be a >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> common code >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> solution >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> for this general pattern. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> We are looking forward to >>>> your >>>> > > > > feedback >>>> > > > > > on >>>> > > > > > > >> G's >>>> > > > > > > >> > >>> >> design. >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> [2] >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> [1] >>>> http://gaborsomogyi.com/ >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> [2] >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >>>> > > > > > > >> >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>>>>>>>>> >>>> > > > > > > >> > >>> >> >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > -- >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > Konstantin Knauf >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > https://twitter.com/snntrable >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> > https://github.com/knaufk >>>> > > > > > > >> > >>> > >>>> > > > > > > >> > >>> >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> -- >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> Konstantin Knauf >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> https://twitter.com/snntrable >>>> > > > > > > >> > >> >>>> > > > > > > >> > >> https://github.com/knaufk >>>> > > > > > > >> > >> >>>> > > > > > > >> > > >>>> > > > > > > >> > >>>> > > > > > > >> >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > > >>>> > > > -- >>>> > > > >>>> > > > Konstantin Knauf >>>> > > > >>>> > > > https://twitter.com/snntrable >>>> > > > >>>> > > > https://github.com/knaufk >>>> > > > >>>> > > >>>> > >>>> >>>
