Hi Gabor, Thanks for your answers. I appreciate the explanations. Please see my responses + further questions below.
* What stability semantics do you envision for this API? >> > As I foresee the API will be as stable as Netty API. Since there is > guarantee on no breaking changes between minor versions we can give the > same guarantee. > If for whatever reason we need to break it we can do it in major version > like every other open source project does. > * Does Flink expose dependencies’ APIs in other places? Since this exposes >> the Netty API, will this make it difficult to upgrade Netty? >> > I don't expect breaking changes between minor versions so such cases there > will be no issues. If there is a breaking change in major version > we need to wait Flink major version too. > To clarify, you are proposing this new API to have the same stability guarantees as @Public currently does? Where we will not introduce breaking changes unless absolutely necessary (and requiring a FLIP, etc.)? If this is the case, I think this puts the community in a tough position where we are forced to maintain compatibility with something that we do not have control over. It also locks Flink into the current major version of Netty (and the Netty framework itself) for the foreseeable future. I am saying we should not do this, perhaps this is the best solution to finding a good compromise here, but I am trying to discover + acknowledge the full implications of this proposal so they can be discussed. What do you think about marking this API as @Experimental and not guaranteeing stability between versions? Then, if we do decide we need to upgrade Netty (or move away from it), we can do so. * I share Till's concern about multiple factories – other HTTP middleware >> frameworks commonly support chaining middlewares. Since the proposed API >> does not include these features/guarantee ordering, do you see any reason >> to allow more than one factory? >> > I personally can't come up with a use-case where ordering is a must. I'm > not telling that this is not a valid use-case but adding a feature w/o > business rationale would include the maintenance cost (though I'm open to > add). > As I've seen Till also can't give example for that (please see the doc > comments). If you have anything in mind please share it and we can add > priority to the API. > There is another option too, namely we can be defensive and we can add the > priority right now. I would do this only if everybody states in mail that > it would be the best option, > otherwise I would stick to the original plan. > Let me try to come up with a use case: * Someone creates an authentication module for integrating with Google's OAuth and publishes it to flink-packages * Another person in another org wants to use Google OAuth and then add internal authorization based on the user * In this scenario, *Google OAuth must come before the internal authorization* * They place their module and the Google OAuth module to be picked up by the service loader * What happens? I do not think that the current proposal has a way to handle this, besides having the implementor of the internal authorization module bundle everything into one, as you have suggested. Since this is the only way to achieve order, why not restrict the service loader to only allow one? This way the API is explicit in what it supports. Let me know what you think, Austin On Wed, Jun 30, 2021 at 5:24 AM Gabor Somogyi <gabor.g.somo...@gmail.com> wrote: > Hi Austin, > > Please see my answers embedded down below. > > BR, > G > > > > On Tue, Jun 29, 2021 at 9:59 PM Austin Cawley-Edwards < > austin.caw...@gmail.com> wrote: > >> Hi all, >> >> Thanks for the updated proposal. I have a few questions about the API, >> please see below. >> >> * What stability semantics do you envision for this API? >> > As I foresee the API will be as stable as Netty API. Since there is > guarantee on no breaking changes between minor versions we can give the > same guarantee. > If for whatever reason we need to break it we can do it in major version > like every other open source project does. > > >> * Does Flink expose dependencies’ APIs in other places? Since this >> exposes the Netty API, will this make it difficult to upgrade Netty? >> > I don't expect breaking changes between minor versions so such cases there > will be no issues. If there is a breaking change in major version > we need to wait Flink major version too. > > >> * I share Till's concern about multiple factories – other HTTP middleware >> frameworks commonly support chaining middlewares. Since the proposed API >> does not include these features/guarantee ordering, do you see any reason >> to allow more than one factory? >> > I personally can't come up with a use-case where ordering is a must. I'm > not telling that this is not a valid use-case but adding a feature w/o > business rationale would include the maintenance cost (though I'm open to > add). > As I've seen Till also can't give example for that (please see the doc > comments). If you have anything in mind please share it and we can add > priority to the API. > There is another option too, namely we can be defensive and we can add the > priority right now. I would do this only if everybody states in mail that > it would be the best option, > otherwise I would stick to the original plan. > > >> >> Best, >> Austin >> >> On Tue, Jun 29, 2021 at 8:55 AM Márton Balassi <balassi.mar...@gmail.com> >> wrote: >> >>> Hi all, >>> >>> I commend Konstantin and Till when it comes to standing up for the >>> community values. >>> >>> Based on your feedback we are withdrawing the original proposal and >>> attaching a more general custom netty handler API proposal [1] written by >>> G. The change necessary to the Flink repository is approximately 500 >>> lines >>> of code. [2] >>> >>> Please let us focus on discussing the details of this API and whether it >>> covers the necessary use cases. >>> >>> [1] >>> >>> https://docs.google.com/document/d/1Idnw8YauMK1x_14iv0rVF0Hqm58J6Dg-hi-hEuL6hwM/edit#heading=h.ijcbce3c5gip >>> [2] >>> >>> https://github.com/gaborgsomogyi/flink/commit/942f23679ac21428bb87fc85557b9b443fcaf310 >>> >>> Thanks, >>> Marton >>> >>> On Wed, Jun 23, 2021 at 9:36 PM Austin Cawley-Edwards < >>> austin.caw...@gmail.com> wrote: >>> >>> > Hi all, >>> > >>> > Thanks, Konstantin and Till, for guiding the discussion. >>> > >>> > I was not aware of the results of the call with Konstantin and was >>> > attempting to resolve the unanswered questions before more, potentially >>> > fruitless, work was done. >>> > >>> > I am also looking forward to the coming proposal, as well as >>> increasing my >>> > understanding of this specific use case + its limitations! >>> > >>> > Best, >>> > Austin >>> > >>> > On Tue, Jun 22, 2021 at 6:32 AM Till Rohrmann <trohrm...@apache.org> >>> > wrote: >>> > >>> > > Hi everyone, >>> > > >>> > > I do like the idea of keeping the actual change outside of Flink but >>> to >>> > > enable Flink to support such a use case (different authentication >>> > > mechanisms). I think this is a good compromise for the community that >>> > > combines long-term maintainability with support for new use-cases. I >>> am >>> > > looking forward to your proposal. >>> > > >>> > > I also want to second Konstantin here that the tone of your last >>> email, >>> > > Marton, does not reflect the values and manners of the Flink >>> community >>> > and >>> > > is not representative of how we conduct discussions. Especially, the >>> more >>> > > senior community members should know this and act accordingly in >>> order to >>> > > be good role models for others in the community. Technical >>> discussions >>> > > should not be decided by who wields presumably the greatest >>> authority but >>> > > by the soundness of arguments and by what is the best solution for a >>> > > problem. >>> > > >>> > > Let us now try to find the best solution for the problem at hand! >>> > > >>> > > Cheers, >>> > > Till >>> > > >>> > > On Tue, Jun 22, 2021 at 11:24 AM Konstantin Knauf <kna...@apache.org >>> > >>> > > wrote: >>> > > >>> > > > Hi everyone, >>> > > > >>> > > > First, Marton and I had a brief conversation yesterday offline and >>> > > > discussed exploring the approach of exposing the authentication >>> > > > functionality via an API. So, I am looking forward to your >>> proposal in >>> > > that >>> > > > direction. The benefit of such a solution would be that it is >>> > extensible >>> > > > for others and it does add a smaller maintenance (in particular >>> > testing) >>> > > > footprint to Apache Flink itself. If we end up going down this >>> route, >>> > > > flink-packages.org would be a great way to promote these third >>> party >>> > > > "authentication modules". >>> > > > >>> > > > Second, Marton, I understand your frustration about the long >>> discussion >>> > > on >>> > > > this "simple matter", but the condescending tone of your last mail >>> > feels >>> > > > uncalled for to me. Austin expressed a valid opinion on the topic, >>> > which >>> > > is >>> > > > based on his experience from other Open Source frameworks (CNCF >>> > mostly). >>> > > I >>> > > > am sure you agree that it is important for Apache Flink to stay >>> open >>> > and >>> > > to >>> > > > consider different approaches and ideas and I don't think it helps >>> the >>> > > > culture of discussion to shoot it down like this ("This is where >>> this >>> > > > discussion stops."). >>> > > > >>> > > > Let's continue to move this discussion forward and I am sure we'll >>> > find a >>> > > > consensus based on product and technological considerations. >>> > > > >>> > > > Thanks, >>> > > > >>> > > > Konstantin >>> > > > >>> > > > On Tue, Jun 22, 2021 at 9:31 AM Márton Balassi < >>> > balassi.mar...@gmail.com >>> > > > >>> > > > wrote: >>> > > > >>> > > > > Hi Austin, >>> > > > > >>> > > > > Thank you for your thoughts. This is where this discussion stops. >>> > This >>> > > > > email thread already contains more characters than the >>> implementation >>> > > and >>> > > > > what is needed for the next 20 years of maintenance. >>> > > > > >>> > > > > It is great that you have a view on modern solutions and thank >>> you >>> > for >>> > > > > offering your help with brainstorming solutions. I am >>> responsible for >>> > > > Flink >>> > > > > at Cloudera and we do need an implementation like this and it is >>> in >>> > > fact >>> > > > > already in production at dozens of customers. We are open to >>> adapting >>> > > > that >>> > > > > to expose a more generic API (and keeping Kerberos to our fork), >>> to >>> > > > > contribute this to the community as others have asked for it and >>> to >>> > > > protect >>> > > > > ourselves from occasionally having to update this critical >>> > > implementation >>> > > > > path based on changes in the Apache codebase. I have worked with >>> > close >>> > > > to a >>> > > > > hundred Big Data customers as a consultant and an engineering >>> manager >>> > > and >>> > > > > committed hundreds of changes to Apache Flink over the past >>> decade, >>> > > > please >>> > > > > trust my judgement on a simple matter like this. >>> > > > > >>> > > > > Please forgive me for referencing authority, this discussion was >>> > > getting >>> > > > > out of hand. Please keep vigilant. >>> > > > > >>> > > > > Best, >>> > > > > Marton >>> > > > > >>> > > > > On Mon, Jun 21, 2021 at 10:50 PM Austin Cawley-Edwards < >>> > > > > austin.caw...@gmail.com> wrote: >>> > > > > >>> > > > > > Hi Gabor + Marton, >>> > > > > > >>> > > > > > I don't believe that the issue with this proposal is the >>> specific >>> > > > > mechanism >>> > > > > > proposed (Kerberos), but rather that it is not the level to >>> > implement >>> > > > it >>> > > > > at >>> > > > > > (Flink). I'm just one voice, so please take this with a grain >>> of >>> > > salt. >>> > > > > > >>> > > > > > In the other solutions previously noted there is no need to >>> > > instrument >>> > > > > > Flink which, in addition to reducing the maintenance burden, >>> > > provides a >>> > > > > > better, decoupled end result. >>> > > > > > >>> > > > > > IMO we should not add any new API in Flink for this use case. I >>> > think >>> > > > it >>> > > > > is >>> > > > > > unfortunate and sympathize with the work that has already been >>> done >>> > > on >>> > > > > this >>> > > > > > feature – perhaps we could brainstorm ways to run this >>> alongside >>> > > Flink >>> > > > in >>> > > > > > your setup. Again, I don't think the proposed solution of an >>> > agnostic >>> > > > API >>> > > > > > would not work, nor is it a bad idea, but is not one that will >>> make >>> > > > Flink >>> > > > > > more compatible with the modern solutions to this problem. >>> > > > > > >>> > > > > > Best, >>> > > > > > Austin >>> > > > > > >>> > > > > > On Mon, Jun 21, 2021 at 2:18 PM Márton Balassi < >>> > > > balassi.mar...@gmail.com >>> > > > > > >>> > > > > > wrote: >>> > > > > > >>> > > > > > > Hi team, >>> > > > > > > >>> > > > > > > Thank you for your input. Based on this discussion I agree >>> with G >>> > > > that >>> > > > > > > selecting and standardizing on a specific strong >>> authentication >>> > > > > mechanism >>> > > > > > > is more challenging than the whole rest of the scope of this >>> > > > > > authentication >>> > > > > > > story. :-) I suggest that G and I go back to the drawing >>> board >>> > and >>> > > > come >>> > > > > > up >>> > > > > > > with an API that can support multiple authentication >>> mechanisms, >>> > > and >>> > > > we >>> > > > > > > would only merge said API to Flink. Specific implementations >>> of >>> > it >>> > > > can >>> > > > > be >>> > > > > > > maintained outside of the project. This way we tackle the >>> main >>> > > > > challenge >>> > > > > > in >>> > > > > > > a truly minimal way. >>> > > > > > > >>> > > > > > > Best, >>> > > > > > > Marton >>> > > > > > > >>> > > > > > > On Mon, Jun 21, 2021 at 4:18 PM Gabor Somogyi < >>> > > > > gabor.g.somo...@gmail.com >>> > > > > > > >>> > > > > > > wrote: >>> > > > > > > >>> > > > > > > > Hi All, >>> > > > > > > > >>> > > > > > > > We see that adding any kind of specific authentication >>> raises >>> > > more >>> > > > > > > > questions than answers. >>> > > > > > > > What would be if a generic API would be added without any >>> real >>> > > > > > > > authentication logic? >>> > > > > > > > That way every provider can add its own protocol >>> implementation >>> > > as >>> > > > > > > > additional jar. >>> > > > > > > > >>> > > > > > > > BR, >>> > > > > > > > G >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > On Thu, Jun 17, 2021 at 7:53 PM Austin Cawley-Edwards < >>> > > > > > > > austin.caw...@gmail.com> wrote: >>> > > > > > > > >>> > > > > > > >> Hi all, >>> > > > > > > >> >>> > > > > > > >> Sorry to be joining the conversation late. I'm also on the >>> > side >>> > > of >>> > > > > > > >> Konstantin, generally, in that this seems to not be a core >>> > goal >>> > > of >>> > > > > > Flink >>> > > > > > > >> as >>> > > > > > > >> a project and adds a maintenance burden. >>> > > > > > > >> >>> > > > > > > >> Would another con of Kerberos be that is likely a fading >>> > project >>> > > > in >>> > > > > > > terms >>> > > > > > > >> of network security? (serious question, please correct me >>> if >>> > > there >>> > > > > is >>> > > > > > > >> reason to believe it is gaining adoption) >>> > > > > > > >> >>> > > > > > > >> The point about Kerberos being independent of >>> infrastructure >>> > is >>> > > a >>> > > > > good >>> > > > > > > one >>> > > > > > > >> but is something that is also solved by modern sidecar >>> > proxies + >>> > > > > > service >>> > > > > > > >> meshes that can run across Kubernetes and bare-metal. >>> These >>> > > > > solutions >>> > > > > > > also >>> > > > > > > >> handle certificate provisioning, rotation, etc. in >>> addition to >>> > > > > > > >> higher-level >>> > > > > > > >> authorization policies. Some examples of projects with >>> this >>> > > > > "universal >>> > > > > > > >> infrastructure support" are Kuma[1] (CNCF Sandbox, I'm a >>> > > > maintainer) >>> > > > > > and >>> > > > > > > >> Istio[2] (Google). >>> > > > > > > >> >>> > > > > > > >> Wondering out loud: has anyone tried to run Flink on top >>> of >>> > > > > cilium[3], >>> > > > > > > >> which also provides zero-trust networking at the kernel >>> level >>> > > > > without >>> > > > > > > >> needing to instrument applications? This currently only >>> runs >>> > on >>> > > > > > > Kubernetes >>> > > > > > > >> on Linux, so that's a major limitation, but solves many >>> of the >>> > > > > request >>> > > > > > > >> forging concerns at all levels. >>> > > > > > > >> >>> > > > > > > >> Thanks, >>> > > > > > > >> Austin >>> > > > > > > >> >>> > > > > > > >> [1]: https://kuma.io/docs/1.1.6/quickstart/universal/ >>> > > > > > > >> [2]: >>> > > https://istio.io/latest/docs/setup/install/virtual-machine/ >>> > > > > > > >> [3]: https://cilium.io/ >>> > > > > > > >> >>> > > > > > > >> On Thu, Jun 17, 2021 at 1:50 PM Till Rohrmann < >>> > > > trohrm...@apache.org >>> > > > > > >>> > > > > > > >> wrote: >>> > > > > > > >> >>> > > > > > > >> > I left some comments in the Google document. It would be >>> > great >>> > > > if >>> > > > > > > >> > someone from the community with security experience >>> could >>> > also >>> > > > > take >>> > > > > > a >>> > > > > > > >> look >>> > > > > > > >> > at it. Maybe Eron you have an opinion on the topic. >>> > > > > > > >> > >>> > > > > > > >> > Cheers, >>> > > > > > > >> > Till >>> > > > > > > >> > >>> > > > > > > >> > On Thu, Jun 17, 2021 at 6:57 PM Till Rohrmann < >>> > > > > trohrm...@apache.org >>> > > > > > > >>> > > > > > > >> > wrote: >>> > > > > > > >> > >>> > > > > > > >> > > Hi Gabor, >>> > > > > > > >> > > >>> > > > > > > >> > > I haven't found time to look into the updated FLIP >>> yet. >>> > I'll >>> > > > try >>> > > > > > to >>> > > > > > > >> do it >>> > > > > > > >> > > asap. >>> > > > > > > >> > > >>> > > > > > > >> > > Cheers, >>> > > > > > > >> > > Till >>> > > > > > > >> > > >>> > > > > > > >> > > On Wed, Jun 16, 2021 at 9:35 PM Konstantin Knauf < >>> > > > > > kna...@apache.org >>> > > > > > > > >>> > > > > > > >> > > wrote: >>> > > > > > > >> > > >>> > > > > > > >> > >> Hi Gabor, >>> > > > > > > >> > >> >>> > > > > > > >> > >> > However representing Kerberos as completely new >>> feature >>> > > is >>> > > > > not >>> > > > > > > true >>> > > > > > > >> > >> because >>> > > > > > > >> > >> it's already in since Flink makes authentication at >>> least >>> > > > with >>> > > > > > HDFS >>> > > > > > > >> and >>> > > > > > > >> > >> Hbase through Kerberos. >>> > > > > > > >> > >> >>> > > > > > > >> > >> True, that is one way to look at it, but there are >>> > > > differences, >>> > > > > > > too: >>> > > > > > > >> > >> Control Plane vs Data Plane, Core vs Connectors. >>> > > > > > > >> > >> >>> > > > > > > >> > >> > Adding OIDC or OAuth2 has the exact same concerns >>> what >>> > > > you've >>> > > > > > > guys >>> > > > > > > >> > just >>> > > > > > > >> > >> raised. Why exactly these? If you think this would be >>> > > > > beneficial >>> > > > > > we >>> > > > > > > >> can >>> > > > > > > >> > >> discuss it in detail >>> > > > > > > >> > >> >>> > > > > > > >> > >> That's exactly my point. Once we start adding authx >>> > > support, >>> > > > we >>> > > > > > > will >>> > > > > > > >> > >> sooner or later discuss other options besides >>> Kerberos, >>> > > too. >>> > > > A >>> > > > > > user >>> > > > > > > >> who >>> > > > > > > >> > >> would like to use OAuth can not easily use Kerberos, >>> > right? >>> > > > > > > >> > >> That is one of the reasons I am skeptical about >>> adding >>> > > > initial >>> > > > > > > authx >>> > > > > > > >> > >> support. >>> > > > > > > >> > >> >>> > > > > > > >> > >> > Related authorization you've mentioned it can be >>> > > > complicated >>> > > > > > over >>> > > > > > > >> > time. >>> > > > > > > >> > >> Can >>> > > > > > > >> > >> you show us an example? We've knowledge with couple >>> of >>> > open >>> > > > > > source >>> > > > > > > >> > >> components >>> > > > > > > >> > >> but authorization was never a horror complex story. I >>> > > > > personally >>> > > > > > > have >>> > > > > > > >> > the >>> > > > > > > >> > >> most experience with Spark which I think is quite >>> simple >>> > > and >>> > > > > > > stable. >>> > > > > > > >> > Users >>> > > > > > > >> > >> can be viewers/admins >>> > > > > > > >> > >> and jobs started by others can't be modified. If you >>> can >>> > > > share >>> > > > > an >>> > > > > > > >> > example >>> > > > > > > >> > >> over-complication we can discuss on facts. >>> > > > > > > >> > >> >>> > > > > > > >> > >> Authorization is a new aspect that needs to be >>> considered >>> > > for >>> > > > > > every >>> > > > > > > >> > >> addition to the REST API. In the future users might >>> ask >>> > for >>> > > > > > > >> additional >>> > > > > > > >> > >> roles (e.g. an editor), user-defined roles and you've >>> > > already >>> > > > > > > >> mentioned >>> > > > > > > >> > >> job-level permissions yourself. And keep in mind that >>> > there >>> > > > > might >>> > > > > > > >> also >>> > > > > > > >> > be >>> > > > > > > >> > >> larger additions in the future like the >>> > flink-sql-gateway. >>> > > > > > > >> Contributions >>> > > > > > > >> > >> like this become more expensive the more aspects we >>> need >>> > to >>> > > > > > > consider. >>> > > > > > > >> > >> >>> > > > > > > >> > >> In general, I believe, it is important that the >>> community >>> > > > > focuses >>> > > > > > > its >>> > > > > > > >> > >> efforts where we can generate the most value to the >>> user >>> > > and >>> > > > - >>> > > > > > > >> > personally - >>> > > > > > > >> > >> I don't think there is much to gain by extending >>> Flink's >>> > > > scope >>> > > > > in >>> > > > > > > >> that >>> > > > > > > >> > >> direction. Of course, this is not black and white and >>> > there >>> > > > are >>> > > > > > > other >>> > > > > > > >> > valid >>> > > > > > > >> > >> opinions. >>> > > > > > > >> > >> >>> > > > > > > >> > >> Thanks, >>> > > > > > > >> > >> >>> > > > > > > >> > >> Konstantin >>> > > > > > > >> > >> >>> > > > > > > >> > >> On Wed, Jun 16, 2021 at 7:38 PM Gabor Somogyi < >>> > > > > > > >> > gabor.g.somo...@gmail.com> >>> > > > > > > >> > >> wrote: >>> > > > > > > >> > >> >>> > > > > > > >> > >>> Hi Konstantin, >>> > > > > > > >> > >>> >>> > > > > > > >> > >>> Thanks for the response. Related new feature >>> > introduction >>> > > in >>> > > > > > case >>> > > > > > > of >>> > > > > > > >> > >>> Basic >>> > > > > > > >> > >>> auth I tend to agree, anything else can be chosen. >>> > > > > > > >> > >>> >>> > > > > > > >> > >>> However representing Kerberos as completely new >>> feature >>> > is >>> > > > not >>> > > > > > > true >>> > > > > > > >> > >>> because >>> > > > > > > >> > >>> it's already in since Flink makes authentication at >>> > least >>> > > > with >>> > > > > > > HDFS >>> > > > > > > >> and >>> > > > > > > >> > >>> Hbase through Kerberos. >>> > > > > > > >> > >>> The main problem with the actual Kerberos >>> implementation >>> > > is >>> > > > > that >>> > > > > > > it >>> > > > > > > >> > >>> contains several bugs and only partially >>> implemented. >>> > > > > Following >>> > > > > > > your >>> > > > > > > >> > >>> suggestion can we agree that we >>> > > > > > > >> > >>> skip the Basic auth implementation and finish an >>> already >>> > > > > started >>> > > > > > > >> > Kerberos >>> > > > > > > >> > >>> story by adding History Server and Job Dashboard >>> > > > > authentication? >>> > > > > > > >> > >>> >>> > > > > > > >> > >>> Adding OIDC or OAuth2 has the exact same concerns >>> what >>> > > > you've >>> > > > > > guys >>> > > > > > > >> just >>> > > > > > > >> > >>> raised. Why exactly these? If you think this would >>> be >>> > > > > beneficial >>> > > > > > > we >>> > > > > > > >> can >>> > > > > > > >> > >>> discuss it in detail >>> > > > > > > >> > >>> but as a side story it would be good to finish a >>> halfway >>> > > > done >>> > > > > > > >> Kerberos >>> > > > > > > >> > >>> story. >>> > > > > > > >> > >>> >>> > > > > > > >> > >>> Related authorization you've mentioned it can be >>> > > complicated >>> > > > > > over >>> > > > > > > >> time. >>> > > > > > > >> > >>> Can >>> > > > > > > >> > >>> you show us an example? We've knowledge with couple >>> of >>> > > open >>> > > > > > source >>> > > > > > > >> > >>> components >>> > > > > > > >> > >>> but authorization was never a horror complex story. >>> I >>> > > > > personally >>> > > > > > > >> have >>> > > > > > > >> > the >>> > > > > > > >> > >>> most experience with Spark which I think is quite >>> simple >>> > > and >>> > > > > > > stable. >>> > > > > > > >> > >>> Users >>> > > > > > > >> > >>> can be viewers/admins >>> > > > > > > >> > >>> and jobs started by others can't be modified. If >>> you can >>> > > > share >>> > > > > > an >>> > > > > > > >> > example >>> > > > > > > >> > >>> over-complication we can discuss on facts. >>> > > > > > > >> > >>> >>> > > > > > > >> > >>> Thank you in advance! >>> > > > > > > >> > >>> >>> > > > > > > >> > >>> BR, >>> > > > > > > >> > >>> G >>> > > > > > > >> > >>> >>> > > > > > > >> > >>> >>> > > > > > > >> > >>> On Wed, Jun 16, 2021 at 5:42 PM Konstantin Knauf < >>> > > > > > > kna...@apache.org >>> > > > > > > >> > >>> > > > > > > >> > >>> wrote: >>> > > > > > > >> > >>> >>> > > > > > > >> > >>> > Hi everyone, >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > sorry for joining late and thanks for the >>> insightful >>> > > > > > discussion. >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > In general, I'd personally prefer not to increase >>> the >>> > > > > surface >>> > > > > > > >> area of >>> > > > > > > >> > >>> > Apache Flink unless there is a good reason. It >>> seems >>> > we >>> > > > all >>> > > > > > > agree >>> > > > > > > >> > that >>> > > > > > > >> > >>> > authx is not part of the core value proposition of >>> > > Apache >>> > > > > > Flink, >>> > > > > > > >> so >>> > > > > > > >> > if >>> > > > > > > >> > >>> we >>> > > > > > > >> > >>> > can delegate this problem to a more specialized >>> tool, >>> > I >>> > > am >>> > > > > in >>> > > > > > > >> favor >>> > > > > > > >> > of >>> > > > > > > >> > >>> > that. Apache Flink is already huge and a lot of >>> work >>> > > goes >>> > > > > into >>> > > > > > > >> > >>> maintenance, >>> > > > > > > >> > >>> > so I personally have become more sensitive to this >>> > > aspect >>> > > > > over >>> > > > > > > >> time. >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > If we add support for Basic Auth and Kerberos now, >>> > users >>> > > > > will >>> > > > > > > >> sooner >>> > > > > > > >> > or >>> > > > > > > >> > >>> > later ask for OIDC, LDAP, SAML,... I acknowledge >>> that >>> > > > > Kerberos >>> > > > > > > is >>> > > > > > > >> > >>> widely >>> > > > > > > >> > >>> > used in the corporate, on-premises context, but >>> isn't >>> > > the >>> > > > > > focus >>> > > > > > > >> > moving >>> > > > > > > >> > >>> more >>> > > > > > > >> > >>> > towards more web-friendly standards like >>> OIDC/OAuth >>> > 2.0? >>> > > > If >>> > > > > we >>> > > > > > > >> only >>> > > > > > > >> > >>> want to >>> > > > > > > >> > >>> > support a single protocol, there is an argument >>> to be >>> > > made >>> > > > > > that >>> > > > > > > it >>> > > > > > > >> > >>> should >>> > > > > > > >> > >>> > be OIDC and Dex [1,2] as a bridge to everything >>> else. >>> > > Have >>> > > > > > OIDC >>> > > > > > > or >>> > > > > > > >> > >>> OAuth2 >>> > > > > > > >> > >>> > been considered instead of Kerberos? How do you >>> see >>> > the >>> > > > > market >>> > > > > > > >> > moving? >>> > > > > > > >> > >>> But >>> > > > > > > >> > >>> > as I said before, in my opinion we can generate >>> more >>> > > value >>> > > > > by >>> > > > > > > >> > investing >>> > > > > > > >> > >>> > into other areas of Apache Flink. >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > Authorization also has the potential to become >>> more >>> > > > > > fine-grained >>> > > > > > > >> and >>> > > > > > > >> > >>> > complex over time: you already mentioned >>> restricting >>> > the >>> > > > > > actions >>> > > > > > > >> > that a >>> > > > > > > >> > >>> > specific user can do in a cluster. >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > Cheers, >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > Konstantin >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > [1] https://github.com/dexidp/dex >>> > > > > > > >> > >>> > [2] https://github.com/dexidp/dex/issues/1903 >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > On Wed, Jun 16, 2021 at 11:44 AM Gabor Somogyi < >>> > > > > > > >> > >>> gabor.g.somo...@gmail.com> >>> > > > > > > >> > >>> > wrote: >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> >> Hi Till, >>> > > > > > > >> > >>> >> >>> > > > > > > >> > >>> >> Did you have the chance to take a look at the >>> doc? >>> > Not >>> > > > yet >>> > > > > > seen >>> > > > > > > >> any >>> > > > > > > >> > >>> >> update. >>> > > > > > > >> > >>> >> >>> > > > > > > >> > >>> >> BR, >>> > > > > > > >> > >>> >> G >>> > > > > > > >> > >>> >> >>> > > > > > > >> > >>> >> >>> > > > > > > >> > >>> >> On Wed, Jun 9, 2021 at 1:43 PM Till Rohrmann < >>> > > > > > > >> trohrm...@apache.org> >>> > > > > > > >> > >>> >> wrote: >>> > > > > > > >> > >>> >> >>> > > > > > > >> > >>> >> > Thanks for the update Gabor. I'll take a look >>> and >>> > > > respond >>> > > > > > in >>> > > > > > > >> the >>> > > > > > > >> > >>> >> document. >>> > > > > > > >> > >>> >> > >>> > > > > > > >> > >>> >> > Cheers, >>> > > > > > > >> > >>> >> > Till >>> > > > > > > >> > >>> >> > >>> > > > > > > >> > >>> >> > On Wed, Jun 9, 2021 at 12:59 PM Gabor Somogyi < >>> > > > > > > >> > >>> >> gabor.g.somo...@gmail.com> >>> > > > > > > >> > >>> >> > wrote: >>> > > > > > > >> > >>> >> > >>> > > > > > > >> > >>> >> >> Hi Till, >>> > > > > > > >> > >>> >> >> >>> > > > > > > >> > >>> >> >> Your proxy suggestion has been considered >>> in-depth >>> > > and >>> > > > > > > updated >>> > > > > > > >> > the >>> > > > > > > >> > >>> FLIP >>> > > > > > > >> > >>> >> >> accordingly. >>> > > > > > > >> > >>> >> >> We've considered 2 proxy implementation >>> (Nginx and >>> > > > > Squid) >>> > > > > > > but >>> > > > > > > >> > >>> according >>> > > > > > > >> > >>> >> >> to our analysis and testing it's not suitable >>> for >>> > > the >>> > > > > > > >> mentioned >>> > > > > > > >> > >>> >> use-cases. >>> > > > > > > >> > >>> >> >> Please take a look at the rejected >>> alternatives >>> > for >>> > > > > > detailed >>> > > > > > > >> > >>> >> explanation. >>> > > > > > > >> > >>> >> >> >>> > > > > > > >> > >>> >> >> Thanks for your time in advance! >>> > > > > > > >> > >>> >> >> >>> > > > > > > >> > >>> >> >> BR, >>> > > > > > > >> > >>> >> >> G >>> > > > > > > >> > >>> >> >> >>> > > > > > > >> > >>> >> >> >>> > > > > > > >> > >>> >> >> On Fri, Jun 4, 2021 at 3:31 PM Till Rohrmann < >>> > > > > > > >> > trohrm...@apache.org >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> >> >> wrote: >>> > > > > > > >> > >>> >> >> >>> > > > > > > >> > >>> >> >>> As I've said I am not a security expert and >>> > that's >>> > > > why >>> > > > > I >>> > > > > > > >> have to >>> > > > > > > >> > >>> ask >>> > > > > > > >> > >>> >> for >>> > > > > > > >> > >>> >> >>> clarification, Gabor. You are saying that if >>> we >>> > > > > > configure a >>> > > > > > > >> > >>> >> truststore for >>> > > > > > > >> > >>> >> >>> the REST endpoint with a single trusted >>> > certificate >>> > > > > which >>> > > > > > > has >>> > > > > > > >> > been >>> > > > > > > >> > >>> >> >>> generated by the operator of the Flink >>> cluster, >>> > > then >>> > > > > the >>> > > > > > > >> > attacker >>> > > > > > > >> > >>> can >>> > > > > > > >> > >>> >> >>> generate a new certificate, sign it and then >>> talk >>> > > to >>> > > > > the >>> > > > > > > >> Flink >>> > > > > > > >> > >>> >> cluster if >>> > > > > > > >> > >>> >> >>> he has access to the node on which the REST >>> > > endpoint >>> > > > > > runs? >>> > > > > > > My >>> > > > > > > >> > >>> >> understanding >>> > > > > > > >> > >>> >> >>> was that you need the corresponding private >>> key >>> > > which >>> > > > > in >>> > > > > > my >>> > > > > > > >> > >>> proposed >>> > > > > > > >> > >>> >> setup >>> > > > > > > >> > >>> >> >>> would be under the control of the operator as >>> > well >>> > > > > (e.g. >>> > > > > > > >> stored >>> > > > > > > >> > >>> in a >>> > > > > > > >> > >>> >> >>> keystore on the same machine but guarded by >>> some >>> > > > > secret). >>> > > > > > > >> That >>> > > > > > > >> > way >>> > > > > > > >> > >>> >> (if I am >>> > > > > > > >> > >>> >> >>> not mistaken), only the entity which has >>> access >>> > to >>> > > > the >>> > > > > > > >> keystore >>> > > > > > > >> > is >>> > > > > > > >> > >>> >> able to >>> > > > > > > >> > >>> >> >>> talk to the Flink cluster. >>> > > > > > > >> > >>> >> >>> >>> > > > > > > >> > >>> >> >>> Maybe we are also getting our wires crossed >>> here >>> > > and >>> > > > > are >>> > > > > > > >> talking >>> > > > > > > >> > >>> about >>> > > > > > > >> > >>> >> >>> different things. >>> > > > > > > >> > >>> >> >>> >>> > > > > > > >> > >>> >> >>> Thanks for listing the pros and cons of >>> Kerberos. >>> > > > > > > Concerning >>> > > > > > > >> > what >>> > > > > > > >> > >>> >> other >>> > > > > > > >> > >>> >> >>> authentication mechanisms are used in the >>> > > industry, I >>> > > > > am >>> > > > > > > not >>> > > > > > > >> > 100% >>> > > > > > > >> > >>> >> sure. >>> > > > > > > >> > >>> >> >>> >>> > > > > > > >> > >>> >> >>> Cheers, >>> > > > > > > >> > >>> >> >>> Till >>> > > > > > > >> > >>> >> >>> >>> > > > > > > >> > >>> >> >>> On Fri, Jun 4, 2021 at 11:09 AM Gabor >>> Somogyi < >>> > > > > > > >> > >>> >> gabor.g.somo...@gmail.com> >>> > > > > > > >> > >>> >> >>> wrote: >>> > > > > > > >> > >>> >> >>> >>> > > > > > > >> > >>> >> >>>> > I did not mean for the user to sign its >>> own >>> > > > > > certificates >>> > > > > > > >> but >>> > > > > > > >> > >>> for >>> > > > > > > >> > >>> >> the >>> > > > > > > >> > >>> >> >>>> operator of the cluster. Once the user >>> request >>> > > hits >>> > > > > the >>> > > > > > > >> proxy, >>> > > > > > > >> > it >>> > > > > > > >> > >>> >> should no >>> > > > > > > >> > >>> >> >>>> longer be under his control. I think I do >>> not >>> > > fully >>> > > > > > > >> understand >>> > > > > > > >> > >>> yet >>> > > > > > > >> > >>> >> why this >>> > > > > > > >> > >>> >> >>>> would not work. >>> > > > > > > >> > >>> >> >>>> I said it's not solving the authentication >>> > problem >>> > > > > over >>> > > > > > > any >>> > > > > > > >> > >>> proxy. >>> > > > > > > >> > >>> >> Even >>> > > > > > > >> > >>> >> >>>> if the operator is signing the certificate >>> one >>> > can >>> > > > > have >>> > > > > > > >> access >>> > > > > > > >> > >>> to an >>> > > > > > > >> > >>> >> >>>> internal node. >>> > > > > > > >> > >>> >> >>>> Such case anybody can craft certificates >>> which >>> > is >>> > > > > > accepted >>> > > > > > > >> by >>> > > > > > > >> > the >>> > > > > > > >> > >>> >> >>>> server. When it's accepted a bad guy can >>> cancel >>> > > jobs >>> > > > > > > causing >>> > > > > > > >> > huge >>> > > > > > > >> > >>> >> impacts. >>> > > > > > > >> > >>> >> >>>> >>> > > > > > > >> > >>> >> >>>> > Also, I am missing a bit the comparison of >>> > > > Kerberos >>> > > > > to >>> > > > > > > >> other >>> > > > > > > >> > >>> >> >>>> authentication mechanisms and why they were >>> > > rejected >>> > > > > in >>> > > > > > > >> favour >>> > > > > > > >> > of >>> > > > > > > >> > >>> >> Kerberos. >>> > > > > > > >> > >>> >> >>>> PROS: >>> > > > > > > >> > >>> >> >>>> * Since it's not depending on cloud provider >>> > > and/or >>> > > > > k8s >>> > > > > > or >>> > > > > > > >> > >>> bare-metal >>> > > > > > > >> > >>> >> >>>> etc. deployment it's the biggest plus >>> > > > > > > >> > >>> >> >>>> * Centralized with tools and no need to >>> write >>> > tons >>> > > > of >>> > > > > > > tools >>> > > > > > > >> > >>> around >>> > > > > > > >> > >>> >> >>>> * There are clients/tools on almost all >>> OS-es >>> > and >>> > > > > > several >>> > > > > > > >> > >>> languages >>> > > > > > > >> > >>> >> >>>> * Super huge users are using it for years in >>> > > > > production >>> > > > > > > w/o >>> > > > > > > >> > huge >>> > > > > > > >> > >>> >> issues >>> > > > > > > >> > >>> >> >>>> * Provides cross-realm trust possibility >>> amongst >>> > > > other >>> > > > > > > >> features >>> > > > > > > >> > >>> >> >>>> * Several open source components using it >>> which >>> > > > could >>> > > > > > > >> increase >>> > > > > > > >> > >>> >> >>>> compatibility >>> > > > > > > >> > >>> >> >>>> >>> > > > > > > >> > >>> >> >>>> CONS: >>> > > > > > > >> > >>> >> >>>> * Not everybody using kerberos >>> > > > > > > >> > >>> >> >>>> * It would increase the code footprint but >>> this >>> > is >>> > > > > true >>> > > > > > > for >>> > > > > > > >> > many >>> > > > > > > >> > >>> >> >>>> features (as a side note I'm here to >>> maintain >>> > it) >>> > > > > > > >> > >>> >> >>>> >>> > > > > > > >> > >>> >> >>>> Feel free to add your points because it only >>> > > > > represents >>> > > > > > a >>> > > > > > > >> > single >>> > > > > > > >> > >>> >> >>>> viewpoint. >>> > > > > > > >> > >>> >> >>>> Also if you have any better option for >>> strong >>> > > > > > > authentication >>> > > > > > > >> > >>> please >>> > > > > > > >> > >>> >> >>>> share it and we can consider the pros/cons >>> here. >>> > > > > > > >> > >>> >> >>>> >>> > > > > > > >> > >>> >> >>>> BR, >>> > > > > > > >> > >>> >> >>>> G >>> > > > > > > >> > >>> >> >>>> >>> > > > > > > >> > >>> >> >>>> >>> > > > > > > >> > >>> >> >>>> On Fri, Jun 4, 2021 at 10:32 AM Till >>> Rohrmann < >>> > > > > > > >> > >>> trohrm...@apache.org> >>> > > > > > > >> > >>> >> >>>> wrote: >>> > > > > > > >> > >>> >> >>>> >>> > > > > > > >> > >>> >> >>>>> I did not mean for the user to sign its own >>> > > > > > certificates >>> > > > > > > >> but >>> > > > > > > >> > >>> for the >>> > > > > > > >> > >>> >> >>>>> operator of the cluster. Once the user >>> request >>> > > hits >>> > > > > the >>> > > > > > > >> proxy, >>> > > > > > > >> > >>> it >>> > > > > > > >> > >>> >> should no >>> > > > > > > >> > >>> >> >>>>> longer be under his control. I think I do >>> not >>> > > fully >>> > > > > > > >> understand >>> > > > > > > >> > >>> yet >>> > > > > > > >> > >>> >> why this >>> > > > > > > >> > >>> >> >>>>> would not work. >>> > > > > > > >> > >>> >> >>>>> >>> > > > > > > >> > >>> >> >>>>> What I would like to avoid is to add more >>> > > > complexity >>> > > > > > into >>> > > > > > > >> > Flink >>> > > > > > > >> > >>> if >>> > > > > > > >> > >>> >> >>>>> there is an easy solution which fulfills >>> the >>> > > > > > > requirements. >>> > > > > > > >> > >>> That's >>> > > > > > > >> > >>> >> why I >>> > > > > > > >> > >>> >> >>>>> would like to exercise thoroughly through >>> the >>> > > > > different >>> > > > > > > >> > >>> >> alternatives. Also, >>> > > > > > > >> > >>> >> >>>>> I am missing a bit the comparison of >>> Kerberos >>> > to >>> > > > > other >>> > > > > > > >> > >>> >> authentication >>> > > > > > > >> > >>> >> >>>>> mechanisms and why they were rejected in >>> favour >>> > > of >>> > > > > > > >> Kerberos. >>> > > > > > > >> > >>> >> >>>>> >>> > > > > > > >> > >>> >> >>>>> Cheers, >>> > > > > > > >> > >>> >> >>>>> Till >>> > > > > > > >> > >>> >> >>>>> >>> > > > > > > >> > >>> >> >>>>> On Fri, Jun 4, 2021 at 10:26 AM Gyula Fóra >>> < >>> > > > > > > >> gyf...@apache.org >>> > > > > > > >> > > >>> > > > > > > >> > >>> >> wrote: >>> > > > > > > >> > >>> >> >>>>> >>> > > > > > > >> > >>> >> >>>>>> Hi! >>> > > > > > > >> > >>> >> >>>>>> >>> > > > > > > >> > >>> >> >>>>>> I think there might be possible >>> alternatives >>> > but >>> > > > it >>> > > > > > > seems >>> > > > > > > >> > >>> Kerberos >>> > > > > > > >> > >>> >> on >>> > > > > > > >> > >>> >> >>>>>> the rest endpoint ticks all the right >>> boxes >>> > and >>> > > > > > > provides a >>> > > > > > > >> > >>> super >>> > > > > > > >> > >>> >> clean and >>> > > > > > > >> > >>> >> >>>>>> simple solution for strong authentication. >>> > > > > > > >> > >>> >> >>>>>> >>> > > > > > > >> > >>> >> >>>>>> I wouldn’t even consider sidecar proxies >>> etc >>> > if >>> > > we >>> > > > > can >>> > > > > > > >> solve >>> > > > > > > >> > >>> it in >>> > > > > > > >> > >>> >> >>>>>> such a simple way as proposed by G. >>> > > > > > > >> > >>> >> >>>>>> >>> > > > > > > >> > >>> >> >>>>>> Cheers >>> > > > > > > >> > >>> >> >>>>>> Gyula >>> > > > > > > >> > >>> >> >>>>>> >>> > > > > > > >> > >>> >> >>>>>> On Fri, 4 Jun 2021 at 10:03, Till >>> Rohrmann < >>> > > > > > > >> > >>> trohrm...@apache.org> >>> > > > > > > >> > >>> >> >>>>>> wrote: >>> > > > > > > >> > >>> >> >>>>>> >>> > > > > > > >> > >>> >> >>>>>>> I am not saying that we shouldn't add a >>> > strong >>> > > > > > > >> > authentication >>> > > > > > > >> > >>> >> >>>>>>> mechanism if there are good reasons for >>> it. I >>> > > > > > primarily >>> > > > > > > >> > would >>> > > > > > > >> > >>> >> like to >>> > > > > > > >> > >>> >> >>>>>>> understand the context a bit better in >>> order >>> > to >>> > > > > give >>> > > > > > > >> > qualified >>> > > > > > > >> > >>> >> feedback and >>> > > > > > > >> > >>> >> >>>>>>> come to a good decision. In order to do >>> > this, I >>> > > > > have >>> > > > > > > the >>> > > > > > > >> > >>> feeling >>> > > > > > > >> > >>> >> that we >>> > > > > > > >> > >>> >> >>>>>>> haven't fully considered all available >>> > options >>> > > > > which >>> > > > > > > are >>> > > > > > > >> on >>> > > > > > > >> > >>> the >>> > > > > > > >> > >>> >> table, tbh. >>> > > > > > > >> > >>> >> >>>>>>> >>> > > > > > > >> > >>> >> >>>>>>> Does the problem of certificate expiry >>> also >>> > > apply >>> > > > > for >>> > > > > > > >> > >>> self-signed >>> > > > > > > >> > >>> >> >>>>>>> certificates? If yes, then this should >>> then >>> > > also >>> > > > > be a >>> > > > > > > >> > problem >>> > > > > > > >> > >>> for >>> > > > > > > >> > >>> >> the >>> > > > > > > >> > >>> >> >>>>>>> internal encryption of Flink's >>> communication. >>> > > If >>> > > > > not, >>> > > > > > > >> then >>> > > > > > > >> > one >>> > > > > > > >> > >>> >> could use >>> > > > > > > >> > >>> >> >>>>>>> self-signed certificates with a longer >>> > validity >>> > > > to >>> > > > > > > solve >>> > > > > > > >> the >>> > > > > > > >> > >>> >> mentioned >>> > > > > > > >> > >>> >> >>>>>>> issue. >>> > > > > > > >> > >>> >> >>>>>>> >>> > > > > > > >> > >>> >> >>>>>>> I think you can set up Flink in such a >>> way >>> > that >>> > > > you >>> > > > > > > don't >>> > > > > > > >> > >>> have to >>> > > > > > > >> > >>> >> >>>>>>> handle all the different certificates. >>> For >>> > > > example, >>> > > > > > you >>> > > > > > > >> > could >>> > > > > > > >> > >>> >> deploy Flink >>> > > > > > > >> > >>> >> >>>>>>> with a "sidecar proxy" which is >>> responsible >>> > for >>> > > > the >>> > > > > > > >> > >>> >> authentication using an >>> > > > > > > >> > >>> >> >>>>>>> arbitrary method (e.g. Kerberos) and then >>> > bind >>> > > > the >>> > > > > > REST >>> > > > > > > >> > >>> endpoint >>> > > > > > > >> > >>> >> to a local >>> > > > > > > >> > >>> >> >>>>>>> network interface. That way, the REST >>> > endpoint >>> > > > > would >>> > > > > > > >> only be >>> > > > > > > >> > >>> >> available >>> > > > > > > >> > >>> >> >>>>>>> through the sidecar proxy. Additionally, >>> one >>> > > > could >>> > > > > > > enable >>> > > > > > > >> > SSL >>> > > > > > > >> > >>> for >>> > > > > > > >> > >>> >> this >>> > > > > > > >> > >>> >> >>>>>>> communication. Would this be a solution >>> for >>> > the >>> > > > > > > problem? >>> > > > > > > >> > >>> >> >>>>>>> >>> > > > > > > >> > >>> >> >>>>>>> Cheers, >>> > > > > > > >> > >>> >> >>>>>>> Till >>> > > > > > > >> > >>> >> >>>>>>> >>> > > > > > > >> > >>> >> >>>>>>> On Thu, Jun 3, 2021 at 10:46 PM Márton >>> > Balassi >>> > > < >>> > > > > > > >> > >>> >> >>>>>>> balassi.mar...@gmail.com> wrote: >>> > > > > > > >> > >>> >> >>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>> That is an interesting idea, Till. >>> > > > > > > >> > >>> >> >>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>> The main issue with it is that TLS >>> > > certificates >>> > > > > have >>> > > > > > > an >>> > > > > > > >> > >>> >> expiration >>> > > > > > > >> > >>> >> >>>>>>>> time, usually they get approved for a >>> couple >>> > > > > years. >>> > > > > > > >> Forcing >>> > > > > > > >> > >>> our >>> > > > > > > >> > >>> >> users to >>> > > > > > > >> > >>> >> >>>>>>>> restart jobs to reprovision TLS >>> certificates >>> > > > would >>> > > > > > be >>> > > > > > > >> weird >>> > > > > > > >> > >>> when >>> > > > > > > >> > >>> >> we could >>> > > > > > > >> > >>> >> >>>>>>>> just implement a single proper strong >>> > > > > authentication >>> > > > > > > >> > >>> mechanism >>> > > > > > > >> > >>> >> instead in a >>> > > > > > > >> > >>> >> >>>>>>>> couple hundred lines of code. :-) >>> > > > > > > >> > >>> >> >>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>> In many cases it is also impractical to >>> go >>> > the >>> > > > TLS >>> > > > > > > >> mutual >>> > > > > > > >> > >>> route, >>> > > > > > > >> > >>> >> >>>>>>>> because the Flink Dashboard can end up >>> on >>> > any >>> > > > node >>> > > > > > in >>> > > > > > > >> the >>> > > > > > > >> > >>> >> k8s/Yarn cluster >>> > > > > > > >> > >>> >> >>>>>>>> which means that we need a certificate >>> per >>> > > node >>> > > > > (due >>> > > > > > > to >>> > > > > > > >> the >>> > > > > > > >> > >>> >> mutual auth), >>> > > > > > > >> > >>> >> >>>>>>>> but if we also want to protect the >>> private >>> > key >>> > > > of >>> > > > > > > these >>> > > > > > > >> > from >>> > > > > > > >> > >>> >> users >>> > > > > > > >> > >>> >> >>>>>>>> accidentally or intentionally leaking >>> them >>> > > then >>> > > > we >>> > > > > > > need >>> > > > > > > >> > this >>> > > > > > > >> > >>> per >>> > > > > > > >> > >>> >> user. As >>> > > > > > > >> > >>> >> >>>>>>>> in we end up managing user*machine >>> number >>> > > > > > certificates >>> > > > > > > >> and >>> > > > > > > >> > >>> >> having to renew >>> > > > > > > >> > >>> >> >>>>>>>> them periodically, which albeit >>> automatable >>> > is >>> > > > > > > >> > unfortunately >>> > > > > > > >> > >>> not >>> > > > > > > >> > >>> >> yet >>> > > > > > > >> > >>> >> >>>>>>>> automated in all large organizations. >>> > > > > > > >> > >>> >> >>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>> I fully agree that TLS certificate >>> mutual >>> > > > > > > authentication >>> > > > > > > >> > has >>> > > > > > > >> > >>> its >>> > > > > > > >> > >>> >> >>>>>>>> nice properties, especially at very >>> large >>> > > > > (multiple >>> > > > > > > >> > thousand >>> > > > > > > >> > >>> >> node) clusters >>> > > > > > > >> > >>> >> >>>>>>>> - but it has its own challenges too. >>> Thanks >>> > > for >>> > > > > > > >> bringing it >>> > > > > > > >> > >>> up. >>> > > > > > > >> > >>> >> >>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>> Happy to have this added to the rejected >>> > > > > alternative >>> > > > > > > >> list >>> > > > > > > >> > so >>> > > > > > > >> > >>> that >>> > > > > > > >> > >>> >> >>>>>>>> we have the full picture documented. >>> > > > > > > >> > >>> >> >>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>> On Thu, Jun 3, 2021 at 5:52 PM Till >>> > Rohrmann < >>> > > > > > > >> > >>> >> trohrm...@apache.org> >>> > > > > > > >> > >>> >> >>>>>>>> wrote: >>> > > > > > > >> > >>> >> >>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>> I guess the idea would then be to let >>> the >>> > > proxy >>> > > > > do >>> > > > > > > the >>> > > > > > > >> > >>> >> >>>>>>>>> authentication job and only forward the >>> > > request >>> > > > > via >>> > > > > > > an >>> > > > > > > >> SSL >>> > > > > > > >> > >>> >> mutually >>> > > > > > > >> > >>> >> >>>>>>>>> encrypted connection to the Flink >>> cluster. >>> > > > Would >>> > > > > > this >>> > > > > > > >> be >>> > > > > > > >> > >>> >> possible? The >>> > > > > > > >> > >>> >> >>>>>>>>> beauty of this setup is in my opinion >>> that >>> > > this >>> > > > > > setup >>> > > > > > > >> > should >>> > > > > > > >> > >>> >> work with all >>> > > > > > > >> > >>> >> >>>>>>>>> kinds of authentication mechanisms. >>> > > > > > > >> > >>> >> >>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>> Cheers, >>> > > > > > > >> > >>> >> >>>>>>>>> Till >>> > > > > > > >> > >>> >> >>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>> On Thu, Jun 3, 2021 at 3:12 PM Gabor >>> > Somogyi >>> > > < >>> > > > > > > >> > >>> >> >>>>>>>>> gabor.g.somo...@gmail.com> wrote: >>> > > > > > > >> > >>> >> >>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>> Thanks for giving options to fulfil >>> the >>> > > need. >>> > > > > > > >> > >>> >> >>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>> Users are looking for a solution where >>> > users >>> > > > can >>> > > > > > be >>> > > > > > > >> > >>> identified >>> > > > > > > >> > >>> >> on >>> > > > > > > >> > >>> >> >>>>>>>>>> the whole cluster and restrict access >>> to >>> > > > > > > >> > resources/actions. >>> > > > > > > >> > >>> >> >>>>>>>>>> A good example for such an action is >>> > > > cancelling >>> > > > > > > other >>> > > > > > > >> > users >>> > > > > > > >> > >>> >> >>>>>>>>>> running jobs. >>> > > > > > > >> > >>> >> >>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>> * SSL does provide mutual >>> authentication >>> > but >>> > > > > when >>> > > > > > > >> > >>> >> authentication >>> > > > > > > >> > >>> >> >>>>>>>>>> passed there is no user based on >>> > > restrictions >>> > > > > can >>> > > > > > be >>> > > > > > > >> > made. >>> > > > > > > >> > >>> >> >>>>>>>>>> * The less problematic part is that >>> > > > > > > >> > generating/maintaining >>> > > > > > > >> > >>> >> short >>> > > > > > > >> > >>> >> >>>>>>>>>> time valid certificates would be a >>> hard >>> > > > (that's >>> > > > > > the >>> > > > > > > >> > reason >>> > > > > > > >> > >>> KDC >>> > > > > > > >> > >>> >> like servers >>> > > > > > > >> > >>> >> >>>>>>>>>> exist). >>> > > > > > > >> > >>> >> >>>>>>>>>> Having long time valid certificates >>> would >>> > > > widen >>> > > > > > the >>> > > > > > > >> > attack >>> > > > > > > >> > >>> >> >>>>>>>>>> surface but since the first concern is >>> > there >>> > > > > this >>> > > > > > is >>> > > > > > > >> > just a >>> > > > > > > >> > >>> >> cosmetic issue. >>> > > > > > > >> > >>> >> >>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>> All in all using TLS certificates is >>> not >>> > > > > > sufficient >>> > > > > > > in >>> > > > > > > >> > >>> these >>> > > > > > > >> > >>> >> >>>>>>>>>> environments unfortunately. >>> > > > > > > >> > >>> >> >>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>> BR, >>> > > > > > > >> > >>> >> >>>>>>>>>> G >>> > > > > > > >> > >>> >> >>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>> On Thu, Jun 3, 2021 at 12:49 PM Till >>> > > Rohrmann >>> > > > < >>> > > > > > > >> > >>> >> >>>>>>>>>> trohrm...@apache.org> wrote: >>> > > > > > > >> > >>> >> >>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>> Thanks for the information Gabor. If >>> it >>> > is >>> > > > > about >>> > > > > > > >> > securing >>> > > > > > > >> > >>> the >>> > > > > > > >> > >>> >> >>>>>>>>>>> communication between the REST >>> client and >>> > > the >>> > > > > > REST >>> > > > > > > >> > server, >>> > > > > > > >> > >>> >> then Flink >>> > > > > > > >> > >>> >> >>>>>>>>>>> already supports enabling mutual SSL >>> > > > > > authentication >>> > > > > > > >> [1]. >>> > > > > > > >> > >>> >> Would this be >>> > > > > > > >> > >>> >> >>>>>>>>>>> enough to secure the communication >>> and to >>> > > > pass >>> > > > > an >>> > > > > > > >> audit? >>> > > > > > > >> > >>> >> >>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>> [1] >>> > > > > > > >> > >>> >> >>>>>>>>>>> >>> > > > > > > >> > >>> >> >>> > > > > > > >> > >>> >>> > > > > > > >> > >>> > > > > > > >> >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/security/security-ssl/#external--rest-connectivity >>> > > > > > > >> > >>> >> >>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>> Cheers, >>> > > > > > > >> > >>> >> >>>>>>>>>>> Till >>> > > > > > > >> > >>> >> >>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>> On Thu, Jun 3, 2021 at 10:33 AM Gabor >>> > > > Somogyi < >>> > > > > > > >> > >>> >> >>>>>>>>>>> gabor.g.somo...@gmail.com> wrote: >>> > > > > > > >> > >>> >> >>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> Hi Till, >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> Since I'm working in security area >>> 10+ >>> > > years >>> > > > > let >>> > > > > > > me >>> > > > > > > >> > >>> share my >>> > > > > > > >> > >>> >> >>>>>>>>>>>> thought. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> I would like to emphasise there are >>> > > experts >>> > > > > > better >>> > > > > > > >> than >>> > > > > > > >> > >>> me >>> > > > > > > >> > >>> >> but >>> > > > > > > >> > >>> >> >>>>>>>>>>>> I have some >>> > > > > > > >> > >>> >> >>>>>>>>>>>> basics. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> The discussion is open and not >>> trying to >>> > > > tell >>> > > > > > > alone >>> > > > > > > >> > >>> things... >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > I mean if an attacker can get >>> access >>> > to >>> > > > one >>> > > > > of >>> > > > > > > the >>> > > > > > > >> > >>> >> machines, >>> > > > > > > >> > >>> >> >>>>>>>>>>>> then it >>> > > > > > > >> > >>> >> >>>>>>>>>>>> should also be possible to obtain >>> the >>> > > right >>> > > > > > > Kerberos >>> > > > > > > >> > >>> token. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> Not necessarily. For example if one >>> gets >>> > > > > access >>> > > > > > > to a >>> > > > > > > >> > >>> specific >>> > > > > > > >> > >>> >> >>>>>>>>>>>> user's >>> > > > > > > >> > >>> >> >>>>>>>>>>>> credentials then it's not possible >>> to >>> > > > > compromise >>> > > > > > > >> other >>> > > > > > > >> > >>> user's >>> > > > > > > >> > >>> >> >>>>>>>>>>>> jobs, data, >>> > > > > > > >> > >>> >> >>>>>>>>>>>> etc... >>> > > > > > > >> > >>> >> >>>>>>>>>>>> Security is like an onion, the more >>> > layers >>> > > > has >>> > > > > > > been >>> > > > > > > >> > >>> added the >>> > > > > > > >> > >>> >> >>>>>>>>>>>> more time an >>> > > > > > > >> > >>> >> >>>>>>>>>>>> attacker needs to proceed. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> At the end of the day if one is in, >>> then >>> > > > most >>> > > > > > > >> probably >>> > > > > > > >> > >>> can >>> > > > > > > >> > >>> >> find >>> > > > > > > >> > >>> >> >>>>>>>>>>>> the way but >>> > > > > > > >> > >>> >> >>>>>>>>>>>> this time is normally enough to >>> > sysadmins >>> > > or >>> > > > > > > >> security >>> > > > > > > >> > >>> >> experts to >>> > > > > > > >> > >>> >> >>>>>>>>>>>> close down the system and minimize >>> the >>> > > > damage. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> The other thing is that all tokens >>> has a >>> > > > > timeout >>> > > > > > > >> and if >>> > > > > > > >> > >>> the >>> > > > > > > >> > >>> >> >>>>>>>>>>>> token is >>> > > > > > > >> > >>> >> >>>>>>>>>>>> invalid then the attacker can't >>> proceed >>> > > > > further. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Is Kerberos also the standard >>> > > > authentication >>> > > > > > > >> protocol >>> > > > > > > >> > >>> for >>> > > > > > > >> > >>> >> >>>>>>>>>>>> Kubernetes >>> > > > > > > >> > >>> >> >>>>>>>>>>>> deployments? >>> > > > > > > >> > >>> >> >>>>>>>>>>>> Kerberos is an industry standard >>> which >>> > is >>> > > > > > > >> > >>> cloud/deployment >>> > > > > > > >> > >>> >> >>>>>>>>>>>> agnostic and it >>> > > > > > > >> > >>> >> >>>>>>>>>>>> can be used in any deployments >>> including >>> > > > k8s. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> The main intention is to use >>> kerberos in >>> > > k8s >>> > > > > > > >> > deployments >>> > > > > > > >> > >>> too >>> > > > > > > >> > >>> >> >>>>>>>>>>>> since we're >>> > > > > > > >> > >>> >> >>>>>>>>>>>> going this direction as well. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> Please see how Spark does this: >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>> > > > > > > >> > >>> >>> > > > > > > >> > >>> > > > > > > >> >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> https://spark.apache.org/docs/latest/security.html#secure-interaction-with-kubernetes >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> Last but not least the most >>> important >>> > > reason >>> > > > > to >>> > > > > > > add >>> > > > > > > >> at >>> > > > > > > >> > >>> least >>> > > > > > > >> > >>> >> >>>>>>>>>>>> one strong >>> > > > > > > >> > >>> >> >>>>>>>>>>>> authentication is that we have >>> users who >>> > > has >>> > > > > > > >> > >>> >> >>>>>>>>>>>> hard requirements on this. They're >>> doing >>> > > > > > security >>> > > > > > > >> > audits >>> > > > > > > >> > >>> and >>> > > > > > > >> > >>> >> if >>> > > > > > > >> > >>> >> >>>>>>>>>>>> they fail >>> > > > > > > >> > >>> >> >>>>>>>>>>>> then it's deal breaking. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> That is why we have added kerberos >>> at >>> > the >>> > > > > first >>> > > > > > > >> place. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> Unfortunately we >>> > > > > > > >> > >>> >> >>>>>>>>>>>> can't name them in this public list, >>> > > however >>> > > > > > > >> > >>> >> >>>>>>>>>>>> the customers who specifically >>> asked for >>> > > > this >>> > > > > > were >>> > > > > > > >> > >>> mainly in >>> > > > > > > >> > >>> >> >>>>>>>>>>>> the banking >>> > > > > > > >> > >>> >> >>>>>>>>>>>> and telco sector. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> BR, >>> > > > > > > >> > >>> >> >>>>>>>>>>>> G >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> On Thu, Jun 3, 2021 at 9:20 AM Till >>> > > > Rohrmann < >>> > > > > > > >> > >>> >> >>>>>>>>>>>> trohrm...@apache.org> wrote: >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Thanks for updating the document >>> > Márton. >>> > > > Why >>> > > > > > is >>> > > > > > > it >>> > > > > > > >> > that >>> > > > > > > >> > >>> >> banks >>> > > > > > > >> > >>> >> >>>>>>>>>>>> will >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > consider it more secure if Flink >>> comes >>> > > > with >>> > > > > > > >> Kerberos >>> > > > > > > >> > >>> >> >>>>>>>>>>>> authentication >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > (assuming a properly secured >>> setup)? I >>> > > > mean >>> > > > > if >>> > > > > > > an >>> > > > > > > >> > >>> attacker >>> > > > > > > >> > >>> >> >>>>>>>>>>>> can get access >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > to one of the machines, then it >>> should >>> > > > also >>> > > > > be >>> > > > > > > >> > >>> possible to >>> > > > > > > >> > >>> >> >>>>>>>>>>>> obtain the right >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Kerberos token. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > I am not an authentication expert >>> and >>> > > > that's >>> > > > > > > why I >>> > > > > > > >> > >>> wanted >>> > > > > > > >> > >>> >> to >>> > > > > > > >> > >>> >> >>>>>>>>>>>> ask what are >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > other authentication protocols >>> other >>> > > than >>> > > > > > > >> Kerberos? >>> > > > > > > >> > >>> Why did >>> > > > > > > >> > >>> >> >>>>>>>>>>>> we select >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Kerberos and not any other >>> > > authentication >>> > > > > > > >> protocol? >>> > > > > > > >> > >>> Maybe >>> > > > > > > >> > >>> >> you >>> > > > > > > >> > >>> >> >>>>>>>>>>>> can list the >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > pros and cons for the different >>> > > protocols. >>> > > > > Is >>> > > > > > > >> > Kerberos >>> > > > > > > >> > >>> also >>> > > > > > > >> > >>> >> >>>>>>>>>>>> the standard >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > authentication protocol for >>> Kubernetes >>> > > > > > > >> deployments? >>> > > > > > > >> > If >>> > > > > > > >> > >>> not, >>> > > > > > > >> > >>> >> >>>>>>>>>>>> what would be >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > the answer when deploying on K8s? >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Cheers, >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Till >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > On Wed, Jun 2, 2021 at 12:07 PM >>> Gabor >>> > > > > Somogyi >>> > > > > > < >>> > > > > > > >> > >>> >> >>>>>>>>>>>> gabor.g.somo...@gmail.com> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > wrote: >>> > > > > > > >> > >>> >> >>>>>>>>>>>> > >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> Hi team, >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> Happy to be here and hope I can >>> > provide >>> > > > > > quality >>> > > > > > > >> > >>> additions >>> > > > > > > >> > >>> >> in >>> > > > > > > >> > >>> >> >>>>>>>>>>>> the future. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> Thank you all for helpful the >>> > > > suggestions! >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> Considering them the FLIP has >>> been >>> > > > modified >>> > > > > > and >>> > > > > > > >> the >>> > > > > > > >> > >>> work >>> > > > > > > >> > >>> >> >>>>>>>>>>>> continues on the >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> already existing Jira. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> BR, >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> G >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> On Wed, Jun 2, 2021 at 11:23 AM >>> > Márton >>> > > > > > Balassi >>> > > > > > > < >>> > > > > > > >> > >>> >> >>>>>>>>>>>> balassi.mar...@gmail.com> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> wrote: >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Thanks, Chesney - I totally >>> missed >>> > > that. >>> > > > > > > >> Answered >>> > > > > > > >> > on >>> > > > > > > >> > >>> the >>> > > > > > > >> > >>> >> >>>>>>>>>>>> ticket too, let >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> us continue there then. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Till, I agree that we should >>> keep >>> > this >>> > > > > > > codepath >>> > > > > > > >> as >>> > > > > > > >> > >>> slim >>> > > > > > > >> > >>> >> as >>> > > > > > > >> > >>> >> >>>>>>>>>>>> possible. It >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> is an important design decision >>> that >>> > > we >>> > > > > aim >>> > > > > > to >>> > > > > > > >> keep >>> > > > > > > >> > >>> the >>> > > > > > > >> > >>> >> >>>>>>>>>>>> list of >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> authentication protocols to a >>> > minimum. >>> > > > We >>> > > > > > > >> believe >>> > > > > > > >> > >>> that >>> > > > > > > >> > >>> >> this >>> > > > > > > >> > >>> >> >>>>>>>>>>>> should not be a >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> primary concern of Flink and a >>> > trusted >>> > > > > proxy >>> > > > > > > >> > service >>> > > > > > > >> > >>> (for >>> > > > > > > >> > >>> >> >>>>>>>>>>>> example Apache >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Knox) should be used to enable a >>> > > > multitude >>> > > > > > of >>> > > > > > > >> > enduser >>> > > > > > > >> > >>> >> >>>>>>>>>>>> authentication >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> mechanisms. The bare minimum of >>> > > > > > authentication >>> > > > > > > >> > >>> mechanisms >>> > > > > > > >> > >>> >> >>>>>>>>>>>> to support >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> consequently consist of a single >>> > > strong >>> > > > > > > >> > >>> authentication >>> > > > > > > >> > >>> >> >>>>>>>>>>>> protocol for which >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Kerberos is the enterprise >>> solution >>> > > and >>> > > > > HTTP >>> > > > > > > >> Basic >>> > > > > > > >> > >>> >> primary >>> > > > > > > >> > >>> >> >>>>>>>>>>>> for development >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> and light-weight scenarios. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Added the above wording to G's >>> doc. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>> > > > > > > >> > >>> >>> > > > > > > >> > >>> > > > > > > >> >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> On Tue, Jun 1, 2021 at 11:47 AM >>> > > Chesnay >>> > > > > > > >> Schepler < >>> > > > > > > >> > >>> >> >>>>>>>>>>>> ches...@apache.org> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> wrote: >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> There's a related effort: >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>> > > > > > > >> https://issues.apache.org/jira/browse/FLINK-21108 >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> On 6/1/2021 10:14 AM, Till >>> Rohrmann >>> > > > > wrote: >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Hi Gabor, welcome to the >>> Flink >>> > > > > community! >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Thanks for sharing this >>> proposal >>> > > with >>> > > > > the >>> > > > > > > >> > >>> community >>> > > > > > > >> > >>> >> >>>>>>>>>>>> Márton. In >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> general, I >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > agree that authentication is >>> > > missing >>> > > > > and >>> > > > > > > that >>> > > > > > > >> > >>> this is >>> > > > > > > >> > >>> >> >>>>>>>>>>>> required for >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> using >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Flink within an enterprise. >>> The >>> > > > thing I >>> > > > > > am >>> > > > > > > >> > >>> wondering >>> > > > > > > >> > >>> >> is >>> > > > > > > >> > >>> >> >>>>>>>>>>>> whether this >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > feature strictly needs to be >>> > > > > implemented >>> > > > > > > >> inside >>> > > > > > > >> > of >>> > > > > > > >> > >>> >> Flink >>> > > > > > > >> > >>> >> >>>>>>>>>>>> or whether a >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> proxy >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > setup could do the job? Have >>> you >>> > > > > > considered >>> > > > > > > >> this >>> > > > > > > >> > >>> >> option? >>> > > > > > > >> > >>> >> >>>>>>>>>>>> If yes, then >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> it >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > would be good to list it >>> under >>> > the >>> > > > > point >>> > > > > > of >>> > > > > > > >> > >>> rejected >>> > > > > > > >> > >>> >> >>>>>>>>>>>> alternatives. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > I do see the benefit of >>> > > implementing >>> > > > > this >>> > > > > > > >> > feature >>> > > > > > > >> > >>> >> inside >>> > > > > > > >> > >>> >> >>>>>>>>>>>> of Flink if >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> many >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > users need it. If not, then >>> it >>> > > might >>> > > > be >>> > > > > > > >> easier >>> > > > > > > >> > >>> for the >>> > > > > > > >> > >>> >> >>>>>>>>>>>> project to not >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > increase the surface area >>> since >>> > it >>> > > > > makes >>> > > > > > > the >>> > > > > > > >> > >>> overall >>> > > > > > > >> > >>> >> >>>>>>>>>>>> maintenance >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> harder. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Cheers, >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Till >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > On Mon, May 31, 2021 at 4:57 >>> PM >>> > > > Márton >>> > > > > > > >> Balassi < >>> > > > > > > >> > >>> >> >>>>>>>>>>>> mbala...@apache.org> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> wrote: >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Hi team, >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Firstly I would like to >>> > introduce >>> > > > > Gabor >>> > > > > > > or G >>> > > > > > > >> > [1] >>> > > > > > > >> > >>> for >>> > > > > > > >> > >>> >> >>>>>>>>>>>> short to the >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> community, he is a Spark >>> > committer >>> > > > who >>> > > > > > has >>> > > > > > > >> > >>> recently >>> > > > > > > >> > >>> >> >>>>>>>>>>>> transitioned to >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> the >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Flink Engineering team at >>> > Cloudera >>> > > > and >>> > > > > > is >>> > > > > > > >> > looking >>> > > > > > > >> > >>> >> >>>>>>>>>>>> forward to >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> contributing >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> to Apache Flink. Previously >>> G >>> > > > > primarily >>> > > > > > > >> focused >>> > > > > > > >> > >>> on >>> > > > > > > >> > >>> >> >>>>>>>>>>>> Spark Streaming >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> and >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> security. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Based on requests from our >>> > > > customers G >>> > > > > > has >>> > > > > > > >> > >>> >> implemented >>> > > > > > > >> > >>> >> >>>>>>>>>>>> Kerberos and >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> HTTP >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Basic Authentication for the >>> > Flink >>> > > > > > > Dashboard >>> > > > > > > >> > and >>> > > > > > > >> > >>> >> >>>>>>>>>>>> HistoryServer. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> Previously >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> lacked an authentication >>> story. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> We are looking to contribute >>> > this >>> > > > > > > >> functionality >>> > > > > > > >> > >>> back >>> > > > > > > >> > >>> >> to >>> > > > > > > >> > >>> >> >>>>>>>>>>>> the >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> community, we >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> believe that given Flink's >>> > > maturity >>> > > > > > there >>> > > > > > > >> > should >>> > > > > > > >> > >>> be a >>> > > > > > > >> > >>> >> >>>>>>>>>>>> common code >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> solution >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> for this general pattern. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> We are looking forward to >>> your >>> > > > > feedback >>> > > > > > on >>> > > > > > > >> G's >>> > > > > > > >> > >>> >> design. >>> > > > > > > >> > >>> >> >>>>>>>>>>>> [2] >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> [1] >>> http://gaborsomogyi.com/ >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> [2] >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>> > > > > > > >> > >>> >>> > > > > > > >> > >>> > > > > > > >> >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> > > > > > > >> > >>> >> >>>>>>>>>>> >>> > > > > > > >> > >>> >> >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > -- >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > Konstantin Knauf >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > https://twitter.com/snntrable >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> > https://github.com/knaufk >>> > > > > > > >> > >>> > >>> > > > > > > >> > >>> >>> > > > > > > >> > >> >>> > > > > > > >> > >> >>> > > > > > > >> > >> -- >>> > > > > > > >> > >> >>> > > > > > > >> > >> Konstantin Knauf >>> > > > > > > >> > >> >>> > > > > > > >> > >> https://twitter.com/snntrable >>> > > > > > > >> > >> >>> > > > > > > >> > >> https://github.com/knaufk >>> > > > > > > >> > >> >>> > > > > > > >> > > >>> > > > > > > >> > >>> > > > > > > >> >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > >>> > > > -- >>> > > > >>> > > > Konstantin Knauf >>> > > > >>> > > > https://twitter.com/snntrable >>> > > > >>> > > > https://github.com/knaufk >>> > > > >>> > > >>> > >>> >>
