Hi folks, I'm happy to share with you a draft of a FLIP for service authorization. As I mentioned at the top of this thread, the goal is to protect a deployed Flink cluster/session from unauthorized use. In the doc, I propose the use of SSL client authentication for internal communication, plus YARN/Kubernetes/Mesos-specific functionality to achieve single sign-on for the web interface/API.
FLIP-? - Service Authorization <https://docs.google.com/document/d/13IRPb2GdL842rIzMgEn0ibOQHNku6W8aMf1p7gCPJjg/edit?usp=sharing> Feel free to make comments and to give feedback on whether this feature would be useful to you. Thanks! Eron On Thu, Aug 3, 2017 at 11:11 AM, Eron Wright <eronwri...@gmail.com> wrote: > Till, with (c) are you suggesting that we'd use Akka 2.3 for Scala 2.10 > and Akka 2.4+ for Scala 2.11+? Sounds reasonable but I don't know how > feasible it is. I will say I'm optimistic because a) Akka 2.4 is said to > be binary compatible, and b) the Flakka fork appears to be subsumed by 2.4. > > Let us then take (c) as the tentative plan. > > I agree the community should discuss dropping Scala 2.10 but I don't want > to drive that conversation. > > Thanks > > > On Thu, Aug 3, 2017 at 6:24 AM, Ufuk Celebi <u...@apache.org> wrote: > >> I haven't followed this discussion in detail nor am I familiar with >> the service authorization topic or Flakka, but a) sounds like a lot of >> maintenance work to me. >> >> If possible I would go with c) and maybe start a discussion about >> dropping Scala 2.10 support to check whether that is a viable option >> or not. >> >> – Ufuk >> >> >> On Thu, Aug 3, 2017 at 1:59 PM, Till Rohrmann <trohrm...@apache.org> >> wrote: >> > Alternatively there would also be an option >> > >> > c) only support mutual auth for Akka 2.4+ if the backport is >> unrealistic to >> > do >> > >> > But this of course would break security for Scala 2.10. On the other >> hand >> > people are already using Flink without this feature. >> > >> > Cheers, >> > Till >> > >> > On Wed, Aug 2, 2017 at 7:21 PM, Eron Wright <eronwri...@gmail.com> >> wrote: >> > >> >> Thanks Till and Aljoscha for the feedback. >> >> >> >> Seems there are two ways to proceed here, if we accept mutual SSL as >> the >> >> basis. >> >> >> >> a) Backport mutual-auth support from Akka 2.4 to Flakka. >> >> b) Drop support for Scala 2.10 (FLINK-?), move to Akka 2.4 >> (FLINK-3662). >> >> >> >> Let's assume (a) for now. >> >> >> >> >> >> >> >> On Tue, Aug 1, 2017 at 3:05 PM, Till Rohrmann <trohrm...@apache.org> >> >> wrote: >> >> >> >> > Dropping Java 7 alone is not enough to move to Akka 2.4+. For that we >> >> need >> >> > at least Scala 2.11. >> >> > >> >> > Cheers, >> >> > Till >> >> > >> >> > On Tue, Aug 1, 2017 at 4:22 PM, Aljoscha Krettek < >> aljos...@apache.org> >> >> > wrote: >> >> > >> >> > > Hi Eron, >> >> > > >> >> > > I think after Dropping support for Java 7 we will move to Akka >> 2.4+, so >> >> > we >> >> > > should be good there. I think quite some users should find a (more) >> >> > secure >> >> > > Flink interesting. >> >> > > >> >> > > Best, >> >> > > Aljoscha >> >> > > > On 24. Jul 2017, at 03:11, Eron Wright <eronwri...@gmail.com> >> wrote: >> >> > > > >> >> > > > Hello, now might be a good time to revisit an important >> enhancement >> >> to >> >> > > > Flink security, so-called service authorization. This means the >> >> > > hardening >> >> > > > of a Flink cluster against unauthorized use with some sort of >> >> > > > authentication and authorization scheme. Today, Flink relies >> >> entirely >> >> > > on >> >> > > > network isolation to protect itself from unauthorized job >> submission >> >> > and >> >> > > > control, and to protect the secrets contained within a Flink >> cluster. >> >> > > > This is a problem in multi-user environments like YARN/Mesos/K8. >> >> > > > >> >> > > > Last fall, an effort was made to implement service authorization >> but >> >> > the >> >> > > PR >> >> > > > was ultimately rejected. The idea was to add a simple secret >> key to >> >> > all >> >> > > > network communication between the client, JM, and TM. Akka >> itself >> >> has >> >> > > > such a feature which formed the basis of the solution. There are >> >> > > usability >> >> > > > challenges with this solution, including a dependency on SSL. >> >> > > > >> >> > > > Since then, the situation has evolved somewhat, and the use of >> SSL >> >> > mutual >> >> > > > authentication is more viable. Mutual auth is supported in Akka >> >> > 2.4.12+ >> >> > > > (or could be backported to Flakka). My proposal is: >> >> > > > >> >> > > > 1. Upgrade Akka or backport the functionality to Flakka (see >> commit >> >> > > > 5d03902c5ec3212cd28f26c9b3ef7c3b628b9451). >> >> > > > 2. Implement SSL on any endpoint that doesn't yet support it >> (e.g. >> >> > > > queryable state). >> >> > > > 3. Enable mutual auth in Akka and implement it on non-Akka >> endpoints. >> >> > > > 4. Implement a simple authorization layer that accepts any >> >> > authenticated >> >> > > > connection. >> >> > > > 5. (stretch) generate and store a certificate automatically in >> YARN >> >> > mode. >> >> > > > 6. (stretch) Develop an alternate authentication method for the >> Web >> >> UI. >> >> > > > >> >> > > > Are folks interested in this capability? Thoughts on the use of >> SSL >> >> > > mutual >> >> > > > auth versus something else? Thanks! >> >> > > > >> >> > > > -Eron >> >> > > >> >> > > >> >> > >> >> >> > >
