Thanks a lot, Eron. I'll draft a proposal and share it with the community. On Thu, Jan 18, 2018 at 4:18 PM, Eron Wright <eronwri...@gmail.com> wrote:
> I would suggest that you draft a proposal that lays out your goals and the > technical challenges that you perceive. Then the community can provide > some feedback on potential solutions to those challenges, culminating in a > concrete improvement proposal. > > Thanks > > On Wed, Jan 17, 2018 at 7:29 PM, Shuyi Chen <suez1...@gmail.com> wrote: > > > Ping, any comments? Thanks a lot. > > > > Shuyi > > > > On Wed, Jan 3, 2018 at 3:43 PM, Shuyi Chen <suez1...@gmail.com> wrote: > > > > > Thanks a lot for the clarification, Eron. That's very helpful. > Currently, > > > we are more concerned about 1) data access, but will get to 2) and 3) > > > eventually. > > > > > > I was thinking doing the following: > > > 1) extend the current HadoopModule to use and refresh DTs as suggested > > on YARN > > > Application Security docs. > > > 2) I found the current SecurityModule interface might be enough for > > > supporting other security mechanisms. However, the loading of security > > > modules are hard-coded, not configuration based. I think we can extend > > > SecurityUtils to load from configurations. So we can implement our own > > > security mechanism in our internal repo, and have flink jobs to load it > > at > > > runtime. > > > > > > Please let me know your comments. Thanks a lot. > > > > > > On Fri, Dec 22, 2017 at 3:05 PM, Eron Wright <eronwri...@gmail.com> > > wrote: > > > > > >> I agree that it is reasonable to use Hadoop DTs as you describe. That > > >> approach is even recommended in YARN's documentation (see Securing > > >> Long-lived YARN Services on the YARN Application Security page). But > > one > > >> of the goals of Kerberos integration is to support Kerberized data > > access > > >> for connectors other than HDFS, such as Kafka, Cassandra, and > > >> Elasticsearch. So your second point makes sense too, suggesting a > > >> general > > >> architecture for managing secrets (DTs, keytabs, certificates, oauth > > >> tokens, etc.) within the cluster. > > >> > > >> There's quite a few aspects to Flink security, including: > > >> 1. data access (e.g. how a connector authenticates to a data source) > > >> 2. service authorization and network security (e.g. how a Flink > cluster > > >> protects itself from unauthorized access) > > >> 3. multi-user support (e.g. multi-user Flink clusters, RBAC) > > >> > > >> I mention these aspects to clarify your point about AuthN, which I > took > > to > > >> be related to (1). Do tell if I misunderstood. > > >> > > >> Eron > > >> > > >> > > >> On Wed, Dec 20, 2017 at 11:21 AM, Shuyi Chen <suez1...@gmail.com> > > wrote: > > >> > > >> > Hi community, > > >> > > > >> > We are working on secure Flink on YARN. The current > > Flink-Yarn-Kerberos > > >> > integration will require each container of a job to log in Kerberos > > via > > >> > keytab every say, 24 hours, and does not use any Hadoop delegation > > token > > >> > mechanism except when localizing the container. As I fixed the > current > > >> > Flink-Yarn-Kerberos (FLINK-8275) and tried to add more > > >> > features(FLINK-7860), I have some concern regarding the current > > >> > implementation. It can pose a scalability issue to the KDC, e.g., if > > >> YARN > > >> > cluster is restarted and all 10s of thousands of containers suddenly > > >> DDOS > > >> > KDC. > > >> > > > >> > I would like to propose to improve the current Flink-YARN-Kerberos > > >> > integration by doing something like the following: > > >> > 1) AppMaster (JobManager) periodically authenticate the KDC, get all > > >> > required DTs for the job. > > >> > 2) all other TM or TE containers periodically retrieve new DTs from > > the > > >> > AppMaster (either through a secure HDFS folder, or a secure Akka > > >> channel) > > >> > > > >> > Also, we want to extend Flink to support pluggable AuthN mechanism, > > >> because > > >> > we have our own internal AuthN mechanism. We would like add support > in > > >> > Flink to authenticate periodically to our internal AuthN service as > > well > > >> > through, e.g., dynamic class loading, and use similar mechanism to > > >> > distribute the credential from the appMaster to containers. > > >> > > > >> > I would like to get comments and feedbacks. I can also write a > design > > >> doc > > >> > or create a Flip if needed. Thanks a lot. > > >> > > > >> > Shuyi > > >> > > > >> > > > >> > > > >> > -- > > >> > "So you have to trust that the dots will somehow connect in your > > >> future." > > >> > > > >> > > > > > > > > > > > > -- > > > "So you have to trust that the dots will somehow connect in your > future." > > > > > > > > > > > -- > > "So you have to trust that the dots will somehow connect in your future." > > > -- "So you have to trust that the dots will somehow connect in your future."
