Hi Sourav! If you want to use Flink in a cluster where neither Hadoop/YARN (not soon Mesos) is available, then I assume you have installed Flink in a standalone mode on the cluster already.
There is no support in Flink currently to manage user authentication. Few thoughts on how that may evolve 1) It should be not too hard to add authentication to the web dashboard. That way, if the cluster is otherwise blocked off (the master's RPC ports are firewalled), one would have restricted job starts. 2) We plan to add authenticated / encrypted connections soon. With that, the client that submits the program would need to have access to the keystore or key and the corresponding password to connect. Greetings, Stephan On Mon, Jan 11, 2016 at 3:46 PM, Sourav Mazumder < sourav.mazumde...@gmail.com> wrote: > Thanks Steven for your details response. Things are more clear to me now. > > A follow up Qs - > Looks like most of the security support depends on Hadoop ? What happens > if anyone wants to use Flink with Hadoop (in a cluster where Hadoop is not > there) ? > > Regards, > Sourav > > On Sun, Jan 10, 2016 at 12:41 PM, Stephan Ewen <se...@apache.org> wrote: > >> Hi Sourav! >> >> There is user-authentication support in Flink via the Hadoop / Kerberos >> infrastructure. If you run Flink on YARN, it should seamlessly work that >> Flink acquires the Kerberos tokens of the user that submits programs, and >> authenticate itself at YARN, HDFS, and HBase with that. >> >> If you run Flink standalone, Flink can still authenticate at HDFS/HBase >> via Kerberos, with a bit of manual help by the user (running kinit on the >> workers). >> >> With Kafka 0.9 and Flink's upcoming connector ( >> https://github.com/apache/flink/pull/1489), streaming programs can >> authenticate themselves as stream brokers via SSL (and read via encrypted >> connections). >> >> >> What we have on the roadmap for the coming months it the following: >> - Encrypt in-flight data streams that are exchanged between worker >> nodes (TaskManagers). >> - Encrypt the coordination messages between client/master/workers. >> Note that these refer to encryption between Flink's own components only, >> which would use transient keys generated just for a specific job or session >> (hence would not need any user involvement). >> >> >> Let us know if that answers your questions, and if that meets your >> requirements. >> >> Greetings, >> Stephan >> >> >> On Fri, Jan 8, 2016 at 3:23 PM, Sourav Mazumder < >> sourav.mazumde...@gmail.com> wrote: >> >>> Hi, >>> >>> Can anyone point me to ant documentation on support for Security in >>> Flink ? >>> >>> The type of information I'm looking for are - >>> >>> 1. How do I do user level authentication to ensure that a job is >>> submitted/deleted/modified by the right user ? Is it possible though the >>> web client ? >>> 2. Authentication across multiple slave nodes (where the task managers >>> are running) and driver program so that they can communicate with each other >>> 3. Support for SSL/encryption for data exchanged happening across the >>> slave nodes >>> 4. Support for pluggable authentication with existing solution like LDAP >>> >>> If not there today is there a roadmap for these security features ? >>> >>> Regards, >>> Sourav >>> >> >> >
