As far as I know there is currently no encryption in-memory in Spark. There are some research projects to create secure enclaves in-memory based on Intel sgx, but there is still a lot to do in terms of performance and security objectives. The more interesting question is why would you need this for your organization. There are very few scenarios where it could be needed and if you have attacker’s in the cluster you have anyway other problems.
> On 23. Sep 2017, at 09:41, Irfan Kabli <irfan.kabli...@gmail.com> wrote: > > Dear All, > > We are looking to position MLLib in our organisation for machine learning > tasks and are keen to understand if their are any challenges that you might > have seen with MLLib in production. We will be going with the pure > open-source approach here, rather than using one of the hadoop distributions > out their in the market. > > Furthemore, with a multi-tenant hadoop cluster, and data in memory, would > spark support encrypting the data in memory with DataFrames. > > -- > Best Regards, > Irfan Kabli >
