I want to add that the major hadoop distributions also offer additional encryption possibilities (for example Ranger from Hortonworks)
Roland Johann Software Developer/Data Engineer phenetic GmbH Lütticher Straße 10, 50674 Köln, Germany Mobil: +49 172 365 26 46 Mail: roland.joh...@phenetic.io Web: phenetic.io Handelsregister: Amtsgericht Köln (HRB 92595) Geschäftsführer: Roland Johann, Uwe Reimann > Am 26.08.2019 um 23:09 schrieb Roland Johann <roland.joh...@phenetic.io>: > > Hi all, > > instead of handling encryption explicit at application level, I suggest that > you investigate into the topic „encryption at rest“, for example encryption > at HDFS level > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html > > <https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html> > - obviously if you don’t have to use PGP. Using encryption at the storage > layer simplifies your application and architecture and you don’t need to > reinvent the wheel. > > Kind Regards > > Roland Johann > Software Developer/Data Engineer > > phenetic GmbH > Lütticher Straße 10, 50674 Köln, Germany > > Mobil: +49 172 365 26 46 > Mail: roland.joh...@phenetic.io <mailto:roland.joh...@phenetic.io> > Web: phenetic.io <http://phenetic.io/> > > Handelsregister: Amtsgericht Köln (HRB 92595) > Geschäftsführer: Roland Johann, Uwe Reimann > > > >> Am 26.08.2019 um 17:47 schrieb Sachit Murarka <connectsac...@gmail.com >> <mailto:connectsac...@gmail.com>>: >> >> Hi Deepak, >> >> Thanks for reply. >> >> Yes. That is the option I am considering now because even apache camel needs >> data in local. I might need to copy data from hdfs to local if I want >> apache camel ( to get rid of shell). >> >> Thanks >> Sachit >> >> On Mon, 26 Aug 2019, 21:11 Deepak Sharma, <deepakmc...@gmail.com >> <mailto:deepakmc...@gmail.com>> wrote: >> Hi Schit >> PGP Encrypt is something that is not inbuilt with spark. >> I would suggest writing a shell script that would do pgp encrypt and use it >> in spark scala program , which would run from driver. >> >> Thanks >> Deepak >> >> On Mon, Aug 26, 2019 at 8:10 PM Sachit Murarka <connectsac...@gmail.com >> <mailto:connectsac...@gmail.com>> wrote: >> Hi All, >> >> I want to encrypt my files available at HDFS location using PGP Encryption >> How can I do it in spark. I saw Apache Camel but it seems camel is used >> when source files are in Local location rather than HDFS. >> >> Kind Regards, >> Sachit Murarka >> >> >> -- >> Thanks >> Deepak >> www.bigdatabig.com <http://www.bigdatabig.com/> >> www.keosha.net <http://www.keosha.net/>
