"fs.s3a.server-side-encryption-algorithm" is honored by s3a support in
hadoop 2.6.0+ as well.
Cheers
On Thu, Jan 29, 2015 at 6:51 AM, Danny wrote:
> On Spark 1.2.0 you have the "s3a" library to work with S3. And there is a
> config param named "fs.s3a.server-side-encryption-algorithm":
>
> http
On Spark 1.2.0 you have the "s3a" library to work with S3. And there is a
config param named "fs.s3a.server-side-encryption-algorithm":
https://github.com/Aloisius/hadoop-s3a
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-and-S3-server-side-encrypti
ng the folder
> containing my jets3t.properties. But, I can’t get things working when
> trying to submit this to a spark stand-alone cluster.
>
> Any insights would be appreciated.
> --------------
> *From:* Thomas Demoor
> *Sent:* Tuesday, January 27, 2015 4:41 A
s. But, I can't get things working when
trying to submit this to a spark stand-alone cluster.
Any insights would be appreciated.?
From: Thomas Demoor
Sent: Tuesday, January 27, 2015 4:41 AM
To: Kohler, Curt E (ELS-STL)
Cc: user@spark.apache.org
Subject: Re:
Adding on what Thomas said. There have been a few bug fixes for s3a since
Hadoop 2.6.0 was released. One example is HADOOP-11446.
The fixes would be in Hadoop 2.7.0
Cheers
> On Jan 27, 2015, at 1:41 AM, Thomas Demoor
> wrote:
>
> Spark uses the Hadoop filesystems.
>
> I assume you are try
Spark uses the Hadoop filesystems.
I assume you are trying to use s3n:// which, under the hood, uses the 3rd
party jets3t library. It is configured through the jets3t.properties file
(google "hadoop s3n jets3t") which you should put on Spark's classpath. The
setting you are looking for is s3servic
