On 17 May 2017, at 00:10, jazzed
mailto:crackshotm...@gmail.com>> wrote:
How did you solve the problem with V4?
which v4 problem? Authentication?
you need to declare the explicit s3a endpoint via fs.s3a.endpoint , otherwise
you get a generic "bad auth" message which is not a good place to st
How did you solve the problem with V4?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/s3-bucket-access-read-file-tp23536p28688.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
I think 2.6 failed to abruptly close streams that weren't fully read, which
we observed as a huge performance hit. We had to backport the 2.7
improvements before being able to use it.
s3a uses amazon's own libraries; it's tested against frankfurt too.
you have to view s3a support in Hadoop 2.6 as beta-release: it works, with some
issues. Hadoop 2.7.0+ has it all working now, though are left with the task of
getting hadoop-aws and the amazon JAR onto your classpath via the --j
Should be able to use s3a (on new hadoop versions), I believe that will try
or at least has a setting for v4
On Tue, Jun 30, 2015 at 8:31 PM, Exie wrote:
> Not sure if this helps, but the options I set are slightly different:
>
> val hadoopConf=sc.hadoopConfiguration
> hadoopConf.set("fs.s3n.aws
Not sure if this helps, but the options I set are slightly different:
val hadoopConf=sc.hadoopConfiguration
hadoopConf.set("fs.s3n.awsAccessKeyId","key")
hadoopConf.set("fs.s3n.awsSecretAccessKey","secret")
Try setting them to s3n as opposed to just s3
Good luck!
--
View this message in conte
We finally managed to find the problem, the s3 files were located in
Frankfurt which only supports the *v4* signature
*Surprising* is the fact that the spark core library method textfile does
not support that!!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.co
Try this way:
val data = sc.textFile("s3n://ACCESS_KEY:SECRET_KEY@mybucket/temp/")
Thanks
Best Regards
On Mon, Jun 29, 2015 at 11:59 PM, didi wrote:
> Hi
>
> *Cant read text file from s3 to create RDD
> *
>
> after setting the configuration
> val hadoopConf=sparkContext.hadoopConfiguration;
>
Pls check your ACL properties.
On Monday, June 29, 2015 11:29 AM, didi wrote:
Hi
*Cant read text file from s3 to create RDD
*
after setting the configuration
val hadoopConf=sparkContext.hadoopConfiguration;
hadoopConf.set("fs.s3.impl",
"org.apache.hadoop.fs.s3native.NativeS3FileSys
