Re: Cannot read from s3 using "sc.textFile"

2014-10-07 Thread Daniil Osipov
Try using s3n:// instead of s3 (for the credential configuration as well). On Tue, Oct 7, 2014 at 9:51 AM, Sunny Khatri wrote: > Not sure if it's supposed to work. Can you try newAPIHadoopFile() passing > in the required configuration object. > > On Tue, Oct 7, 2014 at 4:20 AM, Tomer Benyamini

Re: Cannot read from s3 using "sc.textFile"

2014-10-07 Thread Sunny Khatri
Not sure if it's supposed to work. Can you try newAPIHadoopFile() passing in the required configuration object. On Tue, Oct 7, 2014 at 4:20 AM, Tomer Benyamini wrote: > Hello, > > I'm trying to read from s3 using a simple spark java app: > > - > > SparkConf sparkConf = new Sp

Fwd: Cannot read from s3 using "sc.textFile"

2014-10-07 Thread Tomer Benyamini
Hello, I'm trying to read from s3 using a simple spark java app: - SparkConf sparkConf = new SparkConf().setAppName("TestApp"); sparkConf.setMaster("local"); JavaSparkContext sc = new JavaSparkContext(sparkConf); sc.hadoopConfiguration().set("fs.s3.awsAccessKeyId", "XX");

Cannot read from s3 using "sc.textFile"

2014-10-07 Thread Tomer Benyamini
Hello, I'm trying to read from s3 using a simple spark java app: - SparkConf sparkConf = new SparkConf().setAppName("TestApp"); sparkConf.setMaster("local"); JavaSparkContext sc = new JavaSparkContext(sparkConf); sc.hadoopConfiguration().set("fs.s3.awsAccessKeyId", "XX");