String dirPath = "s3n://s3-us-west-1.amazonaws.com/com.pws.twitter/*json” *
not sure, but can you try to remove s3-us-west-1.amazonaws.com <http://s3-us-west-1.amazonaws.com/com.pws.twitter/> from path ? On 11 February 2016 at 23:15, Andy Davidson <a...@santacruzintegration.com> wrote: > I am using spark 1.6.0 in a cluster created using the spark-ec2 script. I > am using the standalone cluster manager > > My java streaming app is not able to write to s3. It appears to be some > for of permission problem. > > Any idea what the problem might be? > > I tried use the IAM simulator to test the policy. Everything seems okay. > Any idea how I can debug this problem? > > Thanks in advance > > Andy > > JavaSparkContext jsc = new JavaSparkContext(conf); > > // I did not include the full key in my email > // the keys do not contain ‘\’ > // these are the keys used to create the cluster. They belong to > the IAM user andy > > jsc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", "AKIAJREX" > ); > > jsc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey", > "uBh9v1hdUctI23uvq9qR"); > > > > private static void saveTweets(JavaDStream<String> jsonTweets, String > outputURI) { > > jsonTweets.foreachRDD(new VoidFunction2<JavaRDD<String>, Time>() { > > private static final long serialVersionUID = 1L; > > > @Override > > public void call(JavaRDD<String> rdd, Time time) throws > Exception { > > if(!rdd.isEmpty()) { > > // bucket name is ‘com.pws.twitter’ it has a folder ‘json' > > String dirPath = "s3n:// > s3-us-west-1.amazonaws.com/com.pws.twitter/*json” *+ "-" + time > .milliseconds(); > > rdd.saveAsTextFile(dirPath); > > } > > } > > }); > > > > > Bucket name : com.pws.titter > Bucket policy (I replaced the account id) > > { > "Version": "2012-10-17", > "Id": "Policy1455148808376", > "Statement": [ > { > "Sid": "Stmt1455148797805", > "Effect": "Allow", > "Principal": { > "AWS": "arn:aws:iam::123456789012:user/andy" > }, > "Action": "s3:*", > "Resource": "arn:aws:s3:::com.pws.twitter/*" > } > ] > } > > >
