Your output folder specifies
rdd.saveAsTextFile("s3n://nexgen-software/dev/output");
So it will try to write to /dev/output which is as expected. If you create
the directory /dev/output upfront in your bucket, and try to save it to
that (empty) directory, what is the behaviour?
On Tue, Jan 27, 2015 at 6:21 AM, Chen, Kevin <kevin.c...@neustar.biz> wrote:
> Does anyone know if I can save a RDD as a text file to a pre-created
> directory in S3 bucket?
>
> I have a directory created in S3 bucket: //nexgen-software/dev
>
> When I tried to save a RDD as text file in this directory:
> rdd.saveAsTextFile("s3n://nexgen-software/dev/output");
>
>
> I got following exception at runtime:
>
> Exception in thread "main" org.apache.hadoop.fs.s3.S3Exception:
> org.jets3t.service.S3ServiceException: S3 HEAD request failed for '/dev' -
> ResponseCode=403, ResponseMessage=Forbidden
>
>
> I have verified /dev has write permission. However, if I grant the
> bucket //nexgen-software write permission, I don't get exception. But the
> output is not created under dev. Rather, a different /dev/output directory
> is created directory in the bucket (//nexgen-software). Is this how
> saveAsTextFile behalves in S3? Is there anyway I can have output created
> under a pre-defied directory.
>
>
> Thanks in advance.
>
>
>
>
>
