Hi,
I am sorry, I do read this https://wiki.apache.org/hadoop/AmazonS3 which
mentions about s3:// being deprecated. From what I read using s3a is the
preferred way to go.
Ofcourse, I have been using it for writing data from SPARK but not for
reading yet. Let me try that and come back.
Regards,
G
How do I use it? I'm accessing s3a from Spark's textFile API.
On Tue, May 31, 2016 at 7:16 AM, Deepak Sharma
wrote:
> Hi Mayuresh
> Instead of s3a , have you tried the https:// uri for the same s3 bucket?
>
> HTH
> Deepak
>
> On Tue, May 31, 2016 at 4:41 PM, Mayuresh Kunjir
> wrote:
>
>>
>>
>>
On Tue, May 31, 2016 at 7:05 AM, Gourav Sengupta
wrote:
> Hi,
>
> And on another note, is it required to use s3a? Why not use s3:// only? I
> prefer to use s3a:// only while writing files to S3 from EMR
>
Does Spark support s3://? I am using s3a over s3n because I needed to
access files larger
Hi Mayuresh
Instead of s3a , have you tried the https:// uri for the same s3 bucket?
HTH
Deepak
On Tue, May 31, 2016 at 4:41 PM, Mayuresh Kunjir
wrote:
>
>
> On Tue, May 31, 2016 at 5:29 AM, Steve Loughran
> wrote:
>
>> which s3 endpoint?
>>
>>
> I have tried both s3.amazonaws.com and s3-exte
On Tue, May 31, 2016 at 5:29 AM, Steve Loughran
wrote:
> which s3 endpoint?
>
>
I have tried both s3.amazonaws.com and s3-external-1.amazonaws.com.
>
>
> On 29 May 2016, at 22:55, Mayuresh Kunjir wrote:
>
> I'm running into permission issues while accessing data in S3 bucket
> stored using s
Hi,
And on another note, is it required to use s3a? Why not use s3:// only? I
prefer to use s3a:// only while writing files to S3 from EMR.
Regards,
Gourav Sengupta
On Tue, May 31, 2016 at 12:04 PM, Gourav Sengupta wrote:
> Hi,
>
> Is your spark cluster running in EMR or via self created SPARK
Hi,
Is your spark cluster running in EMR or via self created SPARK cluster
using EC2 or from a local cluster behind firewall? What is the SPARK
version you are using?
Regards,
Gourav Sengupta
On Sun, May 29, 2016 at 10:55 PM, Mayuresh Kunjir
wrote:
> I'm running into permission issues while a
which s3 endpoint?
On 29 May 2016, at 22:55, Mayuresh Kunjir
mailto:mayur...@cs.duke.edu>> wrote:
I'm running into permission issues while accessing data in S3 bucket stored
using s3a file system from a local Spark cluster. Has anyone found success with
this?
My setup is:
- Spark 1.6.1 comp
On Sun, May 29, 2016 at 7:49 PM, Ted Yu wrote:
> Have you seen this thread ?
>
>
> http://search-hadoop.com/m/q3RTthWU8o1MbFC2&subj=Re+Forbidded+Error+Code+403
>
>
Thanks for the pointer. I have followed the thread, got no success though.
I am trying out the Spark branch suggested by Teng Qiu
Have you seen this thread ?
http://search-hadoop.com/m/q3RTthWU8o1MbFC2&subj=Re+Forbidded+Error+Code+403
On Sun, May 29, 2016 at 2:55 PM, Mayuresh Kunjir
wrote:
> I'm running into permission issues while accessing data in S3 bucket
> stored using s3a file system from a local Spark cluster. Has
10 matches
Mail list logo
