Re: spark 2.02 error when writing to s3

2017-01-28 Thread Steve Loughran
On 27 Jan 2017, at 23:17, VND Tremblay, Paul mailto:tremblay.p...@bcg.com>> wrote: Not sure what you mean by "a consistency layer on top." Any explanation would be greatly appreciated! Paul netflix's s3mper: https://github.com/Netflix/s3mper EMR consistency: http://docs.aws.amazon.com/emr

RE: spark 2.02 error when writing to s3

2017-01-27 Thread VND Tremblay, Paul
lay, Paul Cc: Takeshi Yamamuro; user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: spark 2.02 error when writing to s3 Can you test by enabling emrfs consistent view and use s3:// uri. http://docs.aws.amazon.com/emr/latest/ManagementGuide/enable-consistent-view.html Orig

Re: spark 2.02 error when writing to s3

2017-01-27 Thread Steve Loughran
uro; user@spark.apache.org<mailto:user@spark.apache.org> Subject: Re: spark 2.02 error when writing to s3 Can you test by enabling emrfs consistent view and use s3:// uri. http://docs.aws.amazon.com/emr/latest/ManagementGuide/enable-consistent-view.html Original message From

RE: spark 2.02 error when writing to s3

2017-01-26 Thread VND Tremblay, Paul
amuro; user@spark.apache.org Subject: Re: spark 2.02 error when writing to s3 Can you test by enabling emrfs consistent view and use s3:// uri. http://docs.aws.amazon.com/emr/latest/ManagementGuide/enable-consistent-view.html Original message From: Steve Loughran Date:20/01/20

Re: spark 2.02 error when writing to s3

2017-01-20 Thread Neil Jonkers
muro ,user@spark.apache.org Subject: Re: spark 2.02 error when writing to s3 AWS S3 is eventually consistent: even after something is deleted, a LIST/GET call may show it. You may be seeing that effect; even after the DELETE has got rid of the files, a listing sees something there, And I suspect

Re: spark 2.02 error when writing to s3

2017-01-20 Thread Steve Loughran
TING GROUP Tel. + ▪ Mobile + _ From: Takeshi Yamamuro [mailto:linguin@gmail.com] Sent: Thursday, January 19, 2017 9:27 PM To: VND Tremblay, Paul Cc: user@spark.apache.org<mailto:user@spark.apache.org> Su

RE: spark 2.02 error when writing to s3

2017-01-20 Thread VND Tremblay, Paul
Subject: Re: spark 2.02 error when writing to s3 Hi, Do you get the same exception also in v2.1.0? Anyway, I saw another guy reporting the same error, I think. https://www.mail-archive.com/user@spark.apache.org/msg60882.html // maropu On Fri, Jan 20, 2017 at 5:15 AM, VND Tremblay, Paul

Re: spark 2.02 error when writing to s3

2017-01-19 Thread Palash Gupta
Hi, You need to add mode overwrite option to avoid this error. //P.Gupta Sent from Yahoo Mail on Android On Fri, 20 Jan, 2017 at 2:15 am, VND Tremblay, Paul wrote: I have come across a problem when writing CSV files to S3 in Spark 2.02. The problem does not exist in Spark 1.6.   19:0

Re: spark 2.02 error when writing to s3

2017-01-19 Thread Takeshi Yamamuro
Hi, Do you get the same exception also in v2.1.0? Anyway, I saw another guy reporting the same error, I think. https://www.mail-archive.com/user@spark.apache.org/msg60882.html // maropu On Fri, Jan 20, 2017 at 5:15 AM, VND Tremblay, Paul wrote: > I have come across a problem when writing CSV