mailto:ste...@hortonworks.com>>
Date: Friday, September 16, 2016 at 3:46 AM
To: Chen Kevin mailto:kevin.c...@neustar.biz>>
Cc: "user@spark.apache.org<mailto:user@spark.apache.org>"
mailto:user@spark.apache.org>>
Subject: Re: Missing output partition file
> Begin forwarded message:
>
> From: "Chen, Kevin"
> Subject: Re: Missing output partition file in S3
> Date: September 19, 2016 at 10:54:44 AM PDT
> To: Steve Loughran
> Cc: "user@spark.apache.org"
>
> Hi Steve,
>
> Our S3 is on US
,
Kevin
From: Steve Loughran mailto:ste...@hortonworks.com>>
Date: Friday, September 16, 2016 at 3:46 AM
To: Chen Kevin mailto:kevin.c...@neustar.biz>>
Cc: "user@spark.apache.org<mailto:user@spark.apache.org>"
mailto:user@spark.apache.org>>
Subject: Re: Missing o
Sent from my iPhone
> On Sep 15, 2016, at 1:37 PM, Chen, Kevin wrote:
>
> Hi,
>
> Has any one encountered an issue of missing output partition file in S3 ? My
> spark job writes output to a S3 location. Occasionally, I noticed one
> partition file is missing. As a resu
are you using speculation?
On 15 September 2016 at 21:37, Chen, Kevin wrote:
> Hi,
>
> Has any one encountered an issue of missing output partition file in S3 ?
> My spark job writes output to a S3 location. Occasionally, I noticed one
> partition file is missing. As a result, on
On 15 Sep 2016, at 19:37, Chen, Kevin
mailto:kevin.c...@neustar.biz>> wrote:
Hi,
Has any one encountered an issue of missing output partition file in S3 ? My
spark job writes output to a S3 location. Occasionally, I noticed one partition
file is missing. As a result, one chunk of da
Hi,
Has any one encountered an issue of missing output partition file in S3 ? My
spark job writes output to a S3 location. Occasionally, I noticed one partition
file is missing. As a result, one chunk of data was lost. If I rerun the same
job, the problem usually goes away. This has been
ata loss during runtime, and then stop our Spark
job completely ASAP if it happens?
Thanks,
Nicolas
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Missing-output-partition-file-in-S3-tp21326.html
Sent from the Apache Spark User List mailing list archive at Na
