are you using speculation? On 15 September 2016 at 21:37, Chen, Kevin <kevin.c...@neustar.biz> wrote:
> Hi, > > Has any one encountered an issue of missing output partition file in S3 ? > My spark job writes output to a S3 location. Occasionally, I noticed one > partition file is missing. As a result, one chunk of data was lost. If I > rerun the same job, the problem usually goes away. This has been happening > pretty random. I observed once or twice a week on a daily run job. I am > using Spark 1.2.1. > > Very much appreciated on any input, suggestion of fix/workaround. > > > >
